restart of HA kills zigbee database

[marc-gist]

I installed your component, and it works great... however, as soon as I restart HA, the zigbee database gets corrupted and on startup i get

[homeassistant.setup] Setup of zha_new is taking over 10 seconds.
[bellows.zigbee.application] watchdog: startup running or failed uart restart 
2019-01-28 08:25:47 DEBUG (MainThread) [bellows.uart] Sending: RESET
2019-01-28 08:25:47 DEBUG (MainThread) [bellows.uart] reset can only be called on a new connection

and the zha_new compoent and all the devices are not present :(

Removing the zigbee.db and pairing the devices again, gets it working... until restart of HA :(

[marc-gist]

Noticed this:

019-01-28 09:05:18 ERROR (MainThread) [homeassistant.setup] Error during setup of component zha_new Traceback (most recent call last): File "/home/vmware/homeassistant/lib/python3.6/site-packages/homeassistant/setup.py", line 145, in _async_setup_component hass, processed_config) File "/home/vmware/.homeassistant/custom_components/zha_new/init.py", line 272, in async_setup APPLICATIONCONTROLLER = ControllerApplication(ezsp, database) File "/home/vmware/homeassistant/lib/python3.6/site-packages/bellows/zigbee/application.py", line 21, in init super().init(database_file=database_file) File "/home/vmware/homeassistant/lib/python3.6/site-packages/zigpy/application.py", line 25, in init self.add_listener(self._dblistener) File "/home/vmware/homeassistant/lib/python3.6/site-packages/zigpy/appdb.py", line 181, in load self._cursor.executemany(q, endpoints) ValueError: too many values to unpack (expected 3) 2019-01-28 09:05:18 INFO (MainThread) [homeassistant.co

[Yoda-x]

Can you verify the zigpy version installed? Should be 100.1.4.4. Do you use skip-pip to start Hass?

[marc-gist]

I have 86.3.0 install of HA in a virtualenv. So likely my issue... do you have a "howto" setup with the packages you need? i start HA via "hass --open-ui"

[Yoda-x]

A venv is perfect. Just switch to your venv and do a 'pip list' and check the versions for bellows and Zigpy.

And do you my current master installed?

[marc-gist]

aiohttp 3.5.4
aiohttp-cors 0.7.0
aiohue 1.5.0
asn1crypto 0.24.0
astral 1.7.1
async-timeout 3.0.1
attrs 18.2.0
bcrypt 3.1.5
bellows 0.7.0
boto3 1.7.75
botocore 1.10.75
certifi 2018.11.29 cffi 1.11.5
chardet 3.0.4
click 6.7
click-datetime 0.2
click-log 0.2.0
crccheck 0.6
cryptography 2.3.1
distro 1.3.0
docutils 0.14
ecdsa 0.13
envs 1.2.6
future 0.16.0
gTTS-token 1.1.3
home-assistant-frontend 20190121.1 homeassistant 0.86.3
homeassistant-pyozw 0.1.2
idna 2.7
idna-ssl 1.1.0
Jinja2 2.10
jmespath 0.9.3
MarkupSafe 1.0
multidict 4.3.1
mutagen 1.42.0
netdisco 1.5.0
netifaces 0.10.7
paho-mqtt 1.4.0
pip 18.1
pkg-resources 0.0.0
PlexAPI 3.0.6
protobuf 3.6.0
pure-pcapy3 1.0.1
PyChromecast 2.1.0
pycparser 2.18
pycryptodome 3.7.3
PyDispatcher 2.0.5
pyharmony 1.0.20
pyHS100 0.3.3
PyJWT 1.6.4
pyotp 2.2.6
PyQRCode 1.2.1
pyserial 3.4
pyserial-asyncio 0.4
python-dateutil 2.7.3
python-forecastio 1.4.0
python-jose-cryptodome 1.3.2
python-openzwave 0.4.10
python-pushover 0.3
python-slugify 1.2.6
pytz 2018.9
pywemo 0.4.34
PyYAML 3.13
requests 2.21.0
responses 0.10.5
ruamel.yaml 0.15.85
s3transfer 0.1.13
setuptools 40.0.0
six 1.11.0
sleekxmpp 1.3.3
SQLAlchemy 1.2.15
tqdm 4.24.0
typing 3.6.4
typing-extensions 3.7.2
ua-parser 0.8.0
Unidecode 1.0.23
urllib3 1.23
user-agents 1.1.0
voluptuous 0.11.5
voluptuous-serialize 2.0.0
warrant 0.6.1
websocket-client 0.48.0
wheel 0.31.1
xmltodict 0.11.0
yarl 1.2.6
zeroconf 0.20.0
zha-quirks 0.0.6
zigpy 0.2.0
zigpy-deconz 0.0.1
zigpy-xbee 0.1.1

I just downloaded the zip file and placed in the .homeassistant folder

[Yoda-x]

Looks like that the zha integration component overwrites my libs. This came in lately.

You may try to ignore zha inside the discovery section of the configuration yaml

[Yoda-x]

Or go back to the latest 84.x

[marc-gist]

:( ok, thanks for your help.

[Yoda-x]

Either I need to rename my stuff to zha, overwriting the build-in or rename the zigpy and bellows modules.

I did not found an option to stop the original modules from loading or disable zha as it's now in the integrations

[alex-v-g]

I am using most recent hassio,HA and previous master build of zha_new. Still working ok. The only problem - to remember to do not try to adjust ZHA preferences in integrations, since this loads official zha with its libs. Current master build of zha_new did not work well. I could not pair few of my smart buttons, Decided to roll back to previous build. Still ok even after reboots/updates of HA.

[Yoda-x]

@alex-v-g can you open a ticket what is not working in my new master build?

I installed newest Hass on my laptop for testing. Once you clicked one time the integrations they get loaded on every restart. Not sure how to disable them. Even disabling the config option in configuration not solves it

[alex-v-g]

I will try to upgrade zha_new one more time and try it one more time when I have a bit more time. The problem is that it looks I need to delete zigbee.db and start pairing from scratch. I have now a lot of items to pair in order to check whether everything works ok or not. I have also one hue bulb and suppose that I will not be able to re-pair it to new zigbee.db without remote control (which I don't have). Do you have maybe a hint how to use original zigbee db with newest zha_new?

[Yoda-x]

@alex-v-g why you need to start with a fresh zigbee.db on newest master? I added new attributes to a table, but there is code to add needed columns to the table if they not exists..

Having a copy of the db is useful for a rollback. Possible problem is if you use skip-pip for Hass start, as you stick with old modules.

[alex-v-g]

@Yoda-x I thought your new master is based on preview which I used before, where I had to create new zigbee.db. I will try to update zigbee_new and use with my current zigbee.db (with backup of DB). Thanks for advice.

[Yoda-x]

Sorry, when you needed to delete the zigbee.db. That should never needed. Possible there were some bugs in preview.

[alex-v-g]

@Yoda-x : see https://github.com/Yoda-x/ha-zha-new/issues/63#issuecomment-445540040

maybe I misunderstood that time.

[Yoda-x]

Hi @alex-v-g, during this time it was needed. But as stated 2month ago, I was working on a patch and for the master release it was in.

[Yoda-x]

@marc-gist BTW, you can load manually my libs with pip in the venv and then run Hass with the - - skip-pip trigger. This prohibits Hass to update any modules

[estevez-dev]

Any solution to make zha_new work on HA 0.87.1?

[marc-gist]

I didn't have time to keep trying and don't really want to maintain a system that's more than a simple command outside the full version. I've setup zigbee2mqtt which so far is awesome

[Yoda-x]

I have my home system running on 0.87.1. If you never open the zha integration under configurations it should be ok. Please give it a try.

[estevez-dev]

Just realized that controller is in "failed" state. Reboot helped. All working fine now except movement sensor. I can't even remove and re-add it with permit. I'll try to update from master because I'm on preview and it was not updated since #71 By the way @marc-gist , zigbee2mqtt doesn't feels like a good replacement with all those usb sticks, strange hardware debuggers and custom firmware flashing )

[estevez-dev]

Oh, forgot to ask @Yoda-x . Does changes for resolving #71 already in master?

[alex-v-g]

Hello! I've tried few times to update zha_new to the latest version and every time something went wrong. Maybe it is somehow relates to this this issue or new home assistant version damaging something. Yesterday tried 0.88.b4 (official release was not in place) and at least I could see most of my devices online. The problem was that all devices got new names where "unknown" was written in the middle of the device id name. For example ikea bulbs got "unknown" instead of "tradfri" word. To repair all my automations I need to rename all the devices in script. Wanted to paste a log but log was really big and I decided to wait official 0.88. @Yoda-x : do you have an idea how "unknown" can appear in all devices' id names?

[Yoda-x]

@alex-v-g : sorry. can you please update it again, There is already a bugfix for this, but it was not used in zha_new. I now updated the needed module versions. this should be fixed after a refresh and reload

[Yoda-x]

@estevez-dev : yepp

[alex-v-g]

updated version still not working under HA 0.88.0 following log 019-02-21 17:58:03 WARNING (MainThread) [homeassistant.loader] Integrations need to be in their own folder. Change switch/zha_new.py to zha_new/switch.py. This will stop working soon. 2019-02-21 17:58:03 INFO (MainThread) [homeassistant.components.switch] Setting up switch.zha_new 2019-02-21 17:58:04 WARNING (MainThread) [homeassistant.loader] Integrations need to be in their own folder. Change switch/zha_new.py to zha_new/switch.py. This will stop working soon. 2019-02-21 17:58:04 INFO (MainThread) [homeassistant.components.switch] Setting up switch.zha_new 2019-02-21 17:58:04 WARNING (MainThread) [homeassistant.loader] Integrations need to be in their own folder. Change binary_sensor/zha_new.py to zha_new/binary_sensor.py. This will stop working soon. 2019-02-21 17:58:04 INFO (MainThread) [homeassistant.components.binary_sensor] Setting up binary_sensor.zha_new 2019-02-21 17:58:04 WARNING (MainThread) [homeassistant.loader] Integrations need to be in their own folder. Change binary_sensor/zha_new.py to zha_new/binary_sensor.py. This will stop working soon. 2019-02-21 17:58:04 INFO (MainThread) [homeassistant.components.binary_sensor] Setting up binary_sensor.zha_new 2019-02-21 17:58:04 INFO (MainThread) [homeassistant.loader] Loaded sensor.zha_new from custom_components.sensor.zha_new 2019-02-21 17:58:04 WARNING (MainThread) [homeassistant.loader] You are using a custom component for sensor.zha_new which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant. 2019-02-21 17:58:04 WARNING (MainThread) [homeassistant.loader] Integrations need to be in their own folder. Change sensor/zha_new.py to zha_new/sensor.py. This will stop working soon. 2019-02-21 17:58:04 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.zha_new 2019-02-21 17:58:04 WARNING (MainThread) [homeassistant.loader] Integrations need to be in their own folder. Change binary_sensor/zha_new.py to zha_new/binary_sensor.py. This will stop working soon.

[estevez-dev]

Ah, there was some directory structure changes for custom components!

[alex-v-g]

Forgot to mention that "unknown" are still there despite to updated master version. So still problem is there.

[estevez-dev]

Indeed there was changes in 0.88.0 that breaks custom components: https://community.home-assistant.io/t/custom-components-directory-structure-changed-with-0-88/100741

[Yoda-x]

@estevez-dev thanks for pointing to this upcoming change.

@alex-v-g what happens if you remove a device and then add it again? The device will only discovered new when its joins new.

[estevez-dev]

Made a PR for changes needed in HA 0.88.0. Tested, it works fine. https://github.com/Yoda-x/ha-zha-new/pull/75

@alex-v-g You can take it in my fork untill it is not merged if you want: https://github.com/estevez-dev/ha-zha-new/tree/ha_0_88_changes

[alex-v-g]

@estevez-dev : looks good... no more complains from HA in log file. Anyway "unknown" is still in device name...

@Yoda-x : For cleaner testing I've removed zigbee.db and rebooted Log following: 19-02-22 21:52:37 INFO (MainThread) [zigpy.application] Device 0xde23 (00:15:8d:00:01:eb:72:94) left the network 2019-02-22 21:52:38 INFO (MainThread) [zigpy.application] Device 0x44eb (00:15:8d:00:01:eb:72:94) joined the network via 19107 2019-02-22 21:52:38 INFO (MainThread) [zigpy.device] [0x44eb] [0x44eb] Discovering endpoints 2019-02-22 21:52:42 INFO (MainThread) [zigpy.device] [0x44eb] [0x44eb] Discovered endpoints: [1] 2019-02-22 21:52:42 INFO (MainThread) [zigpy.endpoint] [0x44eb:1] Discovering endpoint information 2019-02-22 21:52:43 WARNING (MainThread) [bellows.zigbee.application] 0x44eb:1:0x0000 Unexpected response TSN=4 command=266 args=b'\x05\x00B\x16lumi.sensor_switch.aq3' 2019-02-22 21:52:43 WARNING (MainThread) [zigpy.endpoint] [0x44eb:1] Message on unknown cluster 0x0000 2019-02-22 21:52:43 INFO (MainThread) [zigpy.endpoint] [0x44eb:1] Discovered endpoint information: <SimpleDescriptor endpoint=1 profile=260 device_type=24321 device_version=1 input_clusters=[0, 18, 6, 1] output_clusters=[0]> 2019-02-22 21:52:48 INFO (MainThread) [homeassistant.loader] Loaded switch from homeassistant.components.switch 2019-02-22 21:52:48 INFO (MainThread) [homeassistant.setup] Setting up switch 2019-02-22 21:52:48 INFO (MainThread) [homeassistant.setup] Setup of domain switch took 0.0 seconds. 2019-02-22 21:52:48 INFO (MainThread) [homeassistant.loader] Loaded zha_new.switch from custom_components.zha_new.switch 2019-02-22 21:52:48 WARNING (MainThread) [homeassistant.loader] You are using a custom component for zha_new.switch which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant. 2019-02-22 21:52:48 INFO (MainThread) [homeassistant.components.switch] Setting up switch.zha_new

resulted device ID became: "switch.lumi_unknown_01eb7294_1"

strange indeed....

updated: copied lumi_sensor_switch_aq3.py into lumi_sensor_swit.py (this button switch coming truncated sometimes... after reboot got name binary_sensor.lumi_lumi_sensor_swit_01eb7294_1_on_off my previous name for that, in outdated zha_new was: binary_sensor.lumi_lumisensor_swit_01eb7294_1_onoff so it is almost correct besides "" in name

[Yoda-x]

Thanks @estevez-dev

@alex-v-g can you enable also debug for custom_components? There we should see more in the logs for zha_new

[alex-v-g]

@Yoda-x I deleted zigbee.db and paired few things again. First was that button xxxx_swit which I mentioned before. This time given name was without "unknown". That was after I added swit.py in device folder - see my previous posts. Before that it was also "unknown".

Then I paired my xiaomi smart plug...got "unknown" in name. Attaching log here. Search for "switch.lumi_unknown_02674dc8". That name system gave.

example_log.zip

[Yoda-x]

Seems, the information is restored from the state-restore DB. Can you remove the device via the remove function (right below the permit). Removing the zigbee.db is not enough. The mac address of the device is used as key inside Hass to restore device information. A remove service call of the device should remove it from HASS.

Thanks

[alex-v-g]

I have followed your advice and made a remove service call. Was OK with button, I mentioned before. Tried my xiaomi weather sensor. It gave me before 4 sensors in zha_new: weather, humidity, pressure, temperature. This time only sensor "weather" changed name from "unknown" to normal one.The rest sensors kept "unknown" in names, though theystarted show values. So probably ZHA. REMOVE command removes only one of the sensors from Hass. Onother thing I noticed, command remove returned few times errors. I had to switch view in UI and back. Only this helped.

update: after reboot of hassio, button successfully added before, disappeared totally from the list of devices. update: new removing \adding do not help to get it back.

[Yoda-x]

can you provide log files?

[Yoda-x]

Ok, found that my xiaomi temp/humidity sensor just responds with "lumi.sens" on a attribute query. But pressing the button it sends lumi_sensor_ht. So I changed the logic to prefer this button presses. you may check if your weather sensor was same crippled as your button. Please try out the updated preview if it helps. And press the button every few seconds until the device shows up during pairing.

[alex-v-g]

Hello! I have tried new preview. From the first look - after remove - and pair again, then one sensor which was "weather" became unknown. Decided to make a hassio reboot for experiment: after reboot entity id still has "unknown" sensor.lumi_unknown_02322662_1_humidity while friendly name is OK , at least not " unknown". { "friendly_name": "LUMI lumi.weather_humidity", "model": "lumi.weather", "manufacturer": "LUMI", "lqi": null, "rssi": null, "last seen": null, "nwk": 1567, "path": "direct", "unit_of_measurement": "%", "device_class": "humidity" }

log was too long...found something what could be interesting: log short.zip

[Yoda-x]

seems, we are stuck somehow, can you please stop hass, delete the files ".homeassistant/.storage/core.entity_registry" ".homeassistant/.storage/core.restore_state" and start it again. Detection seems ok, but we need to get rid of the old entries

[alex-v-g]

After another hour of experiments I concluded: removing of only registry and state do not help, as after Hassio reboot, files creates automatically from zigbee.db. 019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] Loading application state from zigbee.db 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] load model: <class 'NoneType'> - None 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] get info from attributes <class 'bytes'> - b'lumi.weather' 2019-02-27 23:27:48 DEBUG (MainThread) [zigpy.appdb] Loading model state from 00:15:8d:00:02:32:26:62: None,LUMI same log for all other sensors....

from registry file: { "config_entry_id": null, "device_id": null, "disabled_by": null, "entity_id": "sensor.lumi_unknown_02322662_1_temperature", "name": null, "platform": "zha_new", "unique_id": "00:15:8d:00:02:32:26:62_1_temperature" }, from states file: { "last_seen": "2019-02-27T22:28:56.444333+00:00", "state": { "attributes": { "device_class": "temperature", "friendly_name": "LUMI unknown_temperature", "last seen": null, "lqi": null, "manufacturer": "LUMI", "model": "unknown", "nwk": 55643, "path": "direct", "rssi": null, "unit_of_measurement": "\u00b0C" }, It looks like HA reads information from the wrong place in DB?. This information loading into registry and state with "unknown" in the middle of the device ID. By some reason, during zha.remove command, registry and state do not clearing data for removed items (I believe because of different name in zigbee.db and registry\state). This makes troubles between HA and zha_new.

P.S. After I removed registry, state, zigbee.db, home-assistant_v2.db it looks OK. It seems that I will need to re-pair all my devices into new zigbee.db...

[Yoda-x]

Sorry for that. I checked the newer hass code. Seems there where many breaking changes, that now come mandatory. I suggest to stay on the current hass as it works now.

[Yoda-x]

tested under 0.92.1 .should be ok now