Implement new radio API

[puddly]

This change introduces a network backup/restore utility for any adapter implementing the new radio API:

• [x] zigpy: https://github.com/zigpy/zigpy/pull/848
• [x] zigpy-znp: https://github.com/zigpy/zigpy-znp/pull/97
• [x] zigpy-deconz: https://github.com/zigpy/zigpy-deconz/pull/172
• [x] bellows: https://github.com/zigpy/bellows/pull/445

The following adapters also implement the new radio API but their firmware does not allow much network information to be read, making this form of migration impossible:

• [x] zigpy-zigate: https://github.com/zigpy/zigpy-zigate/pull/117
• [x] zigpy-xbee: https://github.com/zigpy/zigpy-xbee/pull/123

---

Instructions

You'll need access to a Python environment. zigpy-znp has documentation that describes how to do this for every common platform.

# Make a new venv, these packages are heavily modified
virtualenv -p 3 venv  # Python 3.7 or above is required
# Or python3 -m venv venv (if `virtualenv` isn't installed)
source venv/bin/activate

# Install HA OS packages (only if using HA OS)
apk add gcc musl-dev libffi-dev

# Install zigpy-cli
pip install git+https://github.com/zigpy/zigpy-cli.git

# Back up the old radio.      Note: this backup format will change in the future.

# Conbee/Raspbee
zigpy -vv radio deconz /dev/serial/by-id/... backup deconz.json

# CC2531, CC2652, etc.
# zigpy -vv radio znp /dev/serial/by-id/... backup znp.json

# HUSBZB-1, EFR32, etc.
# zigpy -vv radio ezsp /dev/serial/by-id/... backup ezsp.json

# Restore a backup to the new one
zigpy -vv radio znp /dev/serial/by-id/... restore deconz.json
# zigpy -vv radio ezsp /dev/serial/by-id/... restore deconz.json
# zigpy -vv radio deconz /dev/serial/by-id/... restore deconz.json

# Leave the virtualenv
deactivate

Once that's done, edit /config/.storage/core.config_entries (take a backup of the file first!) and change the coordinator's path to the new radio's /dev/serial/by-id/... path. If you're switching to a CC2652, make sure to delete the flow_control key and set the baudrate to 115200.

[connoleg]

Hi, I managed to backup and restore from/to a ConBee II using zigpy. But when I try to restore to a Sonoff 3 plus I get the following exception:

zigpy -vv radio znp /dev/ttyUSB0 restore deconz.json
2022-01-12 17:51:04.390 parallels-Parallels-Virtual-Platform asyncio DEBUG Using selector: EpollSelector
Traceback (most recent call last):
  File "/home/parallels/.local/bin/zigpy", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1134, in invoke
    Command.invoke(self, ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/parallels/.local/lib/python3.8/site-packages/zigpy_cli/cli.py", line 19, in inner
    return loop.run_until_complete(cmd(*args, **kwargs))
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/parallels/.local/lib/python3.8/site-packages/zigpy_cli/radio.py", line 51, in radio
    app = app_cls(config)
TypeError: Can't instantiate abstract class ControllerApplication with abstract methods connect, disconnect, start_network

Any idea whats going wrong? Thanks

[puddly]

Make sure you're using a virtualenv. I believe pip install won't overwrite existing packages when you install from Git.

Or pip uninstall zigpy-znp, re-install it from git, then uninstall it again and install the normal version from PyPI.

[connoleg]

Thanks puddly, bang on the money! virtualenv did the trick. I hadnt figured I needed to run that way as I was not on my HomeAssistant box but a standalone ubuntu

Cheers

[toot]

I would also like to switch from a ConBee II to a Sonoff 3 plus with your tool, but the setup fails ...

I get the following errors after (I tried to) installed wheel

Failed building wheel for zigpy Failed building wheel for zigpy-cli Failed building wheel for zigpy-deconz Failed building wheel for bellows

But at the end: Successfully installed bellows-0.30.0.dev0 zigpy-0.41.0.dev0 zigpy-cli-0.0.1 zigpy-deconz-0.14.0.dev0 zigpy-znp-0.6.4

Did the installation work anyway?

Also I'm still struggeling with the next step:

Using "lsusb" I get this info about my conbee II: Bus 001 Device 003: ID 1cf1:0030 Dresden Elektronik

Where do I get the remaining information for the backup command? zigpy -vv radio deconz /dev/serial/by-id/... backup deconz.json

Sorry, I'm pretty new to this and I'm still learning... Thanks in advance :)

[puddly]

You can ignore the wheel errors, those are just saying that Python didn't cache .whl files so it'll have to rebuild the modules from scratch if you reinstall them.

Run ls /dev/serial/by-id/ to list the serial ports. You'll see which one corresponds to which device (you don't have to have them both plugged in at once).

[toot]

Thanks! :)

Now the backup command results in different errors, stating that some distributions are missing:

bellows 0.30.0.dev0 requires click-log>=0.2.1, which is not installed. bellows 0.30.0.dev0 requires pure_pcapy3==1.0.1, which is not installed. bellows 0.30.0.dev0 requires pyserial, which is not installed. bellows 0.30.0.dev0 requires pyserial-asyncio, which is not installed. bellows 0.30.0.dev0 requires voluptuous, which is not installed.

I already manually installed: scapy, click and coloredlogs, the current error is: The 'voluptuous' distribution was not found and is required by zigpy

But I tried installing python-voluptuous with no effect.

I'm afraid I missed something at the beginning.

[puddly]

This may be my mistake, I just updated the pip install command in the example. Instead of running:

$ pip install --upgrade --no-deps --force-reinstall ...

Try just running:

$ pip install ...

The dependencies should then be correctly installed.

[toot]

Thanks Putty, now everything works as intended :)

I had a friend of mine help me undo the changes I made yesterday and after that your guide worked very well.

Some feedback from him (to make it more fool proof): 1) Adjusting the variable of the device from "..." and adding some additional input:

# Use TAB-autocomplete to get your device YOUR_DEVICE=/dev/by-id/zigbeedevice zigpy -vv radio deconz /dev/serial/by-id/$YOUR_DEVICE backup deconz.json

2) Using another name (less generic) for the venv, to prevent hiccups if there is already another "venv"

But thanks alot again for this really useful tool :)

[puddly]

@toot @connoleg would either of you mind testing out something with your Conbee? They need to be relatively running recent firmware (mid 2020 or newer) for this to work:

CONBEE=/dev/serial/by-id/...

zigpy -vv radio deconz $CONBEE backup conbee1.json

# Edit `conbee1.json` and change `network_key.frame_counter` to something bigger by adding a `0` to the end

zigpy -vv radio deconz $CONBEE restore conbee1.json

# Unplug the Conbee and plug it back in (very important!)

zigpy -vv radio deconz $CONBEE backup conbee2.json

# Compare the two files
diff conbee1.json conbee2.json

The contents of conbee1.json and conbee2.json should be nearly identical and the frame counter value should have changed.

[stiang]

I’m trying to migrate from a Raspberry Pi with RaspBee II to a proper server with the ConBee II USB dongle instead. I believe I have followed all the proper steps from the top of this issue, but getting this error when trying to backup the RaspBee:

(venv) bash-5.1# zigpy -vv radio deconz /dev/serial/by-id/usb-0658_0200-if00 backup deconz.json
2022-01-29 12:04:58.068 homeassistant asyncio DEBUG Using selector: EpollSelector
2022-01-29 12:04:58.242 homeassistant zigpy.appdb DEBUG SQLite version for <module 'sqlite3' from '/usr/local/lib/python3.9/sqlite3/__init__.py'>: 3.34.1
2022-01-29 12:04:58.273 homeassistant zigpy_deconz.uart DEBUG Connection made
2022-01-29 12:04:58.274 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.protocol_version: 34>, b'')
2022-01-29 12:04:58.275 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a02000800010022
2022-01-29 12:05:00.079 homeassistant zigpy_deconz.api WARNING No response to 'Command.read_parameter' command with seq id '0x02'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/venv/bin/zigpy", line 8, in <module>
    sys.exit(cli())
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/config/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/config/venv/lib/python3.9/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/config/venv/lib/python3.9/site-packages/zigpy_cli/cli.py", line 19, in inner
    return loop.run_until_complete(cmd(*args, **kwargs))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/config/venv/lib/python3.9/site-packages/zigpy_cli/radio.py", line 90, in backup
    await app.connect()
  File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/zigbee/application.py", line 71, in connect
    self.version = await api.version()
  File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 470, in version
    (self._proto_ver,) = await self[NetworkParameter.protocol_version]
  File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 435, in read_parameter
    r = await self._command(Command.read_parameter, 1 + len(data), param, data)
  File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 320, in _command
    return await asyncio.wait_for(fut, timeout=COMMAND_TIMEOUT)
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

Am I doing something wrong? I’m not 100% sure that the by-id ID is the correct one, but it’s the only entry in that folder and I get the same result when I use /dev/ttyACM0, which I know is the RaspBee.

[puddly]

0658:0200 looks like it's for an Aeotec Z-Wave stick. Try using /dev/ttyAMA0.

[stiang]

Oof, so much for "which I know is the RaspBee"... Thanks, you are right, that looks like the correct device. And I get what appears to be promising output with no errors:

(venv) bash-5.1# zigpy -vv radio deconz /dev/ttyAMA0 backup deconz.json
2022-01-29 16:31:57.184 homeassistant asyncio DEBUG Using selector: EpollSelector
2022-01-29 16:31:57.360 homeassistant zigpy.appdb DEBUG SQLite version for <module 'sqlite3' from '/usr/local/lib/python3.9/sqlite3/__init__.py'>: 3.34.1
2022-01-29 16:31:57.387 homeassistant zigpy_deconz.uart DEBUG Connection made
2022-01-29 16:31:57.388 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.protocol_version: 34>, b'')
2022-01-29 16:31:57.389 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a02000800010022
2022-01-29 16:31:57.399 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a02000a000300220e01
2022-01-29 16:31:57.400 homeassistant zigpy_deconz.api DEBUG Read parameter protocol_version response: [270]
2022-01-29 16:31:57.400 homeassistant zigpy_deconz.api DEBUG Command Command.version (0,)
2022-01-29 16:31:57.401 homeassistant zigpy_deconz.uart DEBUG Send: 0x0d0300090000000000
2022-01-29 16:31:57.411 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0d0300090000076926
2022-01-29 16:31:57.411 homeassistant zigpy_deconz.api DEBUG Version response: 26690700
2022-01-29 16:31:57.414 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.mac_address: 1>, b'')
2022-01-29 16:31:57.414 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a04000800010001
2022-01-29 16:31:57.426 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a04001000090001fefd08ffff2e2100
2022-01-29 16:31:57.427 homeassistant zigpy_deconz.api DEBUG Read parameter mac_address response: [00:21:2e:ff:ff:08:fd:fe]
2022-01-29 16:31:57.428 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.aps_designed_coordinator: 9>, b'')
2022-01-29 16:31:57.429 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a05000800010009
2022-01-29 16:31:57.438 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0500090002000901
2022-01-29 16:31:57.439 homeassistant zigpy_deconz.api DEBUG Read parameter aps_designed_coordinator response: [1]
2022-01-29 16:31:57.440 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_address: 7>, b'')
2022-01-29 16:31:57.441 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a06000800010007
2022-01-29 16:31:57.451 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a06000a000300070000
2022-01-29 16:31:57.452 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_address response: [0x0000]
2022-01-29 16:31:57.453 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_panid: 5>, b'')
2022-01-29 16:31:57.453 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a07000800010005
2022-01-29 16:31:57.463 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a07000a000300054b32
2022-01-29 16:31:57.464 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_panid response: [0x324b]
2022-01-29 16:31:57.465 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.aps_extended_panid: 11>, b'')
2022-01-29 16:31:57.466 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0800080001000b
2022-01-29 16:31:57.477 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0800100009000b0000000000000000
2022-01-29 16:31:57.478 homeassistant zigpy_deconz.api DEBUG Read parameter aps_extended_panid response: [00:00:00:00:00:00:00:00]
2022-01-29 16:31:57.479 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_extended_panid: 8>, b'')
2022-01-29 16:31:57.480 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a09000800010008
2022-01-29 16:31:57.491 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a09001000090008fefd08ffff2e2100
2022-01-29 16:31:57.493 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_extended_panid response: [00:21:2e:ff:ff:08:fd:fe]
2022-01-29 16:31:57.493 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.current_channel: 28>, b'')
2022-01-29 16:31:57.494 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0a00080001001c
2022-01-29 16:31:57.504 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0a00090002001c0f
2022-01-29 16:31:57.505 homeassistant zigpy_deconz.api DEBUG Read parameter current_channel response: [15]
2022-01-29 16:31:57.505 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.channel_mask: 10>, b'')
2022-01-29 16:31:57.506 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0b00080001000a
2022-01-29 16:31:57.515 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0b000c0005000a00800000
2022-01-29 16:31:57.516 homeassistant zigpy_deconz.api DEBUG Read parameter channel_mask response: [<Channels.CHANNEL_15: 32768>]
2022-01-29 16:31:57.517 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_update_id: 36>, b'')
2022-01-29 16:31:57.518 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0c000800010024
2022-01-29 16:31:57.527 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0c00090002002400
2022-01-29 16:31:57.528 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_update_id response: [0]
2022-01-29 16:31:57.529 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (2, <NetworkParameter.network_key: 24>, b'\x00')
2022-01-29 16:31:57.530 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0d00090002001800
2022-01-29 16:31:57.544 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0d00190012001800af912adb53c0fe5b478542134b7796c9
2022-01-29 16:31:57.546 homeassistant zigpy_deconz.api DEBUG Read parameter network_key response: [0, [175, 145, 42, 219, 83, 192, 254, 91, 71, 133, 66, 19, 75, 119, 150, 201]]
2022-01-29 16:31:57.546 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_frame_counter: 39>, b'')
2022-01-29 16:31:57.547 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0e000800010027
2022-01-29 16:31:57.557 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0e000c00050027d6a60200
2022-01-29 16:31:57.558 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_frame_counter response: [173782]
2022-01-29 16:31:57.559 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.trust_center_address: 14>, b'')
2022-01-29 16:31:57.560 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0f00080001000e
2022-01-29 16:31:57.571 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0f00100009000efefd08ffff2e2100
2022-01-29 16:31:57.572 homeassistant zigpy_deconz.api DEBUG Read parameter trust_center_address response: [00:21:2e:ff:ff:08:fd:fe]
2022-01-29 16:31:57.573 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (9, <NetworkParameter.link_key: 25>, b'\xfe\xfd\x08\xff\xff.!\x00')
2022-01-29 16:31:57.574 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a10001000090019fefd08ffff2e2100
2022-01-29 16:31:57.591 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a10002000190019fefd08ffff2e21005a6967426565416c6c69616e63653039
2022-01-29 16:31:57.593 homeassistant zigpy_deconz.api DEBUG Read parameter link_key response: [00:21:2e:ff:ff:08:fd:fe, [90, 105, 103, 66, 101, 101, 65, 108, 108, 105, 97, 110, 99, 101, 48, 57]]
2022-01-29 16:31:57.594 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.security_mode: 16>, b'')
2022-01-29 16:31:57.595 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a11000800010010
2022-01-29 16:31:57.604 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a1100090002001003
2022-01-29 16:31:57.605 homeassistant zigpy_deconz.api DEBUG Read parameter security_mode response: [3]
2022-01-29 16:31:57.615 homeassistant zigpy_deconz.api DEBUG Command Command.write_parameter (5, <NetworkParameter.watchdog_ttl: 38>, b'\x00\x00\x00\x00')
2022-01-29 16:31:57.616 homeassistant zigpy_deconz.uart DEBUG Send: 0x0b12000c0005002600000000
2022-01-29 16:31:57.627 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0b12000800010026
2022-01-29 16:31:57.627 homeassistant zigpy_deconz.api DEBUG Write parameter watchdog_ttl: SUCCESS
2022-01-29 16:31:57.630 homeassistant zigpy_deconz.uart DEBUG Closed serial connection

However, the backup file is very short and shows no devices ("devices": []):

{
    "metadata": {
        "version": 1,
        "format": "zigpy/open-coordinator-backup",
        "source": "zigpy-cli@0.0.1",
        "internal": {
            "node": {
                "ieee": "00212effff08fdfe",
                "nwk": "0000",
                "type": "coordinator"
            },
            "network": {
                "tc_link_key": {
                    "key": "5a69674265________69616e63653039",
                    "frame_counter": 0
                },
                "tc_address": "00212effff08fdfe",
                "nwk_manager": "0000"
            },
            "link_key_seqs": {}
        }
    },
    "stack_specific": {},
    "coordinator_ieee": "00212effff08fdfe",
    "pan_id": "324b",
    "extended_pan_id": "00212effff08fdfe",
    "nwk_update_id": 0,
    "security_level": 5,
    "channel": 15,
    "channel_mask": [
        15
    ],
    "network_key": {
        "key": "af912adb53c0fe________134b7796c9",
        "sequence_number": 0,
        "frame_counter": 173782
    },
    "devices": []
}

In HA I have 14 devices and 30 entities, which I successfully control via the RaspBee, so something’s still off:

Any ideas what might cause the device list to be empty? I have disabled zha in HA while doing this. If I try to enable it I get different errors when trying to create the backup (about possible simultaneous access etc., which makes sense).

[puddly]

The coordinator normally doesn't keep track of any devices that aren't its direct children, but even that information isn't critical. The Conbee however expects the program controlling it to load in the child table at runtime since it'll be stored in the program's database. This is normal.

After you restore to the Conbee, can you unplug it, plug it back in, and take a second backup of it and post it here?

[stiang]

@puddly Sorry if I’m being dense, but are you saying that the devices don’t need to be listed in the backup file, and are instead "mapped to" the RaspBee/ConBee through other means that are saved in the HA config/storage files? In other words, the zigbee backup file I have now should actually get me the devices I need if paired with a full HA backup?

I’m also unclear on what’s the best procedure to do the restore. Should I plug the ConBee into the Pi and do the restore there, then move the already restored ConBee to my Ubuntu server? Or plug the ConBee into the Ubuntu server, copy the backup file and then do the restore from there? Or doesn’t it matter?

I’ll post the info you want once I have successfully performed the restore. Thanks for helping out, really appreciate it!

[puddly]

Correct, there will be no devices if you're using a Raspbee or a Conbee.

It doesn't matter where you do the restore. Since you already have this virtual environment up and running, it's easiest to do it wherever you did the Raspbee backup.

[stiang]

@puddly Sweet, looks like I was able to restore to the ConBee now. Will try to get it up and running on the new server shortly. Thanks again!

I followed the instructions you gave a few comments back about increasing the frame_counter value before restoring, then creating another backup. As far as I can see the value was reset, but that was all:

(venv) bash-5.1# diff conbee1.json conbee2.json
--- conbee1.json
+++ conbee2.json
@@ -33,7 +33,7 @@
     "network_key": {
         "key": "af912adb53c0fe5b478542134b7796c9",
         "sequence_number": 0,
-        "frame_counter": 1
+        "frame_counter": 0
     },
     "devices": []
 }
\ No newline at end of file
(venv) bash-5.1#

Let me know if your need anything else!

[puddly]

Looks like your Conbee is running an old firmware and doesn't support setting the frame counter. The one in your Raspbee backup was 173782 but the value read from the Conbee was 0.

Can you upgrade it to the latest firmware and re-run the migration from the Raspbee again? Some devices are strict about the frame counter rolling back and may not respond to commands sent from the Conbee.

[toot]

@toot @connoleg would either of you mind testing out something with your Conbee? They need to be relatively running recent firmware (mid 2020 or newer) for this to work:

The contents of conbee1.json and conbee2.json should be nearly identical and the frame counter value should have changed.

Unfortunately I've got my Conbee not around at the moment, I'm trying to convince a friend from HA and borrowed him mine :D

[connoleg]

I'm away for the weekend, I'll take a look when back.krOn 31 Jan 2022 08:24, toot @.***> wrote:

@toot @connoleg would either of you mind testing out something with your Conbee? They need to be relatively running recent firmware (mid 2020 or newer) for this to work: The contents of conbee1.json and conbee2.json should be nearly identical and the frame counter value should have changed.

Unfortunately I've got my Conbee not around at the moment, I'm trying to convince a friend from HA and borrowed him mine :D

—Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>

[mliudvikas]

Confirming to have successfully migrated from 1st gen Conbee stick (with latest fw but which was misbehaving on ZHA) to the Sonoff Zigbee 3.0 USB stick with no issues.

[balcanuc]

I'm getting similar timeout related error as @stiang

My Setup I'm running Hassio on a Raspberry PI 4, 4G.

I use a ZHA dongle with following properties:

CC2531, Z-Stack Home 1.2 (build 20180507)
by Texas Instruments
Connected via Texas Instruments CC2531, Z-Stack Home 1.2 (build 20180507)
Zigbee info
IEEE: 00:12:4b:00:1c:cc:a8:b1
Nwk: 0x0000
Device Type: Coordinator
LQI: Unknown
RSSI: Unknown
Last Seen: 2022-02-17T14:14:28
Power Source: Mains

How I get the zigpy binary

In order to be able to run commands above I had to run a new docker container on my RP4. I've used command below.

docker run --privileged -d --name zigpy \
--mount type=bind,source=/mnt/data/supervisor/homeassistant,target=/config \
 -v /dev:/dev  python:3.9-buster tail /dev/null -f

In that container I've run commands

pip install \
    git+https://github.com/puddly/zigpy.git@puddly/new-radio-settings-api \
    git+https://github.com/puddly/zigpy-cli.git@puddly/zigpy-radio-api \
    git+https://github.com/puddly/zigpy-znp.git@puddly/new-radio-settings-api \
    git+https://github.com/puddly/zigpy-deconz.git@puddly/new-radio-api \
    git+https://github.com/puddly/bellows.git@puddly/zigpy-radio-api

Commands I've run

Fist, I've deactivated the ZHA integration

Next I've run command below

zigpy -vv radio deconz /dev/ttyACM0 backup deconz.json

The command fails with the output below. I've run it multiple times but each time it failed in the same way. I've also run it with /dev/serial/by-id/usb-Texas_Instruments_TI_CC2531_USB_CDC___0X00124B001CCCA8B1-if00 instead of /dev/ttyACM0 with same results. Deserves mentioning that after it shows Send: 0x0a02000800010022 it makes a pause of about 1 second after that it prints the rest of the text.

Am I doing something wrong? Should a timeout be increased?

2022-02-17 20:29:43.183 1e36778fa6b6 asyncio DEBUG Using selector: EpollSelector
2022-02-17 20:29:43.594 1e36778fa6b6 zigpy.appdb DEBUG SQLite version for <module 'sqlite3' from '/usr/local/lib/python3.9/sqlite3/__init__.py'>: 3.27.2
2022-02-17 20:29:43.645 1e36778fa6b6 zigpy_deconz.uart DEBUG Connection made
2022-02-17 20:29:43.647 1e36778fa6b6 zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.protocol_version: 34>, b'')
2022-02-17 20:29:43.648 1e36778fa6b6 zigpy_deconz.uart DEBUG Send: 0x0a02000800010022
2022-02-17 20:29:45.452 1e36778fa6b6 zigpy_deconz.api WARNING No response to 'Command.read_parameter' command with seq id '0x02'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 490, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/zigpy3/venv/bin/zigpy", line 8, in <module>
    sys.exit(cli())
  File "/config/zigpy3/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/config/zigpy3/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/config/zigpy3/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/config/zigpy3/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/config/zigpy3/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/config/zigpy3/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/config/zigpy3/venv/lib/python3.9/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/config/zigpy3/venv/lib/python3.9/site-packages/zigpy_cli/cli.py", line 19, in inner
    return loop.run_until_complete(cmd(*args, **kwargs))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/config/zigpy3/venv/lib/python3.9/site-packages/zigpy_cli/radio.py", line 90, in backup
    await app.connect()
  File "/config/zigpy3/venv/lib/python3.9/site-packages/zigpy_deconz/zigbee/application.py", line 71, in connect
    self.version = await api.version()
  File "/config/zigpy3/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 470, in version
    (self._proto_ver,) = await self[NetworkParameter.protocol_version]
  File "/config/zigpy3/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 435, in read_parameter
    r = await self._command(Command.read_parameter, 1 + len(data), param, data)
  File "/config/zigpy3/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 320, in _command
    return await asyncio.wait_for(fut, timeout=COMMAND_TIMEOUT)
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

Update I was able to successfully run after I've changed the command line to the one below. Key difference here is use of radio znp instead of radio deconz

zigpy -vv radio znp /dev/ttyACM0 backup znp.json

The output file seems to contain 8 device entries. This match with my ha config that shows that are 9 (it includes the dongle).

Update2 I successfully restored the configuration on a SONOFF Zigbee 3.0 USB Dongle Plus(https://itead.cc/product/sonoff-zigbee-3-0-usb-dongle-plus/) using command below

zigpy -vv radio znp /dev/ttyACM0 restore znp.json

Next, I've modified the home assistant file /config/.storage/core.config_entries as described on https://www.youtube.com/watch?v=ql58ZfXso1M (after I've made a backup). Next, I've restarted home-assistant

[finity69x2]

dumb question...

in the command examples above you use:

"zigpy -vv radio ezsp /dev/serial/by-id/... backup ezsp.json"

is that supposed to be used explicitly and exactly like that at the "/dev/serial/by-id/..." part or can I use the device address of my zigbee stick? I have a udev rule setting the dev path to "/dev/zigbee" so can i use that instead?

IOW, would this be OK:

"zigpy -vv radio ezsp /dev/zigbee backup ezsp.json"

and when the file is created where (what path/location) would it write that to?

I haven't used a venv in linux for a long time since I switched HA from a venv to a Docker container and I'm a bit rusty on the details.

I won't be restoring to another stick (hopefully ever...) but I would just like a backup just in case so I need to store the backup file somewhere. I don't care where the file is created as long as I know where it is to copy it to a safe place for later use.

[stephan48]

Hi, I am trying to go migrate from a conbee2 with ZHA to a TubeSZ cc2652p2 - the export worked nicely but sadly the import seems to fail with the error that the network can not be formed: RuntimeError: Network formation refused, RF environment is likely too noisy. Temporarily unscrew the antenna or shield the coordinator with metal until a network is formed.

Can you give me a hint where i might start debugging?

It does not matter if i try to import the deconz backup or a backup i made after spinning up a Z2M instance. The files look very similar.

[JMan7777]

Hello,

I'm trying to migrate my ZHA coordinator from a Nortek HUSBZB-1 stick (Firmware 6.7.8) to a SONOFF ZigBee 3.0 USB Dongle Plus (Firmware CC1352P2_CC2652P_launchpad_coordinator_20220219).

I'm using a Virtual Machine on my NAS to do the migration (Ubuntu Linux) and the backup step works totally fine (no errors).

However the restore of the backup file I created (I named it zigbee-ezsp2.json) fails with some error.

Thx a lot for your help.

(venv) root@linux:~# zigpy -vv radio znp /dev/ttyUSB2 restore zigbee-ezsp2.json
2022-02-25 00:39:19.613 linux asyncio DEBUG Using selector: EpollSelector
2022-02-25 00:39:19.977 linux zigpy.appdb DEBUG SQLite version for <module 'sqlite3' from '/usr/lib/python3.8/sqlite3/__init__.py'>: 3.31.1
2022-02-25 00:39:20.220 linux zigpy_znp.uart DEBUG Connecting to /dev/ttyUSB2 at 115200 baud
2022-02-25 00:39:20.273 linux zigpy_znp.uart DEBUG Opened /dev/ttyUSB2 serial port
2022-02-25 00:39:20.275 linux zigpy_znp.uart DEBUG Connected to /dev/ttyUSB2 at 115200 baud
2022-02-25 00:39:20.279 linux zigpy_znp.api DEBUG Toggling RTS/DTR pins to skip bootloader or reset chip
2022-02-25 00:39:20.280 linux zigpy_znp.uart DEBUG Setting serial pin states: DTR=False, RTS=False
2022-02-25 00:39:20.442 linux zigpy_znp.uart DEBUG Setting serial pin states: DTR=False, RTS=True
2022-02-25 00:39:20.617 linux zigpy_znp.uart DEBUG Setting serial pin states: DTR=False, RTS=False
2022-02-25 00:39:20.803 linux zigpy_znp.api DEBUG Sending request: SYS.Ping.Req()
2022-02-25 00:39:21.300 linux zigpy_znp.api DEBUG Sending CC253x bootloader skip bytes
2022-02-25 00:39:23.053 linux zigpy_znp.api DEBUG Received command: SYS.ResetInd.Callback(Reason=<ResetReason.PowerUp: 0>, TransportRev=2, ProductId=1, MajorRel=2, MinorRel=7, MaintRel=1)
2022-02-25 00:39:23.057 linux zigpy_znp.api DEBUG Radio is alive: SYS.ResetInd.Callback(Reason=<ResetReason.PowerUp: 0>, TransportRev=2, ProductId=1, MajorRel=2, MinorRel=7, MaintRel=1)
2022-02-25 00:39:23.058 linux zigpy_znp.api DEBUG Giving ping task 0.50s to finish
2022-02-25 00:39:23.562 linux zigpy_znp.api DEBUG Sending request: SYS.Ping.Req()
2022-02-25 00:39:23.575 linux zigpy_znp.api DEBUG Received command: SYS.Ping.Rsp(Capabilities=<MTCapabilities.APP_CNF|GP|UTIL|ZDO|AF|SYS: 1625>)
2022-02-25 00:39:23.578 linux zigpy_znp.nvram DEBUG Detecting struct alignment
2022-02-25 00:39:23.579 linux zigpy_znp.api DEBUG Sending request: UTIL.AssocFindDevice.Req(Index=0)
2022-02-25 00:39:23.592 linux zigpy_znp.api DEBUG Received command: UTIL.AssocFindDevice.Rsp(Device=b'\xFE\xFF\x00\x20\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF')
2022-02-25 00:39:23.595 linux zigpy_znp.nvram DEBUG Detected struct alignment: True
2022-02-25 00:39:23.597 linux zigpy_znp.api DEBUG Sending request: SYS.NVLength.Req(SysId=<NvSysIds.ZSTACK: 1>, ItemId=<ExNvIds.TCLK_TABLE: 4>, SubId=0)
2022-02-25 00:39:23.608 linux zigpy_znp.api DEBUG Received command: SYS.NVLength.Rsp(Length=20)
2022-02-25 00:39:23.611 linux zigpy_znp.api DEBUG Sending request: SYS.NVRead.Req(SysId=<NvSysIds.ZSTACK: 1>, ItemId=<ExNvIds.TCLK_TABLE: 4>, SubId=0, Offset=0, Length=20)
2022-02-25 00:39:23.624 linux zigpy_znp.api DEBUG Received command: SYS.NVRead.Rsp(Status=<Status.SUCCESS: 0>, Value=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xFF\x00\x00\x00')
2022-02-25 00:39:23.626 linux zigpy_znp.nvram DEBUG Read NVRAM[NvSysIds.ZSTACK][ExNvIds.TCLK_TABLE][0x0000] = b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xFF\x00\x00\x00'
2022-02-25 00:39:23.627 linux zigpy_znp.api DEBUG Detected Z-Stack 3.3
2022-02-25 00:39:23.629 linux zigpy_znp.api DEBUG Connected to /dev/ttyUSB2 at 115200 baud
2022-02-25 00:39:23.637 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVLength.Req(Id=<OsalNvIds.HAS_CONFIGURED_ZSTACK1: 3840>)
2022-02-25 00:39:23.645 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVLength.Rsp(ItemLen=0)
2022-02-25 00:39:23.648 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVLength.Req(Id=<OsalNvIds.HAS_CONFIGURED_ZSTACK3: 96>)
2022-02-25 00:39:23.663 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVLength.Rsp(ItemLen=0)
2022-02-25 00:39:23.667 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVLength.Req(Id=<OsalNvIds.ZIGPY_ZNP_MIGRATION_ID: 95>)
2022-02-25 00:39:23.683 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVLength.Rsp(ItemLen=0)
2022-02-25 00:39:23.687 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVLength.Req(Id=<OsalNvIds.BDBNODEISONANETWORK: 85>)
2022-02-25 00:39:23.699 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVLength.Rsp(ItemLen=1)
2022-02-25 00:39:23.702 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVDelete.Req(Id=<OsalNvIds.BDBNODEISONANETWORK: 85>, ItemLen=1)
2022-02-25 00:39:23.724 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVDelete.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-25 00:39:23.727 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVLength.Req(Id=<OsalNvIds.STARTUP_OPTION: 3>)
2022-02-25 00:39:23.744 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVLength.Rsp(ItemLen=1)
2022-02-25 00:39:23.745 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVWriteExt.Req(Id=<OsalNvIds.STARTUP_OPTION: 3>, Offset=0, Value=b'\x03')
2022-02-25 00:39:23.762 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVWriteExt.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-25 00:39:23.765 linux zigpy_znp.api DEBUG Sending request: SYS.ResetReq.Req(Type=<ResetType.Soft: 1>)
2022-02-25 00:39:23.767 linux zigpy_znp.api DEBUG Request has no response, not waiting for one.
2022-02-25 00:39:25.567 linux zigpy_znp.api DEBUG Received command: SYS.ResetInd.Callback(Reason=<ResetReason.PowerUp: 0>, TransportRev=2, ProductId=1, MajorRel=2, MinorRel=7, MaintRel=1)
2022-02-25 00:39:25.571 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVLength.Req(Id=<OsalNvIds.PANID: 131>)
2022-02-25 00:39:25.583 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVLength.Rsp(ItemLen=2)
2022-02-25 00:39:25.587 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVWriteExt.Req(Id=<OsalNvIds.PANID: 131>, Offset=0, Value=b'\xFF\xFF')
2022-02-25 00:39:25.601 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVWriteExt.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-25 00:39:25.603 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVLength.Req(Id=<OsalNvIds.APS_USE_EXT_PANID: 71>)
2022-02-25 00:39:25.612 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVLength.Rsp(ItemLen=8)
2022-02-25 00:39:25.613 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVWriteExt.Req(Id=<OsalNvIds.APS_USE_EXT_PANID: 71>, Offset=0, Value=b'\xC3\x2F\xF4\x0F\xF6\x35\xE5\xA8')
2022-02-25 00:39:25.629 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVWriteExt.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-25 00:39:25.631 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVLength.Req(Id=<OsalNvIds.PRECFGKEY: 98>)
2022-02-25 00:39:25.641 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVLength.Rsp(ItemLen=16)
2022-02-25 00:39:25.642 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVWriteExt.Req(Id=<OsalNvIds.PRECFGKEY: 98>, Offset=0, Value=b'\x65\x26\x1E\x95\x73\x9C\xC8\x5E\x8F\x39\x11\x0D\xB7\x3C\x11\xE6')
2022-02-25 00:39:25.657 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVWriteExt.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-25 00:39:25.659 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVLength.Req(Id=<OsalNvIds.PRECFGKEYS_ENABLE: 99>)
2022-02-25 00:39:25.670 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVLength.Rsp(ItemLen=1)
2022-02-25 00:39:25.672 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVWriteExt.Req(Id=<OsalNvIds.PRECFGKEYS_ENABLE: 99>, Offset=0, Value=b'\x00')
2022-02-25 00:39:25.686 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVWriteExt.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-25 00:39:25.688 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVLength.Req(Id=<OsalNvIds.CHANLIST: 132>)
2022-02-25 00:39:25.700 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVLength.Rsp(ItemLen=4)
2022-02-25 00:39:25.702 linux zigpy_znp.api DEBUG Sending request: SYS.OSALNVWriteExt.Req(Id=<OsalNvIds.CHANLIST: 132>, Offset=0, Value=b'\x00\x80\x10\x02')
2022-02-25 00:39:25.716 linux zigpy_znp.api DEBUG Received command: SYS.OSALNVWriteExt.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-25 00:39:25.719 linux zigpy_znp.api DEBUG Sending request: AppConfig.BDBSetChannel.Req(IsPrimary=<Bool.true: 1>, Channel=<Channels.CHANNEL_25|CHANNEL_20|CHANNEL_15: 34635776>)
2022-02-25 00:39:25.729 linux zigpy_znp.api DEBUG Received command: AppConfig.BDBSetChannel.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-25 00:39:25.732 linux zigpy_znp.api DEBUG Sending request: AppConfig.BDBSetChannel.Req(IsPrimary=<Bool.false: 0>, Channel=<Channels.NO_CHANNELS: 0>)
2022-02-25 00:39:25.740 linux zigpy_znp.api DEBUG Received command: AppConfig.BDBSetChannel.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-25 00:39:25.742 linux zigpy_znp.api DEBUG Forming temporary network
2022-02-25 00:39:25.744 linux zigpy_znp.api DEBUG Sending request: AppConfig.BDBStartCommissioning.Req(Mode=<BDBCommissioningMode.NwkFormation: 4>)
2022-02-25 00:39:28.977 linux zigpy_znp.api DEBUG Received command: AppConfig.BDBStartCommissioning.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-25 00:39:28.979 linux zigpy_znp.api DEBUG Received command: AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.InProgress: 1>, Mode=<BDBCommissioningMode.NwkSteering: 2>, RemainingModes=<BDBCommissioningMode.NwkFormation: 4>)
2022-02-25 00:39:29.252 linux zigpy_znp.api DEBUG Received command: ZDO.StateChangeInd.Callback(State=<DeviceState.StartingAsCoordinator: 8>)
2022-02-25 00:39:30.965 linux zigpy_znp.api DEBUG Received command: ZDO.StateChangeInd.Callback(State=<DeviceState.StartedAsCoordinator: 9>)
2022-02-25 00:39:30.967 linux zigpy_znp.api DEBUG Received command: AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.Success: 0>, Mode=<BDBCommissioningMode.NwkSteering: 2>, RemainingModes=<BDBCommissioningMode.NONE: 0>)
2022-02-25 00:39:30.971 linux zigpy_znp.api DEBUG Sending request: SYS.ResetReq.Req(Type=<ResetType.Soft: 1>)
2022-02-25 00:39:30.971 linux zigpy_znp.api DEBUG Request has no response, not waiting for one.
2022-02-25 00:39:34.027 linux zigpy_znp.api DEBUG Received command: SYS.ResetInd.Callback(Reason=<ResetReason.PowerUp: 0>, TransportRev=2, ProductId=1, MajorRel=2, MinorRel=7, MaintRel=1)
2022-02-25 00:39:34.030 linux zigpy_znp.uart DEBUG Closing serial port
Traceback (most recent call last):
  File "/root/venv/bin/zigpy", line 8, in <module>
    sys.exit(cli())
  File "/root/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/root/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/root/venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/root/venv/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/root/venv/lib/python3.8/site-packages/zigpy_cli/cli.py", line 19, in inner
    return loop.run_until_complete(cmd(*args, **kwargs))
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/root/venv/lib/python3.8/site-packages/zigpy_cli/radio.py", line 113, in restore
    await app.write_network_info(network_info=network_info, node_info=node_info)
  File "/root/venv/lib/python3.8/site-packages/zigpy_znp/zigbee/application.py", line 168, in write_network_info
    return await self._znp.write_network_info(
  File "/root/venv/lib/python3.8/site-packages/zigpy_znp/api.py", line 355, in write_network_info
    await self.start_network()
  File "/root/venv/lib/python3.8/site-packages/zigpy_znp/api.py", line 257, in start_network
    raise RuntimeError(
RuntimeError: Network formation failed: AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.Success: 0>, Mode=<BDBCommissioningMode.NwkSteering: 2>, RemainingModes=<BDBCommissioningMode.NONE: 0>)

[puddly]

@JMan7777 Try uninstalling and re-installing zigpy-znp, I just pushed a change that should fix the problem:

$ pip uninstall zigpy-znp
$ pip install git+https://github.com/puddly/zigpy-znp.git@puddly/new-radio-settings-api

[JMan7777]

@JMan7777 Try uninstalling and re-installing zigpy-znp, I just pushed a change that should fix the problem:

$ pip uninstall zigpy-znp
$ pip install git+https://github.com/puddly/zigpy-znp.git@puddly/new-radio-settings-api

@puddly Thx for the quick fix. It worked. Home Assistant is now working fine with the Sonoff stick and I see my Zigbee devices slowly getting connected. The backup/restore saved me a ton of time with over 60 Zigbee devices in the house :).

[anoril1979]

Hello! I am having troubles trying to make use of this tool for backuping a Sonoff Bridge and migrating to Sonoff Zigbee 3.0+. First of all: I'm working on Windows as my Linux refuses to upgrade SQLite3 to an acceptable version. Note on Windows, another problem comes as zigpy-cli module is not found despite it is listed with "pip list". If I manage to solve these issues, I would like to know:

• how do I get the address of the device to backup? Where does "/dev/..." come from? How to get similar address on Windows platform? Can I make use of the "socket://[ip-address]:[port]" form I use in ZHA?

Many thanks!

[puddly]

First of all: I'm working on Windows as my Linux refuses to upgrade SQLite3 to an acceptable version.

Are you seeing this runtime error? You should be able to pip install pysqlite-binary if you cannot upgrade your system SQLite version.

Note on Windows, another problem comes as zigpy-cli module is not found despite it is listed with "pip list".

How exactly are you installing these packages? The instructions above should be compatible with Windows if you have git installed, you'll just have to use Windows-specific commands to enable the virtualenv: https://github.com/zigpy/zigpy-znp/blob/dev/TOOLS.md#creating-a-virtualenv-recommended

how do I get the address of the device to backup?

Windows uses COM ports, like COM3 instead of /dev/serial/by-id/example-device-id. You can see them in Device Manager.

Can I make use of the "socket://[ip-address]:[port]" form I use in ZHA?

That would work too as long as nothing else is connected to the same serial port (i.e. ZHA).

[stephan48]

Hi,

i am still getting this error, the dump i tried to apply a fresh dump from a fresh z2m instance which successfully accepted one device(convincing me that the radio is OK)

I can't pull a "info" because the device is "not part of a network" but it is a TubeSZ cc2652p2 with the last flashed firmware from z2m as per TubeSZ recommendation.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/root/zha-backup/venv/bin/zigpy", line 8, in sys.exit(cli()) File "/root/zha-backup/venv/lib/python3.7/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/root/zha-backup/venv/lib/python3.7/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/root/zha-backup/venv/lib/python3.7/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/root/zha-backup/venv/lib/python3.7/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/root/zha-backup/venv/lib/python3.7/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/root/zha-backup/venv/lib/python3.7/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/root/zha-backup/venv/lib/python3.7/site-packages/click/decorators.py", line 38, in new_func return f(get_current_context().obj, *args, *kwargs) File "/root/zha-backup/venv/lib/python3.7/site-packages/zigpy_cli/cli.py", line 19, in inner return loop.run_until_complete(cmd(args, kwargs)) File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete return future.result() File "/root/zha-backup/venv/lib/python3.7/site-packages/zigpy_cli/radio.py", line 113, in restore await app.write_network_info(network_info=network_info, node_info=node_info) File "/root/zha-backup/venv/lib/python3.7/site-packages/zigpy_znp/zigbee/application.py", line 169, in write_network_info network_info=network_info, node_info=node_info File "/root/zha-backup/venv/lib/python3.7/site-packages/zigpy_znp/api.py", line 356, in write_network_info await self.start_network() File "/root/zha-backup/venv/lib/python3.7/site-packages/zigpy_znp/api.py", line 281, in start_network ) from e RuntimeError: Network formation refused, RF environment is likely too noisy. Temporarily unscrew the antenna or shield the coordinator with metal until a network is formed.

This is the dump and the full log:

zigpy-cli-restore.txt dump.json.txt

I tried both removing the antenna and disconnecting the pigtail.

Kind Regards, Stephan

[puddly]

It seems like your radio is getting stuck during formation.

Have you tried physically powering it down and then powering it back on? If that doesn't help, can you take an NVRAM backup, erase it, then try to restore?

$ TUBE="socket://tube-zb-gw-cc2652p2.fritz.box:6638"
$ python -m zigpy_znp.tools.nvram_read $TUBE -o nvram.json
$ python -m zigpy_znp.tools.nvram_reset $TUBE

If this fixes it, attach nvram.json so that I can try replicating your environment locally and hopefully fix this problem automatically.

[stephan48]

Hi,

thank you, i tried:

• nvme read/reset + restore - the log is attached + nvme pre reset, post reset and post restore nvram-202202252216.json.txt nvram-202202252216-post-failed-restore.json.txt nvram-202202252216-post-reset.json.txt zigpy-cli-restore2.txt
• a nvme reset followed by a 2 min physicall poweroff + restore: Listener(matching_commands=(ZDO.StateChangeInd.Callback(State=<DeviceState.StartingAsCoordinator: 8>), AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.InProgress: 1>, Mode=None, RemainingModes=None)), callback=<bound method Queue.put_nowait of <Queue at 0xf5736150 maxsize=0 _queue=[AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.InProgress: 1>, Mode=<BDBCommissioningMode.NwkSteering: 2>, RemainingModes=<BDBCommissioningMode.NwkFormation: 4>)] tasks=1>>) 2022-02-25 22:27:21.182 rpi-sj zigpy_znp.api TRACE AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.InProgress: 1>, Mode=<BDBCommissioningMode.NwkSteering: 2>, RemainingModes=<BDBCommissioningMode.NwkFormation: 4>) does not match OneShotResponseListener(matching_commands=(AppConfig.BDBCommissioningNotification.Callback(Status=None, Mode=None, RemainingModes=<BDBCommissioningMode.NONE: 0>),), future=<Future pending cb=[ZNP.wait_for_responses..() at /root/zha-backup/venv/lib/python3.7/site-packages/zigpy_znp/api.py:879]>) 2022-02-25 22:27:21.182 rpi-sj zigpy_znp.api TRACE Removing listener OneShotResponseListener(matching_commands=(AppConfig.BDBStartCommissioning.Rsp(Status=None), RPCError.CommandNotRecognized.Rsp(ErrorCode=None, RequestHeader=CommandHeader(id=0x05, subsystem=Subsystem.APPConfig, type=CommandType.SREQ))), future=<Future finished result=AppConfig.BDB...s.SUCCESS: 0>)>) 2022-02-25 22:27:21.183 rpi-sj zigpy_znp.api TRACE Cleaning up empty listener list for header CommandHeader(id=0x00, subsystem=Subsystem.RPCError, type=CommandType.SRSP) 2022-02-25 22:27:21.183 rpi-sj zigpy_znp.api TRACE Cleaning up empty listener list for header CommandHeader(id=0x05, subsystem=Subsystem.APPConfig, type=CommandType.SRSP) 2022-02-25 22:27:21.184 rpi-sj zigpy_znp.api TRACE There are 9 callbacks and 2 one-shot listeners remaining 2022-02-25 22:27:21.434 rpi-sj zigpy_znp.uart TRACE Received data: b'\xFE\x01\x45\xC0\x08\x8C' 2022-02-25 22:27:21.435 rpi-sj zigpy_znp.uart TRACE Parsed frame: TransportFrame(payload=GeneralFrame(header=CommandHeader(id=0xC0, subsystem=Subsystem.ZDO, type=CommandType.AREQ), data=b'\x08')) 2022-02-25 22:27:21.435 rpi-sj zigpy_znp.api DEBUG Received command: ZDO.StateChangeInd.Callback(State=<DeviceState.StartingAsCoordinator: 8>) 2022-02-25 22:27:21.436 rpi-sj zigpy_znp.api TRACE ZDO.StateChangeInd.Callback(State=<DeviceState.StartingAsCoordinator: 8>) does not match OneShotResponseListener(matching_commands=(ZDO.StateChangeInd.Callback(State=<DeviceState.StartedAsCoordinator: 9>),), future=<Future pending cb=[ZNP.wait_for_responses..() at /root/zha-backup/venv/lib/python3.7/site-packages/zigpy_znp/api.py:879]>) 2022-02-25 22:27:21.436 rpi-sj zigpy_znp.api TRACE ZDO.StateChangeInd.Callback(State=<DeviceState.StartingAsCoordinator: 8>) matches CallbackResponseListener(matching_commands=(ZDO.StateChangeInd.Callback(State=<DeviceState.StartingAsCoordinator: 8>), AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.InProgress: 1>, Mode=None, RemainingModes=None)), callback=<bound method Queue.put_nowait of <Queue at 0xf5736150 maxsize=0 _queue=[AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.InProgress: 1>, Mode=<BDBCommissioningMode.NwkSteering: 2>, RemainingModes=<BDBCommissioningMode.NwkFormation: 4>), ZDO.StateChangeInd.Callback(State=<DeviceState.StartingAsCoordinator: 8>)] tasks=2>>) 2022-02-25 22:27:46.360 rpi-sj zigpy_znp.api TRACE Removing listener OneShotResponseListener(matching_commands=(AppConfig.BDBCommissioningNotification.Callback(Status=None, Mode=None, RemainingModes=<BDBCommissioningMode.NONE: 0>),), future=) 2022-02-25 22:27:46.360 rpi-sj zigpy_znp.api TRACE There are 9 callbacks and 1 one-shot listeners remaining 2022-02-25 22:27:46.361 rpi-sj zigpy_znp.api TRACE Removing listener OneShotResponseListener(matching_commands=(AppConfig.BDBCommissioningNotification.Callback(Status=None, Mode=None, RemainingModes=<BDBCommissioningMode.NONE: 0>),), future=) 2022-02-25 22:27:46.361 rpi-sj zigpy_znp.api TRACE There are 9 callbacks and 1 one-shot listeners remaining 2022-02-25 22:27:46.361 rpi-sj zigpy_znp.api TRACE Removing listener CallbackResponseListener(matching_commands=(ZDO.StateChangeInd.Callback(State=<DeviceState.StartingAsCoordinator: 8>), AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.InProgress: 1>, Mode=None, RemainingModes=None)), callback=<bound method Queue.put_nowait of <Queue at 0xf5736150 maxsize=0 _queue=[AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.InProgress: 1>, Mode=<BDBCommissioningMode.NwkSteering: 2>, RemainingModes=<BDBCommissioningMode.NwkFormation: 4>), ZDO.StateChangeInd.Callback(State=<DeviceState.StartingAsCoordinator: 8>)] tasks=2>>) 2022-02-25 22:27:46.362 rpi-sj zigpy_znp.api TRACE Cleaning up empty listener list for header CommandHeader(id=0x80, subsystem=Subsystem.APPConfig, type=CommandType.AREQ) 2022-02-25 22:27:46.362 rpi-sj zigpy_znp.api TRACE There are 7 callbacks and 1 one-shot listeners remaining 2022-02-25 22:27:46.363 rpi-sj zigpy_znp.api TRACE Creating one-shot listener OneShotResponseListener(matching_commands=(SYS.ResetInd.Callback(Reason=None, TransportRev=None, ProductId=None, MajorRel=None, MinorRel=None, MaintRel=None),), future=) 2022-02-25 22:27:46.363 rpi-sj zigpy_znp.api DEBUG Sending request: SYS.ResetReq.Req(Type=<ResetType.Soft: 1>) 2022-02-25 22:27:46.364 rpi-sj zigpy_znp.api DEBUG Request has no response, not waiting for one. 2022-02-25 22:27:46.364 rpi-sj zigpy_znp.uart TRACE Sending data: b'\xFE\x01\x41\x00\x01\x41' 2022-02-25 22:27:46.390 rpi-sj zigpy_znp.uart TRACE Received data: b'\x00' 2022-02-25 22:27:49.868 rpi-sj zigpy_znp.uart TRACE Received data: b'\xFE\x06\x41\x80\x00\x02\x01\x02\x07\x01\xC0' 2022-02-25 22:27:49.868 rpi-sj zigpy_znp.uart TRACE Parsed frame: TransportFrame(payload=GeneralFrame(header=CommandHeader(id=0x80, subsystem=Subsystem.SYS, type=CommandType.AREQ), data=b'\x00\x02\x01\x02\x07\x01')) 2022-02-25 22:27:49.869 rpi-sj zigpy_znp.api DEBUG Received command: SYS.ResetInd.Callback(Reason=<ResetReason.PowerUp: 0>, TransportRev=2, ProductId=1, MajorRel=2, MinorRel=7, MaintRel=1) 2022-02-25 22:27:49.870 rpi-sj zigpy_znp.api TRACE SYS.ResetInd.Callback(Reason=<ResetReason.PowerUp: 0>, TransportRev=2, ProductId=1, MajorRel=2, MinorRel=7, MaintRel=1) matches OneShotResponseListener(matching_commands=(SYS.ResetInd.Callback(Reason=None, TransportRev=None, ProductId=None, MajorRel=None, MinorRel=None, MaintRel=None),), future=) 2022-02-25 22:27:49.870 rpi-sj zigpy_znp.api TRACE Removing listener OneShotResponseListener(matching_commands=(SYS.ResetInd.Callback(Reason=None, TransportRev=None, ProductId=None, MajorRel=None, MinorRel=None, MaintRel=None),), future=) 2022-02-25 22:27:49.871 rpi-sj zigpy_znp.api TRACE Cleaning up empty listener list for header CommandHeader(id=0x80, subsystem=Subsystem.SYS, type=CommandType.AREQ) 2022-02-25 22:27:49.871 rpi-sj zigpy_znp.api TRACE There are 7 callbacks and 1 one-shot listeners remaining 2022-02-25 22:27:49.872 rpi-sj zigpy_znp.api TRACE Removing listener OneShotResponseListener(matching_commands=(SYS.ResetInd.Callback(Reason=None, TransportRev=None, ProductId=None, MajorRel=None, MinorRel=None, MaintRel=None),), future=) 2022-02-25 22:27:49.872 rpi-sj zigpy_znp.api TRACE Cleaning up empty listener list for header CommandHeader(id=0x80, subsystem=Subsystem.SYS, type=CommandType.AREQ) 2022-02-25 22:27:49.873 rpi-sj zigpy_znp.api TRACE There are 7 callbacks and 1 one-shot listeners remaining 2022-02-25 22:27:49.873 rpi-sj zigpy_znp.uart DEBUG Closing serial port

For this case i also removed the antenna.

Anything more i could try?

Kind Regards, Stephan

[anoril1979]

Hi! Many thanks for your reply.

Are you seeing this runtime error? You should be able to pip install pysqlite-binary if you cannot upgrade your system SQLite version.

Yes, I do. Note that I'm a very noob using linux (and Windows, either...) I tried to install PySQLite3 with no success: first, "pip install pysqlite3-binary" attempt ends with this error:

ERROR: Could not find a version that satisfies the requirement pysqlite3-binary (from versions: none)
ERROR: No matching distribution found for pysqlite3-binary

And when I try with "pip install pysqlite3", I have this output:

Collecting pysqlite3
  Using cached pysqlite3-0.4.6.tar.gz (40 kB)
  Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for pysqlite3, since package 'wheel' is not installed.
Installing collected packages: pysqlite3
  Running setup.py install for pysqlite3 ... done
Successfully installed pysqlite3

But 'pip list' does not list it. And running zigpy returns the same error about missing SQLite3 module.

How exactly are you installing these packages? The instructions above should be compatible with Windows if you have git installed, you'll just have to use Windows-specific commands to enable the virtualenv: https://github.com/zigpy/zigpy-znp/blob/dev/TOOLS.md#creating-a-virtualenv-recommended

I installed it using the command lines above BUT for zigpy-cli as it is not on the main branch. For this last, I get the code manually, unzip, replace the changed files from the PR and then run setup.py. It tells me it is installed and I got it listed with "pip list" but running zigpy returns the "zigpy-cli missing" error...

Windows uses COM ports, like COM3 instead of /dev/serial/by-id/example-device-id. You can see them in Device Manager. Great! Thanks for this answer! I would spend more xp points in "look for the obvious".

[puddly]

@stephan48 I'm unable to replicate your problem, unfortunately. It works fine for me:

...
2022-02-26 11:45:02.021 ubuntu zigpy_znp.api DEBUG Forming temporary network
2022-02-26 11:45:02.021 ubuntu zigpy_znp.api DEBUG Sending request: AppConfig.BDBStartCommissioning.Req(Mode=<BDBCommissioningMode.NwkFormation: 4>)
2022-02-26 11:45:05.509 ubuntu zigpy_znp.api DEBUG Received command: AppConfig.BDBStartCommissioning.Rsp(Status=<Status.SUCCESS: 0>)
2022-02-26 11:45:05.511 ubuntu zigpy_znp.api DEBUG Received command: AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.InProgress: 1>, Mode=<BDBCommissioningMode.NwkSteering: 2>, RemainingModes=<BDBCommissioningMode.NwkFormation: 4>)
2022-02-26 11:45:05.726 ubuntu zigpy_znp.api DEBUG Received command: ZDO.StateChangeInd.Callback(State=<DeviceState.StartingAsCoordinator: 8>)

# Yours never gets to this step
2022-02-26 11:45:07.448 ubuntu zigpy_znp.api DEBUG Received command: ZDO.StateChangeInd.Callback(State=<DeviceState.StartedAsCoordinator: 9>)
2022-02-26 11:45:07.449 ubuntu zigpy_znp.api DEBUG Received command: AppConfig.BDBCommissioningNotification.Callback(Status=<BDBCommissioningStatus.Success: 0>, Mode=<BDBCommissioningMode.NwkSteering: 2>, RemainingModes=<BDBCommissioningMode.NONE: 0>)
...

You can keep the antennas plugged in, they're not the issue. Some things to try:

1. Use a new USB power adapter.
2. Temporarily move the coordinator to a new location, away from any potential interference sources.
3. Flash the latest Z-Stack firmware.

[puddly]

For this last, I get the code manually, unzip, replace the changed files from the PR and then run setup.py.

You don't need to do that. If you absolutely must use Windows and don't have Git installed, you can still run the above commands after translating them to PowerShell:

py -3 -m venv venv
.\venv\Scripts\Activate.ps1

pip install `
    https://github.com/puddly/zigpy/tarball/puddly/new-radio-settings-api `
    https://github.com/puddly/zigpy-cli/tarball/puddly/zigpy-radio-api `
    https://github.com/puddly/zigpy-znp/tarball/puddly/new-radio-settings-api `
    https://github.com/puddly/zigpy-deconz/tarball/puddly/new-radio-api `
    https://github.com/puddly/bellows/tarball/puddly/zigpy-radio-api

zigpy -vv radio deconz COM7 backup conbee.json
zigpy -vv radio znp COM8 restore conbee.json

deactivate

[stephan48]

Thank you for testing and your assistance.

I moved the radio into the garage. There i only have one WiFi AP on another channel. And another usb cable+power brick.

I also tried all versions of firmware in the https://github.com/Koenkk/Z-Stack-firmware/tree/8df8cdfbc63738910edfc46b679a858849741213/coordinator/Z-Stack_3.x.0 tree which i could find from 2021 - the last general available being 17.12.2021. Doing a nvram reset pre and post. And then a restore.

This seemed to have affected nothing to the positive. Short of believing the Radio is faulty I am out of ideas - it worked with. Are there newer firmwares you know of?

The network i am trying to restore is the backup of the Z2M minimal instance, so its not online right now.

I will try setting up a ZHA instance against the radio without any prior devices whatsoever to see if it would come up and allow joining of devices(as it did with Z2M last week). If that works, at least I could migrate all my stuff manually - still annoying through.

Still strange through.

[hervadore]

For anyone migrating from the Sonoff ZBBridge with Tasmota you can run the backup with: zigpy -vv radio ezsp socket://x.x.x.x:port backup znp.json

I was able to restore to the Sonoff 3.0 Dongle after several attempts but for me I had to remove the flow_control entry from core.config_entries for it to work. Flow_control was set to "software" for me and also tried setting it to "hardware" which also did not work.

[ark-]

I am backing up a Raspbee II and I see no devices in the json file. I see an earlier comment saying this is OK and expected. What I'd like to know please is would it be likely I could restore to a Zig-Ah-Zig-Ah (zzh) and not have to re pair all devices using this?

[puddly]

@ark- It's not a problem

[stephan48]

hi,

Maybe this is a bit premature yet, I will test the stability over the next weeks... but for now I think I successfully migrated from the Conbee2 to the TubesZB Gateway - with a workaround.

Since direct restore to the TubesZB would not work(see my comments above) I bought a Sonoff Zigbee 3.0 Stick. After a bit of back and forth I managed to restore the Conbee2 Network to the Sonoff from which I then took a nvram backup. This backup i applied to the TubesZB Gateway - and verified with the info command that the same data as on the conbee/sonoff 2 is present.

After editing the Homeassistant Config to point to the new Gateway - everything is working again after a short phase of resynchronisation.

\o/

Steps I followed: export TUBE=socket://tube-zb-gw-cc2652p2:6638/ export SONOFF=socket://wks-sj3:20108 export CONBEE=/dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DEXX-if00 zigpy -vv radio znp $CONBEE info zigpy -vv radio deconz $CONBEE backup deconz20220326.json

zigpy -vv radio znp $SONOFF restore deconz20220326.json zigpy -vv radio znp $SONOFF info python -m zigpy_znp.tools.nvram_read $SONOFF -o backup_conbee_sonoff.json python -m zigpy_znp.tools.nvram_write $TUBE -i backup_conbee_sonoff.json zigpy -vv radio znp $TUBE info Edit the ZHA entry("domain": "zha") in "/usr/share/hassio/homeassistant/.storage/core.config_entries" (I did it from the host after stopping the homeassistant docker container) to match this: ... "data": { "device": { "path": "socket://tube-zb-gw-cc2652p2:6638/", "baudrate": 115200, "flow_control": "software" }, "radio_type": "znp" }, ... Please don't follow the advice on https://github.com/zigpy/zigpy-znp/blob/dev/TOOLS.md#backup-and-restore to remove the ZHA config and readd it - yes your devices will be found again almost instantly - but... they will have new IDs - hence all automation/naming/etc. is lost :/ - Because I got cocky I had to restore last weeks backup. If you temporarily want to expose the Conbee or Sonoff via Network you can use something like this for ser2net 4.x(See this for the source: https://www.zigbee2mqtt.io/advanced/remote-adapter/connect_to_a_remote_adapter.html): $ cat /etc/ser2net.yaml ... connection: &con01 accepter: tcp,20108 connector: serialdev,/dev/ttyUSB0,115200n81,local options: kickolduser: true ... I hope this helps someone in a similar situation :) Thanks @puddly for all your work and help! Kind Regards, Stephan

[puddly]

Interesting. Are you able to join new devices?

[stephan48]

I am currently trying it with an IKEA Tradfri Remote I reset and removed from ZHA, but not much luck. I could "rejoin" a Aquara Switch earlier on the new Setup. I don't think i have any new devices around atm. Lets see - if needed i will need to pair a plain thing with a ptvo firmware

[stephan48]

Looks like it works, the IKEA remotes did struggle a bit but I could join them. Why are you asking about joing in particular?

[anoril1979]

Hi! I found a working Ubuntu and got Zigpi cli installed as shown above and It worked fine! After some TIMEOUT issues I realized I did not disabled ZHA in HomeAssistant resulting in conflicts during backup. Now I managed to backup my wifi gateway and restore it to the new Sonoff dongle.

BTW, I would be happy to get some help to get further as HA is telling me he can't start ZHA with the new dongle (configuration issue, Z-Stack cannot be started...) (First, it was because ttyUSB0 was not accessible so I chmoded it; then it was because of hardware/software flow control... Now I got a Broken Pipe Error (code 32)) :/

Anyway, many thanks Puddly for your advices and patience!

[stephan48]

Did you change the type of dongle(the radio_type) if your old wifi gw was no znp based device?

[anoril1979]

Hi Stephan, I updated first the title (to /dev/ttySUB0 even if useless), the path (from socket://xxx:8888 to /dev/ttyUSB0), the type (from ezsp to znp) with no success. Searching the net, I decided to try changing the data flow from software to hardware with no success neither, then tried to play with the baudrate...

The good point is that restoring my config and plug-in back my old gateway made all back to normal :D

[puddly]

Make sure to set the baudrate to 115200 and remove the flow control key entirely. If that doesn't work, can you post the ZHA startup debug log (https://www.home-assistant.io/integrations/zha/#debug-logging)?

[anoril1979]

... :O

Thanks for your incredible help (every one of you)! Setting back the baudrate to 115200 AND removing the flow control key did the trick!

I had to activate all my Zigbee devices, BTW... Note: 10 minutes later, my entities are still indentified as "not available"... OK... Got to activate all entities after devices... A bit long but after a restart, everything seems to work... Got to check twice :D

[robtpaton]

Hi! Trying to take a backup from a Conbee II stick with this command

zigpy -vv radio deconz /dev/serial/by-id/<ID HERE> backup deconz.json

but I'm seeing this error

2022-04-19 11:09:25.083 homeassistant asyncio DEBUG Using selector: EpollSelector
2022-04-19 11:09:25.212 homeassistant zigpy.appdb DEBUG SQLite version for <module 'sqlite3' from '/usr/local/lib/python3.9/sqlite3/__init__.py'>: 3.34.1
Traceback (most recent call last):
  File "/config/venv2/bin/zigpy", line 8, in <module>
    sys.exit(cli())
  File "/config/venv2/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/config/venv2/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/config/venv2/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/config/venv2/lib/python3.9/site-packages/click/core.py", line 1654, in invoke
    super().invoke(ctx)
  File "/config/venv2/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/config/venv2/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/config/venv2/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/config/venv2/lib/python3.9/site-packages/zigpy_cli/cli.py", line 19, in inner
    return loop.run_until_complete(cmd(*args, **kwargs))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/config/venv2/lib/python3.9/site-packages/zigpy_cli/radio.py", line 52, in radio
    app = app_cls(config)
TypeError: Can't instantiate abstract class ControllerApplication with abstract method register_endpoint

This is inside a virtualenv. Have also tried --force-reinstall when installing the dependencies above. Any ideas? To be able to install the dependencies, I also had to install gcc, not sure if that's relevant.

Thanks in advance!

[puddly]

@robtpaton oops! Re-create the virtualenv and reinstall all of the packages, I forgot to push a few changes to one of the packages.

[robtpaton]

Done and fixed thank you! Can confirm I was able to move from a Conbee II to Sonoff Zigbee Dongle 👍