Reconstruct OTA images from PCAP files

[puddly]

This is a (hacky and WIP) tool to reconstruct Zigbee OTA images from raw traffic. It combines blocks in any order from multiple independent PCAPs.

There's unfortunately no simple, pure-Python way to parse PCAP files so it relies on Wireshark's tshark executable to be available in the PATH. Sample usage:

$ zigpy ota reconstruct-from-pcaps --add-network-key aa:bb:cc:dd:ee:ff:00:11:22:33:44:55:66:77:88:99 --output-root ./extracted/ *.pcap
Constructing image type=0x298b, version=0x00000005, manuf_code=0x115f: 157424 bytes
2023-02-22 03:39:51.406 ubuntu zigpy_cli.ota ERROR Missing 48 bytes starting at offset 0x0000ADA0: filling with 0xAB
2023-02-22 03:39:51.406 ubuntu zigpy_cli.ota ERROR Missing 48 bytes starting at offset 0x000106B0: filling with 0xAB
Constructing image type=0x298b, version=0x00000009, manuf_code=0x115f: 163136 bytes

It will generate OTA files that correspond to every detected image:

$ ls extracted
ota_t0x298b_m0x115f_v0x00000005_partial.ota
ota_t0x298b_m0x115f_v0x00000009.ota

Hardware-specific images aren't currently supported.

CC: @TheJulianJES

[pipiche38]

This looks interesting, I did try some time ago ( https://github.com/pipiche38/Capture-OTA-from-Wireshark ). Some of the obstacles are:

• Sniff do not report all packets
• Some of the devices keep what have been already downloaded, so if the OTA is interrupted and you restart, it uploads only the missing packets.

[MattWestb]

Good morning our Puddly !!

I have my Dirigera with one GU10-WS2 and one E27-CWS3 plus 2 OnOff dimmer switches (E1743). Im have one E1743 that have getting one OpenClose user data flashed and is hocked up with SWD so i was flashing one older firmware on it and paring with Dirigera. And it was doing one node update from 0x0.

Instruction is in the update ;-))

Edit with filtered sniff: IKEAOTA.pcapng.gz

[TheJulianJES]

Linking another somewhat similar project for reference: https://github.com/compujuckel/ZigbeeOtaExtractor

[MattWestb]

Using this as filter i only getting the response with the payload ((wpan.dst16 == 0x90d9) && (zbee_zcl_general.ota.cmd.srv_tx.id == 0x05)) (the node is 0x90d9 so only need changing it to your node if more is doing OTA). The question is if one better format exporting the cap file for doing the puzeling easier ?

[codecov-commenter]

Codecov Report

Base: 2.61% // Head: 2.20% // Decreases project coverage by -0.42% :warning:

Coverage data is based on head (c3d7c76) compared to base (125b537). Patch coverage: 0.00% of modified lines in pull request are covered.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

```diff @@ Coverage Diff @@ ## dev #29 +/- ## ======================================== - Coverage 2.61% 2.20% -0.42% ======================================== Files 8 8 Lines 420 498 +78 ======================================== Hits 11 11 - Misses 409 487 +78 ``` | [Impacted Files](https://codecov.io/gh/zigpy/zigpy-cli/pull/29?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=zigpy) | Coverage Δ | | |---|---|---| | [zigpy\_cli/ota.py](https://codecov.io/gh/zigpy/zigpy-cli/pull/29?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=zigpy#diff-emlncHlfY2xpL290YS5weQ==) | `0.00% <0.00%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=zigpy). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=zigpy)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[puddly]

@MattWestb Thanks! You can leave them unfiltered, the network key will automatically be extracted by tshark and only the relevant Zigbee packets will be used. Your captures worked fine for me without modification and the extracted .ota file passed GBL validation and CRC checks:

$ $ zigpy ota reconstruct-from-pcaps --output-root ./extracted/ ~/Downloads/E1743-244005.pcapng ~/Downloads/IKEAOTA.pcapng
Constructing image type=0x11c5, version=0x24040005, manuf_code=0x117c: 205676 bytes
$ zigpy ota info ./extracted/ota_t0x11c5_m0x117c_v0x24040005.ota
./extracted/ota_t0x11c5_m0x117c_v0x24040005.ota
Type: <class 'zigpy.ota.image.OTAImage'>
Header: OTAImageHeader(upgrade_file_id=200208670, header_version=256, header_length=56, field_control=<FieldControl.0: 0>, manufacturer_id=4476, image_type=4549, file_version=604241925, stack_version=2, header_string='GBL GBL_tradfri_onoff_controller', image_size=205676, *device_specific_file=False, *hardware_versions_present=False, *key=ImageKey(manufacturer_id=4476, image_type=4549), *security_credential_version_present=False)
Number of subelements: 3
Validation result: ValidationResult.VALID

Attached: ota_t0x11c5_m0x117c_v0x24040005.ota.zip

[MattWestb]

Great work !!!

I flashing the old firmware on the "OpenClose" and testing one ZHA RCP updating it with your file and reporting back :-)))

Then you have it reedy for testing (= not so technical user like my can installing it) i like using it for more OTA sniffs from Dirigera.

[MattWestb]

2023-02-22 21:12:55.647 DEBUG (MainThread) [zigpy.zcl] [0xA004:1:0x0019] Decoded ZCL frame: Ota:upgrade_end(status=<Status.SUCCESS: 0>, manufacturer_code=4476, image_type=4549, file_version=604241925)

So the signing is not broken and the device have flashing it OK and after restart its showing Firmware: 0x24040005 so ZHA can using the Puddly OTA files :-)))

Great work done !!

And large thanks for helping getting our system working better and getting more great futures.

[MattWestb]

By the was @puddly was you looking in the sniff how IKEA is doing the paring in Dirigera ? Its very interesting then using commands ZHA and Z2M is not using for getting the device not going in to sleep and also configuring all things they needing.

[MattWestb]

May i request implanting of metadata patching that Julian have doing with commands and its also working https://github.com/zigpy/zigpy/issues/1153#issuecomment-1415991246 without braking the signing? Shall being great having it in zigby-CLI as standard command (perhaps having override commands for manufacture, device ID, Hardware version and software version).

[MattWestb]

One warning for Windows users !!

If having wireshark installed you need updating the system PATH and the instruction that is floating around is working very well but its overwriting you current system PATH with only the new one and don NOT appending it = loosing the system ground PATH. If have done very mush things stop working OK and if you have system installed on SSD and no restore point its only possible with copy the old register settings with shadow copy or restore factory default and all installed programs must being reinstalled.

So do one register backup before doing some experiment or adding the path thru system settings GUI.

[MattWestb]

Was trying installing with pip and using the branch or the commit as tag but it was not working. Last try was pip install git+https://github.com/puddly/zigpy-cli@2674f45 but was missing the OTA.py so i was adding it manually in my test system and it looks working with 2 sniffs:

0x11C6 4550 TRADFRI SHORTCUT Button (E1812)

zigpy ota reconstruct-from-pcaps --output-root ./extracted/  E1812-244005.pcapng
Constructing image type=0x11c6, version=0x24040005, manuf_code=0x117c: 205180 bytes

zigpy ota info .\extracted\ota_t0x11c6_m0x117c_v0x24040005.ota
extracted\ota_t0x11c6_m0x117c_v0x24040005.ota
Type: <class 'zigpy.ota.image.OTAImage'>
Header: OTAImageHeader(upgrade_file_id=200208670, header_version=256, header_length=56, field_control=<FieldControl: 0>, manufacturer_id=4476, image_type=4550, file_version=604241925, stack_version=2, header_string='GBL GBL_tradfri_shortcut_button', image_size=205180, *device_specific_file=False, *hardware_versions_present=False, *key=ImageKey(manufacturer_id=4476, image_type=4550), *security_credential_version_present=False)
Number of subelements: 3
Validation result: ValidationResult.VALID

0x11CA 4554 SYMFONISK Sound Controller (E1744)

zigpy ota reconstruct-from-pcaps --output-root ./extracted/  E1744-244005.pcapng
Constructing image type=0x11ca, version=0x24040005, manuf_code=0x117c: 214692 bytes
2023-02-24 20:19:20.334 MW-E3222 zigpy_cli.ota ERROR Missing 63 bytes starting at offset 0x00006ACE: filling with 0xAB
2023-02-24 20:19:20.334 MW-E3222 zigpy_cli.ota ERROR Missing 63 bytes starting at offset 0x00010E75: filling with 0xAB

So i need "finding" 2 lost packages for getting Sxmfonisk to working but Shortcutt button looks being OK.

Can some one hinting doing the PIP right for installing on commits or forked branches ?

[puddly]

@MattWestb It should work with the branch name:

pip install git+https://github.com/puddly/zigpy-cli.git@puddly/ota-pcap-extraction

You may have to uninstall and reinstall the package to get it to actually "upgrade".

[MattWestb]

Thanks Puddly !! I was using the -U that shall doing the same but perhaps somthing was going wrong.

In 2 hours i shall have the next sniff for testing :-)))

[MattWestb]

zigpy ota reconstruct-from-pcaps --output-root ./extracted/  E1744-244005A.pcapng
Constructing image type=0x11ca, version=0x24040005, manuf_code=0x117c: 214692 bytes

zigpy ota info .\extracted\ota_t0x11ca_m0x117c_v0x24040005.ota
extracted\ota_t0x11ca_m0x117c_v0x24040005.ota
Type: <class 'zigpy.ota.image.OTAImage'>
Header: OTAImageHeader(upgrade_file_id=200208670, header_version=256, header_length=56, field_control=<FieldControl: 0>, manufacturer_id=4476, image_type=4554, file_version=604241925, stack_version=2, header_string='GBL GBL_tradfri_dimmer', image_size=214692, *device_specific_file=False, *hardware_versions_present=False, *key=ImageKey(manufacturer_id=4476, image_type=4554), *security_credential_version_present=False)
Number of subelements: 3
Validation result: ValidationResult.VALID

Tomorrow testing SWD flashing older version and updating them with ZHA for validating and if going well putting them in the IKEA OTA Matrix :-)))

[MattWestb]

I have made sniffs of paring and updating 6 device types that is having Dirigera updates and only missed 3 packages at all from 2 devices and was SWD flashing back old firmware for redoing it. The best is getting the device connecting to one router and sniffing in between so getting the frames sniffed 2 times (End device > parent > coordinator) you i getting 100% higher chance not loosing some frames.

I think Starkvind have getting one update but i dont have device but i shall trying flashing the firmware on one "Billy module" and see if i can getting it starting OK.

Also we have getting the firmware from Symfonisk 2 and our Danish friends have making one updated PCB for there wall switch but the device is debug locked so we cant dumping the firmware or the userdata also they was baying some new E14 lights with MG21 module and they was also debug locked but was possible unlocking and getting one chip erase. Looks working OK flashing one Styrbar dump and then the extracted Symfonisk 2 OTA file on it and its starting with new extra end points but its not 100% tested.

So also all IKEA blinds is now running 24.4.5 :-)))

Great thanks to our Puddly !!!

[Hedda]

FYI, posted about a another similar tool here -> https://github.com/zigpy/zigpy/discussions/723

compujuckel written a small program for extracting and reassembling Zigbee OTA images file from Wireshark packet captures:

https://github.com/compujuckel/ZigbeeOtaExtractor

"Zigbee OTA Extractor - Tool to extract Zigbee OTA images from packet captures"

I originally read about that in this other thread -> https://github.com/Koenkk/zigbee2mqtt/issues/14926

[Hedda]

compujuckel written a small program for extracting and reassembling Zigbee OTA images file from Wireshark packet captures:

https://github.com/compujuckel/ZigbeeOtaExtractor

"Zigbee OTA Extractor - Tool to extract Zigbee OTA images from packet captures"

I originally read about that in this other thread -> Koenkk/zigbee2mqtt#14926

By the way, note his prerequisites:

_A Zigbee sniffer is required to get the packet capture. Follow this guide to get started. If your device needs an install code, you'll have to derive a link key using AES-MMO first. For convenience, I've written a small tool to do that here._

https://jsfiddle.net/p7yroctn/

That was mentioned for Bosch Radiator Thermostat II device which is a device that can not be joined without its install code:

https://github.com/Koenkk/zigbee2mqtt/issues/14926

[MattWestb]

If your device needs an install code, you'll have to derive a link key using AES-MMO first. For convenience, I've written a small tool to do that here.

I think his install code program is great so can getting the TC-Link Key for decryption packages !! Can being somthing for Puddly :-))

[MattWestb]

Future request

Adding check of the OTA file then its finished assembled like with the info command so not need rerunning zigpy-CLI one time after having the file made and is getting all nice printed in the CLI with one command :-))

[MattWestb]

Thanks P ! !

[MattWestb]

I think one comment in readmy that if pairing the one device in the PCAP is the network key not needed the tshark is extracting it and using it for the file(s?).