husqvarnagroup / smart-garden-gateway-public

Open Source Components of the GARDENA smart Gateway
https://www.gardena.com/de/produkte/smart/smartsystem/
102 stars 13 forks source link

Gateway stuck in bootloop "damaged volume, update marker is setresetting ..." #3

Closed juliansteenbakker closed 2 years ago

juliansteenbakker commented 2 years ago

First of all, It's great to see a commercial company taking open-source serious! I was googling around to see if there was some code available of the gateway and i stumbled upon this presentation which was really fun to look through!

My problem in short: I have a Smart Gateway with artnr 19005, which i was setting up a few days ago. Everything went well until i wanted to switch from wired to wireless connection. It seemed like it rebooted over and over, and was in some kind of a bootloop. I tried resetting the device, however it would not make any difference. The green led would stay on for 2 seconds, after which it turns off for 1 second and then repeats the pattern.

When i found this repository i decided to solder some headers on the serial connector and see if there was some kind of output. It seems like it's crashing on damaged volume, update marker is setresetting .... I was wondering if there is a way to reflash/reset the memory.

The logs:

U-Boot 2021.04-gardena-6 (Jun 10 2021 - 16:05:31 +0000)

CPU:   MediaTek MT7688A ver:1 eco:2
Boot:  DDR2, SPI-NOR 3-Byte Addr, CPU clock from XTAL
Clock: CPU: 580MHz, Bus: 193MHz, XTAL: 40MHz
Model: GARDENA smart Gateway (MT7688)
DRAM:  128 MiB
WDT:   Started with servicing (60s timeout)
Loading Environment from SPIFlash... SF: Detected XM25QH64A with page size 256 Bytes, erase size 4 KiB, total 8 MiB
OK
F-Data:factory-data version 1 detected
Net:   eth0: eth@10110000
Autoboot in 0 seconds
linuxmoduleid=redacted
gatewayid=redacted
radiomoduleid=redacted
ubi0: default fastmap pool size: 50
ubi0: default fastmap WL pool size: 25
ubi0: attaching mtd7
ubi0: scanning is finished
ubi0: attached mtd7 (name "nand", size 128 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 1023, bad PEBs: 1, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 33/15, WL threshold: 4096, image sequence number: 0
ubi0: available PEBs: 18, total reserved PEBs: 1005, PEBs reserved for bad PEB handling: 21
gpio: pin PA11 (gpio 11) value is 1
recovery_status=in_progress
damaged volume, update marker is setresetting ...

U-Boot SPL 2021.04-gardena-6 (Jun 10 2021 - 16:05:31 +0000)
Trying to boot from NOR

U-Boot 2021.04-gardena-6 (Jun 10 2021 - 16:05:31 +0000)

CPU:   MediaTek MT7688A ver:1 eco:2
Boot:  DDR2, SPI-NOR 3-Byte Addr, CPU clock from XTAL
Clock: CPU: 580MHz, Bus: 193MHz, XTAL: 40MHz
... (repeat)

Logs when holding the reset button:

CPU:   MediaTek MT7688A ver:1 eco:2
Boot:  DDR2, SPI-NOR 3-Byte Addr, CPU clock from XTAL
Clock: CPU: 580MHz, Bus: 193MHz, XTAL: 40MHz
Model: GARDENA smart Gateway (MT7688)
DRAM:  128 MiB
WDT:   Started with servicing (60s timeout)
Loading Environment from SPIFlash... SF: Detected XM25QH64A with page size 256 Bytes, erase size 4 KiB, total 8 MiB
OK
F-Data:factory-data version 1 detected
Net:   eth0: eth@10110000
Autoboot in 0 seconds
linuxmoduleid=redacted
gatewayid=redacted
radiomoduleid=redacted
ubi0: default fastmap pool size: 50
ubi0: default fastmap WL pool size: 25
ubi0: attaching mtd7
ubi0: scanning is finished
ubi0: attached mtd7 (name "nand", size 128 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 1023, bad PEBs: 1, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 33/15, WL threshold: 4096, image sequence number: 0
ubi0: available PEBs: 18, total reserved PEBs: 1005, PEBs reserved for bad PEB handling: 21
gpio: pin PA11 (gpio 11) value is 0
Button pressed, issuing factory reset
ubi0: detaching mtd7
ubi0: mtd7 is detached
ubi0: default fastmap pool size: 50
ubi0: default fastmap WL pool size: 25
ubi0: attaching mtd7
ubi0: scanning is finished
ubi0: attached mtd7 (name "nand", size 128 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 1023, bad PEBs: 1, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 33/15, WL threshold: 4096, image sequence number: 0
ubi0: available PEBs: 18, total reserved PEBs: 1005, PEBs reserved for bad PEB handling: 21
Remove UBI volume overlay (id 4)
Creating dynamic volume overlay of size 31457280
recovery_status=in_progress
damaged volume, update marker is setresetting ...
rettichschnidi commented 2 years ago

Your gateway is in a sticky situation - basically, the (content of the) NOR flash seems to be fine, but the active UBI volume is broken. This is supposed to not happen, ever. Sorry!

There is a undocumented feature (we named it "resurrection reset"), which might help in this situation. Basically, you keep the reset button pressed for ~30 seconds. The power led will turn from green to yellow as it did for the reset, but after 30 seconds or so turn to red. This will switch back to the other boot slot. On the console you will see many lines gpio: pin PA11 (gpio 11) value is 0 followed by resurrection reset triggered:

U-Boot SPL 2021.04-gardena-6 (Jun 10 2021 - 16:05:31 +0000)
Trying to boot from NOR

U-Boot 2021.04-gardena-6 (Jun 10 2021 - 16:05:31 +0000)

CPU:   MediaTek MT7688A ver:1 eco:2
Boot:  DDR2, SPI-NOR 3-Byte Addr, CPU clock from XTAL
Clock: CPU: 580MHz, Bus: 193MHz, XTAL: 40MHz
Model: GARDENA smart Gateway (MT7688)
DRAM:  128 MiB
WDT:   Started with servicing (60s timeout)
Loading Environment from SPIFlash... SF: Detected XM25QH64A with page size 256 Bytes, erase size 4 KiB, total 8 MiB
OK
F-Data:factory-data version 1 detected
Net:   eth0: eth@10110000
Autoboot in 0 seconds
linuxmoduleid=censored
gatewayid=censored
radiomoduleid=censored
ubi0: default fastmap pool size: 50
ubi0: default fastmap WL pool size: 25
ubi0: attaching mtd7
ubi0: scanning is finished
ubi0: attached mtd7 (name "nand", size 128 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 1023, bad PEBs: 1, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 96/82, WL threshold: 4096, image sequence number: 0
ubi0: available PEBs: 18, total reserved PEBs: 1005, PEBs reserved for bad PEB handling: 21
gpio: pin PA11 (gpio 11) value is 0
Button pressed, issuing factory reset
ubi0: detaching mtd7
ubi0: mtd7 is detached
ubi0: default fastmap pool size: 50
ubi0: default fastmap WL pool size: 25
ubi0: attaching mtd7
ubi0: scanning is finished
ubi0: attached mtd7 (name "nand", size 128 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 1023, bad PEBs: 1, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 96/82, WL threshold: 4096, image sequence number: 0
ubi0: available PEBs: 18, total reserved PEBs: 1005, PEBs reserved for bad PEB handling: 21
Remove UBI volume overlay (id 4)
Creating dynamic volume overlay of size 31457280
## Error: "recovery_status" not defined
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 0
resurrection reset triggered
Saving Environment to SPIFlash... Erasing SPI flash...Writing to SPI flash...done
Valid environment: 2
OK
gpio: pin PA11 (gpio 11) value is 0
gpio: pin PA11 (gpio 11) value is 1
No size specified -> Using max size (3375440)
Read 3375440 bytes from volume kernel0 to 80a00000
## Loading kernel from FIT Image at 80a00000 ...
   Using 'conf-ralink_gardena_smart_gateway_mt7688.dtb' configuration
   Verifying Hash Integrity ... OK
   Trying 'kernel-1' kernel subimage
     Description:  Linux kernel
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x80a00118
     Data Size:    3365824 Bytes = 3.2 MiB
     Architecture: MIPS
     OS:           Linux
     Load Address: 0x80000000
     Entry Point:  0x80000400
     Hash algo:    sha256
     Hash value:   67044479e9d91be1a551d20afd5ba1e6e75236343cba7ce315e08fa669ea39e5
   Verifying Hash Integrity ... sha256+ OK
## Loading fdt from FIT Image at 80a00000 ...
   Using 'conf-ralink_gardena_smart_gateway_mt7688.dtb' configuration
   Verifying Hash Integrity ... OK
   Trying 'fdt-ralink_gardena_smart_gateway_mt7688.dtb' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x80d35e00
     Data Size:    7574 Bytes = 7.4 KiB
     Architecture: MIPS
     Hash algo:    sha256
     Hash value:   5cc523f9207fbc3435951bfdcbf5cd85e9d64cbaac6dc586c34f11581b91292e
   Verifying Hash Integrity ... sha256+ OK
   Booting using the fdt blob at 0x80d35e00
   Uncompressing Kernel Image
   Loading Device Tree to 86f46000, end 86f4ad95 ... OK
[    0.000000] timer_probe: no matching timers found

If this does not work and you still have some nerves left, I suggest the following to analyze the gateway some more:

Put together, this would look like:

U-Boot SPL 2021.04-gardena-6 (Jun 10 2021 - 16:05:31 +0000)
Trying to boot from NOR

U-Boot 2021.04-gardena-6 (Jun 10 2021 - 16:05:31 +0000)

CPU:   MediaTek MT7688A ver:1 eco:2
Boot:  DDR2, SPI-NOR 3-Byte Addr, CPU clock from XTAL
Clock: CPU: 580MHz, Bus: 193MHz, XTAL: 40MHz
Model: GARDENA smart Gateway (MT7688)
DRAM:  128 MiB
WDT:   Started with servicing (60s timeout)
Loading Environment from SPIFlash... SF: Detected XM25QH64A with page size 256 Bytes, erase size 4 KiB, total 8 MiB
OK
F-Data:factory-data version 1 detected
Net:   eth0: eth@10110000
Autoboot in 0 seconds
=> xxxxxxxxxxxxxxxxxx
Unknown command 'xxxxxxxxxxxxxxxxxx' - try 'help'
=> ubi part nand
ubi0: default fastmap pool size: 50
ubi0: default fastmap WL pool size: 25
ubi0: attaching mtd7
ubi0: scanning is finished
ubi0: attached mtd7 (name "nand", size 128 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 1023, bad PEBs: 1, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 96/82, WL threshold: 4096, image sequence number: 0
ubi0: available PEBs: 18, total reserved PEBs: 1005, PEBs reserved for bad PEB handling: 21
=> ubi info l
Volume information dump:
        vol_id          0
        reserved_pebs   331
        alignment       1
        data_pad        0
        vol_type        4
        name_len        7
        usable_leb_size 126976
        used_ebs        297
        used_bytes      37703680
        last_eb_bytes   118784
        corrupted       0
        upd_marker      0
        skip_check      0
        name            rootfs0
Volume information dump:
        vol_id          1
        reserved_pebs   331
        alignment       1
        data_pad        0
        vol_type        4
        name_len        7
        usable_leb_size 126976
        used_ebs        297
        used_bytes      37703680
        last_eb_bytes   118784
        corrupted       0
        upd_marker      0
        skip_check      0
        name            rootfs1
Volume information dump:
        vol_id          2
        reserved_pebs   34
        alignment       1
        data_pad        0
        vol_type        4
        name_len        7
        usable_leb_size 126976
        used_ebs        27
        used_bytes      3375440
        last_eb_bytes   74064
        corrupted       0
        upd_marker      0
        skip_check      0
        name            kernel0
Volume information dump:
        vol_id          3
        reserved_pebs   34
        alignment       1
        data_pad        0
        vol_type        4
        name_len        7
        usable_leb_size 126976
        used_ebs        27
        used_bytes      3375440
        last_eb_bytes   74064
        corrupted       0
        upd_marker      0
        skip_check      0
        name            kernel1
Volume information dump:
        vol_id          4
        reserved_pebs   248
        alignment       1
        data_pad        0
        vol_type        3
        name_len        7
        usable_leb_size 126976
        used_ebs        248
        used_bytes      31490048
        last_eb_bytes   126976
        corrupted       0
        upd_marker      0
        skip_check      0
        name            overlay
Volume information dump:
        vol_id          2147479551
        reserved_pebs   2
        alignment       1
        data_pad        0
        vol_type        3
        name_len        13
        usable_leb_size 126976
        used_ebs        2
        used_bytes      253952
        last_eb_bytes   2
        corrupted       0
        upd_marker      0
        skip_check      0
        name            layout volume
=> env print bootslot
bootslot=1

None of this information is secret and it would help me to understand your current situation a bit better.

ps: Kudos for a) being the very first one to open an issue here and b) censoring the relevant (semi-secret) parts of the boot logs.

juliansteenbakker commented 2 years ago

I tried pressing the button for 30 seconds (and longer) however it would just keep printing this at the end and then reboot. The LED turns yellow for about 0.1 second but then turns off and repeats the boot process.

Remove UBI volume overlay (id 4)
Creating dynamic volume overlay of size 31457280
recovery_status=in_progress
damaged volume, update marker is setresetting ...

I am able to enter the shell so i executed ubi part nand, ubi info l, and env print bootslot and got the following information:


U-Boot SPL 2021.04-gardena-6 (Jun 10 2021 - 16:05:31 +0000)
Trying to boot from NOR

U-Boot 2021.04-gardena-6 (Jun 10 2021 - 16:05:31 +0000)

CPU:   MediaTek MT7688A ver:1 eco:2
Boot:  DDR2, SPI-NOR 3-Byte Addr, CPU clock from XTAL
Clock: CPU: 580MHz, Bus: 193MHz, XTAL: 40MHz
Model: GARDENA smart Gateway (MT7688)
DRAM:  128 MiB  
WDT:   Started with servicing (60s timeout)
Loading Environment from SPIFlash... SF: Detected XM25QH64A with page size 256 Bytes, erase size 4 KiB, total 8 MiB
OK
F-Data:factory-data version 1 detected
Net:   eth0: eth@10110000
Autoboot in 0 seconds
=> ubi part nand
ubi0: default fastmap pool size: 50
ubi0: default fastmap WL pool size: 25
ubi0: attaching mtd7
ubi0: scanning is finished
ubi0: attached mtd7 (name "nand", size 128 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 1023, bad PEBs: 1, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 33/15, WL threshold: 4096, image sequence number: 0
ubi0: available PEBs: 18, total reserved PEBs: 1005, PEBs reserved for bad PEB handling: 21
=> ubi info l   
Volume information dump:
        vol_id          0
        reserved_pebs   331
        alignment       1
        data_pad        0
        vol_type        4
        name_len        7
        usable_leb_size 126976
        used_ebs        259
        used_bytes      32780288
        last_eb_bytes   20480
        corrupted       0
        upd_marker      0
        skip_check      0
        name            rootfs0
Volume information dump:
        vol_id          1
        reserved_pebs   331
        alignment       1
        data_pad        0
        vol_type        4
        name_len        7
        usable_leb_size 126976
        used_ebs        267
        used_bytes      33808384
        last_eb_bytes   32768
        corrupted       0
        upd_marker      0
        skip_check      0
        name            rootfs1
Volume information dump:
        vol_id          2
        reserved_pebs   34
        alignment       1
        data_pad        0
        vol_type        4
        name_len        7
        usable_leb_size 126976
        used_ebs        25
        used_bytes      3144016
        last_eb_bytes   96592
        corrupted       0
        upd_marker      0
        skip_check      0
        name            kernel0
Volume information dump:
        vol_id          3
        reserved_pebs   34
        alignment       1
        data_pad        0
        vol_type        4
        name_len        7
        usable_leb_size 126976
        used_ebs        0
        used_bytes      0
        last_eb_bytes   0
        corrupted       0
        upd_marker      1
        skip_check      0
        name            kernel1
Volume information dump:
        vol_id          4
        reserved_pebs   248
        alignment       1
        data_pad        0
        vol_type        3
        name_len        7
        usable_leb_size 126976
        used_ebs        248
        used_bytes      31490048
        last_eb_bytes   126976
        corrupted       0
        upd_marker      0
        skip_check      0
        name            overlay
Volume information dump:
        vol_id          2147479551
        reserved_pebs   2
        alignment       1
        data_pad        0
        vol_type        3
        name_len        13
        usable_leb_size 126976
        used_ebs        2
        used_bytes      253952
        last_eb_bytes   2
        corrupted       0
        upd_marker      0
        skip_check      0
        name            layout volume
=> env print bootslot
bootslot=1
=> 
rettichschnidi commented 2 years ago

Sorry for the late reply.

This did not work out at all as the UBI volume kernel1 does not actually have anything stored at all.

Best you can do now is to flash the kernel and rootfs to a known-good state:

Once done I hope your gateway will work again.

Please let me know if something is unclear and/or if it worked - I intend to the extend the README accordingly.

juliansteenbakker commented 2 years ago

Sorry for the late reply.

No problem, we all have things to do ;)

I will try it out later this week and let you know!

juliansteenbakker commented 2 years ago

Success !! I've got a working gateway again 🎉

A few tips for if you want to add this to a readme:

Set up a DHCP/TFTP server

It is good to mention that the easiest way to achieve this is to host the TFPT server on the same machine as the DHCP server. I achieved this by following this guide on how to host a DHCP server on macOS (don't forget to change the interface name to your LAN interface), and then this guide for hosting the TFPT server.

After that it was as easy as plugging in the LAN cable, firing up the gateway and executing the steps you said.

Thanks again for the work, I really enjoyed it. Now let's see if the mower is still working 😉

juliansteenbakker commented 2 years ago

I was wondering: Is there only a repository for the embedded side of the Gardena smart system, or is there also a place to report bugs for the mobile apps? (There are quite a few problems that I've discovered)

rettichschnidi commented 2 years ago

It is good to mention that the easiest way to achieve this is to host the TFPT server on the same machine as the DHCP server.

This is pretty much required as the the dhcp command assumes the TFTP server IP address be the same as the DHCP server one. Not even sure if/how to do it differently.

Thanks again for the work, I really enjoyed it. Now let's see if the mower is still working wink

Good luck! Also, because I will not be able to help there... :zany_face:)

I was wondering: Is there only a repository for the embedded side of the Gardena smart system

Actually "only" for the base system of the gateway. Some services on the gateway as well as the garden device firmwares are proprietary.

... or is there also a place to report bugs for the mobile apps? (There are quite a few problems that I've discovered)

The app is proprietary too. For this reason, there is no public repository or the like. The best/official way is to contact support and let them know. If you insist on your feedback being forwarded, it will reach the appropriate people eventually. Naturally, this will not give you (or anyone else) public transparency however.

easybe commented 8 months ago

Instructions on how to restore/unbrick your gateway can now be found in README.md.