amouiche / qnap_mtd_resize_for_bullseye

Script for resizing MTD partitions on a QNAP device in order to be able to upgrade from buster to bullseye
GNU General Public License v2.0
51 stars 11 forks source link

TS-219P+ - Resize corrupts uboot configuration #13

Closed davefiddes closed 2 years ago

davefiddes commented 2 years ago

Discovered a problem when using qnap_mtd_resize.py at c9ccf4d4b6fe9802d6558b9a88609c68709b2ddf to upgrade my TS-219P+. This unit uses Debian Bullseye with the Buster kernel pinned as per the official release notes. The upgrade appeared to work correctly but when rebooting it failed to boot reporting "Bad Magic" and returning to the u-boot prompt.

The log of the script runing follows:

[Check of the QNAP model and see if supported]
kirkwood-qnap: machine: QNAP TS219 family
DTB file: kirkwood-ts219-6282.dtb
Checking: flashcp -V
Checking: flash_erase --version
ici
Using 'u-boot-tools' package

[find on which MTD device partitions are currently mounted]
   spi0.0

[Dump current U-boot config']
Current U-boot bootcmd:
    cp.l 0xf8200000 0x800000 0x080000;cp.l 0xf8400000 0xa00000 0x240000;bootm 0x800000
Current U-boot bootargs:
    console=ttyS0,115200 root=/dev/ram initrd=0xa00000,0x900000 ramdisk=34816

[Prepare new 'bootcmd']
   Old: cp.l 0xf8200000 0x800000 0x080000;cp.l 0xf8400000 0xa00000 0x240000;bootm 0x800000
   New: cp.l 0xf8100000 0x800000 0xc0000;cp.l 0xf8400000 0xb00000 0x300000;bootm 0x800000;echo Kernel_legacy layout fallback;bootm 0x900000

[Prepare new 'bootargs']
   Old: console=ttyS0,115200 root=/dev/ram initrd=0xa00000,0x900000 ramdisk=34816
   New: console=ttyS0,115200 root=/dev/ram initrd=0xb00000,0xc00000 ramdisk=34816 cmdlinepart.mtdparts="spi0.0:512k@0(uboot)ro,3M@0x100000(Kernel),12M@0x400000(RootFS1),2M@0x200000(Kernel_legacy),256k@0x80000(U-Boot Config),256k@0xc0000(NAS Config)" mtdparts="spi0.0:512k@0(uboot)ro,3M@0x100000(Kernel),12M@0x400000(RootFS1),2M@0x200000(Kernel_legacy),256k@0x80000(U-Boot Config),256k@0xc0000(NAS Config)"

[Prepare fw_setenv script (/tmp/fw_setenv.script)]

[Dump current 'NAS config' and 'Kernel' images]
+ cat /dev/mtd5
[Resize 'NAS config' dump from 1280KB to 256KB.]
+ modprobe loop
+ losetup --show -f /tmp/mtd_nas_config.dump
+ loopdev=/dev/loop0
+ e2fsck -f -p -v /dev/loop0

          17 inodes used (13.28%, out of 128)
           2 non-contiguous files (11.8%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
          51 blocks used (4.98%, out of 1024)
           0 bad blocks
           0 large files

           6 regular files
           2 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
           8 files
+ e2fsck -f -p -v /dev/loop0

          17 inodes used (13.28%, out of 128)
           2 non-contiguous files (11.8%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
          51 blocks used (4.98%, out of 1024)
           0 bad blocks
           0 large files

           6 regular files
           2 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
           8 files
+ resize2fs /dev/loop0 128
resize2fs 1.46.2 (28-Feb-2021)
Resizing the filesystem on /dev/loop0 to 128 (1k) blocks.
The filesystem on /dev/loop0 is now 128 (1k) blocks long.

+ losetup -d /dev/loop0

[Concatenate first 256K of 'NAS config' with first 1MB of Kernel > /tmp/mtd_nas_config.new]

[Prepare second 1MB of kernel tail > /tmp/mtd_kernel.tail]
------------------------------------------------------------
    !!!! Warning !!!!

    Everything is fine up to now.
    Next steps will write the flash and may be subject to failures.

    It is highly recommended to perform a MTD backup and save the files
    somewhere (USB device, PC)

        cat /dev/mtd0 > /tmp/mtd0.uboot.backup
        cat /dev/mtd1 > /tmp/mtd1.kernel.backup
        cat /dev/mtd2 > /tmp/mtd2.rootfs1.backup
        cat /dev/mtd3 > /tmp/mtd3.rootfs2.backup
        cat /dev/mtd4 > /tmp/mtd4.uboot-config.backup
        cat /dev/mtd5 > /tmp/mtd5.nas-config.backup
        fw_printenv -c /tmp/fw_env.config  > /tmp/uboot_config.backup.txt
        cd /tmp
        tar cvzf mtd_backup.tgz mtd?.*.backup uboot_config.backup.txt

        # now use scp / sftp to push/pull mtd_backup.tgz on another PC.

    Be sure you will not cut the power until the end of operations.
    In case of failure, you way need to recover with a Serial Console 
    to run U-boot commands

        https://www.cyrius.com/debian/kirkwood/qnap/ts-219/serial/

Continue and flash the new partitions ? (y/N)
y

[Flash 'NAS config' partition content (ie 'NAS config' + head of Kernel) (still a 'safe' op)]
+ flashcp -v /tmp/mtd_nas_config.new /dev/mtd5
Erasing blocks: 5/5 (100%)
Writing data: 1280k/1280k (100%)
Verifying data: 1280k/1280k (100%)

[Change U-boot config with new values)]
+ fw_setenv -c /tmp/fw_env.config -s /tmp/fw_setenv.script

[Flash tail of the kernel in old 'Kernel' Partition]
+ flashcp -v /tmp/mtd_kernel.tail /dev/mtd1
Erasing blocks: 4/4 (100%)
Writing data: 1024k/1024k (100%)
Verifying data: 1024k/1024k (100%)
------------------------------------------------------------

    SUCCESS. 

    Now, REBOOT !

    Notes: 
    - DO NOT PERFORM A KERNEL OR SYSTEM UPDATE before the next reboot !... 
      so don't wait too long.
    - Consider compressing initrd with 'xz' to optimize the size with:

        echo "COMPRESS=xz" > /etc/initramfs-tools/conf.d/compress

I have a serial console connected so was able to figure out what was happening and temporarily change the bootcmd and bootargs to what the script said they should be and the machine started up.

This was my original configuration: uboot_config.backup.txt

The failing configuration looked like this: qnap_mtd_resize-post-run-config-2021-12-21.txt

I was able to permanently fix the config using fw_setenv after boot so it looks like: fixed-uboot-config-2021-12-21.txt. This now reboots cleanly and I was able to upgrade to the newer Bullseye 5.10 kernel and reboot successfully. Thanks!

amouiche commented 2 years ago

Hello David.

Thank you for reporting this issue. Looks like fw_setenv had some issues to properly configure the uboot env.

as root:     cat /dev/mtd4 > /tmp/mtd4.dump

If you zip those files they will be very small.

Regards, Arnaud

Le 21/12/2021 à 19:38, David J. Fiddes a écrit :

Discovered a problem when using qnap_mtd_resize.py at c9ccf4d https://github.com/amouiche/qnap_mtd_resize_for_bullseye/commit/c9ccf4d4b6fe9802d6558b9a88609c68709b2ddf to upgrade my TS-219P+. This unit uses Debian Bullseye with the Buster kernel pinned as per the official release notes. The upgrade appeared to work correctly but when rebooting it failed to boot reporting "Bad Magic" and returning to the u-boot prompt.

The log of the script runing follows:

|[Check of the QNAP model and see if supported] kirkwood-qnap: machine: QNAP TS219 family DTB file: kirkwood-ts219-6282.dtb Checking: flashcp -V Checking: flash_erase --version ici Using 'u-boot-tools' package [find on which MTD device partitions are currently mounted] spi0.0 [Dump current U-boot config'] Current U-boot bootcmd: cp.l 0xf8200000 0x800000 0x080000;cp.l 0xf8400000 0xa00000 0x240000;bootm 0x800000 Current U-boot bootargs: console=ttyS0,115200 root=/dev/ram initrd=0xa00000,0x900000 ramdisk=34816 [Prepare new 'bootcmd'] Old: cp.l 0xf8200000 0x800000 0x080000;cp.l 0xf8400000 0xa00000 0x240000;bootm 0x800000 New: cp.l 0xf8100000 0x800000 0xc0000;cp.l 0xf8400000 0xb00000 0x300000;bootm 0x800000;echo Kernel_legacy layout fallback;bootm 0x900000 [Prepare new 'bootargs'] Old: console=ttyS0,115200 root=/dev/ram initrd=0xa00000,0x900000 ramdisk=34816 New: console=ttyS0,115200 root=/dev/ram initrd=0xb00000,0xc00000 ramdisk=34816 @.**@*.**@*.**@*.**@*.(U-Boot @.(NAS Config)" @.**@*.**@*.**@*.**@*.(U-Boot @.(NAS Config)" [Prepare fw_setenv script (/tmp/fw_setenv.script)] [Dump current 'NAS config' and 'Kernel' images] + cat /dev/mtd5 [Resize 'NAS config' dump from 1280KB to 256KB.] + modprobe loop + losetup --show -f /tmp/mtd_nas_config.dump + loopdev=/dev/loop0 + e2fsck -f -p -v /dev/loop0 17 inodes used (13.28%, out of 128) 2 non-contiguous files (11.8%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 51 blocks used (4.98%, out of 1024) 0 bad blocks 0 large files 6 regular files 2 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 8 files + e2fsck -f -p -v /dev/loop0 17 inodes used (13.28%, out of 128) 2 non-contiguous files (11.8%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 51 blocks used (4.98%, out of 1024) 0 bad blocks 0 large files 6 regular files 2 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 8 files + resize2fs /dev/loop0 128 resize2fs 1.46.2 (28-Feb-2021) Resizing the filesystem on /dev/loop0 to 128 (1k) blocks. The filesystem on /dev/loop0 is now 128 (1k) blocks long. + losetup -d /dev/loop0 [Concatenate first 256K of 'NAS config' with first 1MB of Kernel > /tmp/mtd_nas_config.new] [Prepare second 1MB of kernel tail > /tmp/mtd_kernel.tail] ------------------------------------------------------------ !!!! Warning !!!! Everything is fine up to now. Next steps will write the flash and may be subject to failures. It is highly recommended to perform a MTD backup and save the files somewhere (USB device, PC) cat /dev/mtd0 > /tmp/mtd0.uboot.backup cat /dev/mtd1 > /tmp/mtd1.kernel.backup cat /dev/mtd2 > /tmp/mtd2.rootfs1.backup cat /dev/mtd3 > /tmp/mtd3.rootfs2.backup cat /dev/mtd4 > /tmp/mtd4.uboot-config.backup cat /dev/mtd5 > /tmp/mtd5.nas-config.backup fw_printenv -c /tmp/fw_env.config > /tmp/uboot_config.backup.txt cd /tmp tar cvzf mtd_backup.tgz mtd?.*.backup uboot_config.backup.txt # now use scp / sftp to push/pull mtd_backup.tgz on another PC. Be sure you will not cut the power until the end of operations. In case of failure, you way need to recover with a Serial Console to run U-boot commands https://www.cyrius.com/debian/kirkwood/qnap/ts-219/serial/ Continue and flash the new partitions ? (y/N) y [Flash 'NAS config' partition content (ie 'NAS config' + head of Kernel) (still a 'safe' op)] + flashcp -v /tmp/mtd_nas_config.new /dev/mtd5 Erasing blocks: 5/5 (100%) Writing data: 1280k/1280k (100%) Verifying data: 1280k/1280k (100%) [Change U-boot config with new values)] + fw_setenv -c /tmp/fw_env.config -s /tmp/fw_setenv.script [Flash tail of the kernel in old 'Kernel' Partition] + flashcp -v /tmp/mtd_kernel.tail /dev/mtd1 Erasing blocks: 4/4 (100%) Writing data: 1024k/1024k (100%) Verifying data: 1024k/1024k (100%) ------------------------------------------------------------ SUCCESS. Now, REBOOT ! Notes: - DO NOT PERFORM A KERNEL OR SYSTEM UPDATE before the next reboot !... so don't wait too long. - Consider compressing initrd with 'xz' to optimize the size with: echo "COMPRESS=xz" > /etc/initramfs-tools/conf.d/compress |

I have a serial console connected so was able to figure out what was happening and temporarily change the bootcmd and bootargs to what the script said they should be and the machine started up.

This was my original configuration: uboot_config.backup.txt https://github.com/amouiche/qnap_mtd_resize_for_bullseye/files/7757290/uboot_config.backup.txt

The failing configuration looked like this: qnap_mtd_resize-post-run-config-2021-12-21.txt https://github.com/amouiche/qnap_mtd_resize_for_bullseye/files/7757323/qnap_mtd_resize-post-run-config-2021-12-21.txt

I was able to permanently fix the config using fw_setenv after boot so it looks like: fixed-uboot-config-2021-12-21.txt https://github.com/amouiche/qnap_mtd_resize_for_bullseye/files/7757329/fixed-uboot-config-2021-12-21.txt. This now reboots cleanly and I was able to upgrade to the newer Bullseye 5.10 kernel and reboot successfully. Thanks!

— Reply to this email directly, view it on GitHub https://github.com/amouiche/qnap_mtd_resize_for_bullseye/issues/13, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXHI5GX3EJV6QN5Y4ZSJIDUSDCSDANCNFSM5KQ2GWKA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

davefiddes commented 2 years ago

I've attached the before and after mtd4 images mtd4.uboot-config.zip. I'm not sure what has happened to the u-boot config over the life of the machine. It may be untouched from the factory but I can't remember.

Hope this shed some light.

Thanks,

Dave

davefiddes commented 2 years ago

I don't have a image from the intermediate state (i.e. immediately after the script ran). Sorry.

cavokz commented 2 years ago

@davefiddes, please have a look at https://github.com/amouiche/qnap_mtd_resize_for_bullseye/issues/9, it could be the very same problem.

amouiche commented 2 years ago

Yes it looks like this issue, but in meantime I fixed it and tested it with in similar conditions (bullseye upgrade + buster kernel)

But I see the issue. I test the "fw_setenv" provider (u-boot-tools for buster vs. libubootenv-tool for bullseye) by testing "fw_setenv -v" exit code

Looks like David was using a specific libubootenv-tool version returning 0.... I need to do another way....

arnaud

Le 22/12/2021 à 08:51, Domenico Andreoli a écrit :

@davefiddes https://github.com/davefiddes, please have a look at #9 https://github.com/amouiche/qnap_mtd_resize_for_bullseye/issues/9, it could be the very same problem.

— Reply to this email directly, view it on GitHub https://github.com/amouiche/qnap_mtd_resize_for_bullseye/issues/13#issuecomment-999358475, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXHI5BIHMWVGBQP4XKEODLUSF7PDANCNFSM5KQ2GWKA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

davefiddes commented 2 years ago

These are the versions of tools I have installed:

libubootenv-tool 0.3.2-0.1
u-boot-tools   2021.01+dfsg-5

This seems to be the current version of these packages.

My fw_setenv -v seems to behave as you suggest:

root@qnap-bullseye:~# fw_setenv -v
fw_setenv: invalid option -- 'v'
root@qnap-bullseye:~# echo $?
0
root@qnap-bullseye:~# 

I checked on a Sheevaplug I have that's still running buster:

sheevaplug:~# fw_setenv -v
Compiled with U-Boot 2019.01+dfsg-7
sheevaplug:~# echo $?
0
sheevaplug:~# 
amouiche commented 2 years ago

Thanks.

I did a fix by changing the way I detect the two version. It is under test

Arnaud

Le 22/12/2021 à 15:54, David J. Fiddes a écrit :

These are the versions of tools I have installed:

|libubootenv-tool 0.3.2-0.1 u-boot-tools 2021.01+dfsg-5 |

This seems to be the current version of these packages.

My |fw_setenv -v| seems to behave as you suggest:

@.:~# fw_setenv -v fw_setenv: invalid option -- 'v' @.:~# echo $? 0 @.***:~# |

I checked on a Sheevaplug I have that's still running buster:

|sheevaplug:~# fw_setenv -v Compiled with U-Boot 2019.01+dfsg-7 sheevaplug:~# echo $? 0 sheevaplug:~# |

— Reply to this email directly, view it on GitHub https://github.com/amouiche/qnap_mtd_resize_for_bullseye/issues/13#issuecomment-999634653, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXHI5CMU5VUSRQ7OBLYJVDUSHRCDANCNFSM5KQ2GWKA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

amouiche commented 2 years ago

the fw_setenv version detection is now fixed in today's version (22 dec)