search

compulab-yokneam / meta-mender-compulab

Mender Layer for CompuLab Boards
4 stars 1 forks source link

standalone deployment problems #2

Closed arrow53 closed 3 years ago

arrow53 commented 3 years ago

I'm trying to use mender for standalone deployments with the imx8m plus module.

Trying to install or figure out which image I'm booting to just throws an error

root@ucm-imx8m-plus:~# mender show-artifact
ERRO[0000] Failed to read the current active partition: No match between boot an
d root partitions.: exit status 1 
release-1

I would expect to see a mender_boot_part but I don't see any mender variables in uboot so maybe I don't get the boot process.

root@ucm-imx8m-plus:~# fw_printenv | grep mender                                
root@ucm-imx8m-plus:~#

Do you have any tips for upgrading via .mender files, e.g., standalone. thanks!

vraevsky commented 3 years ago

On a running system) Please issue this command on all rootfs: dpkg -r u-boot-compulab-fw-utils. On the build environment) Comment out the line:

CORE_IMAGE_EXTRA_INSTALL += " u-boot-compulab-fw-utils "

arrow53 commented 3 years ago

@vraevsky ok. This helped. I can now run mender show-artifact. thanks!

I still can't upgrade though and the issue I think is the partition map is different than I would expect. When I flash the sdimg I get the sd card to look like (I've increased the MENDER_STORAGE_TOTAL_SIZE_MB to match the card)

   8        0   31260672 sda
   8        1      65536 sda1
   8        2   13590528 sda2
   8        3   13590528 sda3
   8        4    4005888 sda4

but, after I do mr-deploy I see on the target

root@ucm-imx8m-plus:/data# cat /proc/partitions 
major minor  #blocks  name

  31        0       8192 mtdblock0
 179        0   30535680 mmcblk2
 179        1      85196 mmcblk2p1
 179        2   30437376 mmcblk2p2
 179        3       7168 mmcblk2p3
 179        4       4096 mmcblk2p4

I would expect p2 and p3 to be the same size as they are the rootfs partitions. Obviously, when I try to run mender -install it bails as it's out of space.

Am I using mr-deploy incorrectly?

vraevsky commented 3 years ago

Hi,

Please follow the procedure: 1) bootup the device from a bootable sd-card While in Linux issue: lsblk, save the output; 2) Issue cl-deploy and choose mmcblk2 as the target; don't reboot the device; issue lsblk again. 3) Make sure that emmc p2 & p3 are the same size. 3)Issue cl-uboot & choose mmcblk2boot0 as the target. 4) power off the device, remove the sd-card 5) power on the device. 6) share the results.

BR, Val.

arrow53 commented 3 years ago

@vraevsky on the first lsblk I see

root@ucm-imx8m-plus:~# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mtdblock0     31:0    0    8M  0 disk 
mmcblk2      179:0    0 29.1G  0 disk 
|-mmcblk2p1  179:1    0 83.2M  0 part 
|-mmcblk2p2  179:2    0   29G  0 part 
|-mmcblk2p3  179:3    0    7M  0 part 
`-mmcblk2p4  179:4    0    4M  0 part 
mmcblk2boot0 179:32   0    4M  1 disk 
mmcblk2boot1 179:64   0    4M  1 disk 
mmcblk1      179:96   0 29.8G  0 disk 
|-mmcblk1p1  179:97   0   64M  0 part /boot/efi
|-mmcblk1p2  179:98   0   13G  0 part /
|-mmcblk1p3  179:99   0   13G  0 part 
`-mmcblk1p4  179:100  0  3.8G  0 part /data

After issuing cl-deploy (only mmclk2 appears so selected that) and not rebooting I see this when I exit and run lsblk again

root@ucm-imx8m-plus:~# lsblk                                                    
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mtdblock0     31:0    0    8M  0 disk 
mmcblk2      179:0    0 29.1G  0 disk 
|-mmcblk2p1  179:1    0 83.2M  0 part 
|-mmcblk2p2  179:2    0   29G  0 part 
|-mmcblk2p3  179:3    0    7M  0 part 
`-mmcblk2p4  179:4    0    4M  0 part 
mmcblk2boot0 179:32   0    4M  1 disk 
mmcblk2boot1 179:64   0    4M  1 disk 
mmcblk1      179:96   0 29.8G  0 disk 
|-mmcblk1p1  179:97   0   64M  0 part /boot/efi
|-mmcblk1p2  179:98   0   13G  0 part /
|-mmcblk1p3  179:99   0   13G  0 part 
`-mmcblk1p4  179:100  0  3.8G  0 part /data

I stopped there as this doesn't seem correct.

vraevsky commented 3 years ago

Hi,

Please post the /tmp/cl-deploy log.

BR, Val.

arrow53 commented 3 years ago

cl-deploy.log

@vraevsky here you go

vraevsky commented 3 years ago

Hi,

I'd appreciate it if you could upload the compressed sd-card image and post a link to. One more: please pack and post the ${BUILDDIR}/conf folder. Thank in advance.

arrow53 commented 3 years ago

@vraevsky I don't have a good way to share the sd card image. But, I basically just followed your instructions verbatim here

https://github.com/compulab-yokneam/meta-mender-compulab/tree/gatesgarth

Here is my conf directory

conf.zip

vraevsky commented 3 years ago

I'd appreciate it if you could re-run the cl-deploy procedure, but issue these commands first:

sgdisk --zap-all /dev/mmcblk2 partprobe /dev/mmcblk2

arrow53 commented 3 years ago 
root@ucm-imx8m-plus:~# sgdisk --zap-all /dev/mmcblk2

***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. 
***************************************************************

Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
root@ucm-imx8m-plus:~# partprobe /dev/mmcblk2
root@ucm-imx8m-plus:~#

I get a different output

root@ucm-imx8m-plus:~# lsblk                                                    
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mtdblock0     31:0    0    8M  0 disk 
mmcblk2      179:0    0 29.1G  0 disk 
mmcblk2boot0 179:32   0    4M  1 disk 
mmcblk2boot1 179:64   0    4M  1 disk 
mmcblk1      179:96   0 29.8G  0 disk 
|-mmcblk1p1  179:97   0   64M  0 part /boot/efi
|-mmcblk1p2  179:98   0   13G  0 part /
|-mmcblk1p3  179:99   0   13G  0 part 
`-mmcblk1p4  179:100  0  3.8G  0 part /data
root@ucm-imx8m-plus:~#
arrow53 commented 3 years ago

@vraevsky one thing that is interesting is when I run cl-deploy it seems to not be able to create the partition structure. Do you have hardware to re-create?

this is part of the output I captured before it goes back to the text gui

/dev/mmcblk2p1: Created a new partition 1 of type 'W95 FAT32 (LBA)' and of size 
64 MiB.
Partition #1 contains a vfat signature.
/dev/mmcblk2p2: Created a new partition 2 of type 'Linux' and of size 13 GiB.
/dev/mmcblk2p3: Created a new partition 3 of type 'Linux' and of size 13 GiB.
/dev/mmcblk2p4: No free sectors available.
Failed to add #4 partition: No space left on device
Leaving.

sfdisk: /dev/mmcblk2: cannot modify partition 4: no partition table was found
sfdisk: /dev/mmcblk2: no partition table found
mkfs.fat 4.2 (2021-01-31)
mkfs.vfat: unable to open /dev/mmcblk2p1: No such file or directory
mount: /run/tmp.pja3Xp0Qeq: special device /dev/mmcblk2p1 does not exist.
[ /dev/mmcblk1p1 => /dev/mmcblk2p1 ]: 31.3MiB 0:00:00 [ 449MiB/s] [<=>         ]
umount: /var/run/tmp.pja3Xp0Qeq: not mounted.
sfdisk: /dev/mmcblk2: no partition table found
mke2fs 1.45.6 (20-Mar-2020)
The file /dev/mmcblk2p2 does not exist and no size was specified.
mount: /run/tmp.pja3Xp0Qeq: special device /dev/mmcblk2p2 does not exist.
[ /dev/mmcblk1p2 => /dev/mmcblk2p2 ]:  354MiB 0:00:01 [ 238MiB/s] [<=>         ]
umount: /var/run/tmp.pja3Xp0Qeq: not mounted.
sfdisk: /dev/mmcblk2: no partition table found
mke2fs 1.45.6 (20-Mar-2020)
The file /dev/mmcblk2p3 does not exist and no size was specified.
mount: /run/tmp.pja3Xp0Qeq: special device /dev/mmcblk2p3 does not exist.
[ /dev/mmcblk1p3 => /dev/mmcblk2p3 ]:  250MiB 0:00:10 [28.1MiB/s] [ <=>
arrow53 commented 3 years ago

@vraevsky so the errors above gave me a clue. I reduced the MENDER_STORAGE_TOTAL_SIZE_MB = "14384" and used a 16 GB SD card and now I see the below. The only issue is now my mender partitions are pretty small. I can play around. The EMMC on the board is about the same size as a 32 GB SD card but using the full space of the SD card seems to be a problem for writing. Lemme know if you have any thoughts.

root@ucm-imx8m-plus:~# lsblk                                                    
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mtdblock0     31:0    0    8M  0 disk 
mmcblk2      179:0    0 29.1G  0 disk 
|-mmcblk2p1  179:1    0   64M  0 part 
|-mmcblk2p2  179:2    0    6G  0 part 
|-mmcblk2p3  179:3    0    6G  0 part 
`-mmcblk2p4  179:4    0 17.1G  0 part 
mmcblk2boot0 179:32   0    4M  1 disk 
mmcblk2boot1 179:64   0    4M  1 disk 
mmcblk1      179:96   0 14.8G  0 disk 
|-mmcblk1p1  179:97   0   64M  0 part /boot/efi
|-mmcblk1p2  179:98   0    6G  0 part /
|-mmcblk1p3  179:99   0    6G  0 part 
`-mmcblk1p4  179:100  0  2.8G  0 part /data
vraevsky commented 3 years ago

Fixed: https://github.com/compulab-yokneam/meta-compulab/commit/76199fa1e1a32c9fca6adf8e19bab7419d4f62b9

arrow53 commented 3 years ago

@vraevsky before I close this I'm hoping you can help with me with the size issue. If I flash with a 16 GB SD card it's fine as the EMMC is well above that. But, if I flash with a 32 GB SD card it will always fail with your fix.

A 32G SD is 29.7 GiB but the EMMC is 29.1 GiB so any image created with bmaptool for a SD card that is big enough will fail.

Ideally, what would happen is the mmcblk2p4 would just use whatever leftover space there is.

Do you have any thoughts on how to make that happen? Or, other ideas how to create larger rootfs partitions.

vraevsky commented 3 years ago

@arrow53, it works this way:

Ideally, what would happen is the mmcblk2p4 would just use whatever leftover space there is.

In order to avoid the sfdisk layout issue, just reduse the MENDER_STORAGE_TOTAL_SIZE_MB to fit the emmc size. It works well with 28000.

arrow53 commented 3 years ago

@vraevsky the issue is mender is resizing the data partition on first boot. So, you're right setting the total size does make the SD image that size but it expands it out on first boot. I tried MENDER_DATA_PART_GROWFS = "n" but it didn't seem to have any effect.

vraevsky commented 3 years ago

@arrow53 MENDER_DATA_PART_GROWFS = "n" does not make sense here, 'cause it is the cl-deploy issue.

arrow53 commented 3 years ago

@vraevsky hmm, well, you understand this much better than I do, but i'll just note that this is my sd card prior to first boot

sda                      8:0    1  29.8G  0 disk  
├─sda1                   8:1    1    64M  0 part  
├─sda2                   8:2    1   8.7G  0 part  
├─sda3                   8:3    1   8.7G  0 part  
└─sda4                   8:4    1     2G  0 part

and then after it finishes booting I get this, which causes cl-deploy to fail as the expanded SD image is too big to fit.

mmcblk1      179:96   0 29.8G  0 disk 
|-mmcblk1p1  179:97   0   64M  0 part /boot/efi
|-mmcblk1p2  179:98   0  8.7G  0 part /
|-mmcblk1p3  179:99   0  8.7G  0 part 
`-mmcblk1p4  179:100  0 12.3G  0 part /data

note the change in the data partition

arrow53 commented 3 years ago

@vraevsky got it MENDER_FEATURES_DISABLE_append = " mender-growfs-data "

I'm going to close. As a final note. I'm using cl-deploy but the readme says to use mr-deploy. I have no idea what the difference is

arrow53 commented 3 years ago

@vraevsky not sure whether to keep this here or open new issue so I'll keep for here now. cl-deploy seems to work and I can boot off the emmc. But, after I issue a mender install new_artifact.mender I get errors such as these on reboot and then it just dumps me to an emergency shell. 

[*     ] (1 of 2) A start job is running for��…rtuuid/0dd26200-01 (4s / 1min 30s
[**    ] (1 of 2) A start job is running for��…rtuuid/0dd26200-01 (4s / 1min 30s
[***   ] (1 of 2) A start job is running for��…rtuuid/0dd26200-01 (5s / 1min 30s

Any ideas?

vraevsky commented 3 years ago

@arrow53 boot up the device using the sd-card, then issue: blkid /dev/mmcblk2* mount /dev/mmcblk2p2 /media/ cat /media/etc/fstab mount /dev/mmcblk2p3 /media/ cat /media/etc/fstab

arrow53 commented 3 years ago

@vraevsky here ya go

root@ucm-imx8m-plus:~# blkid /dev/mmcblk2*
/dev/mmcblk2: PTUUID="09501fb1" PTTYPE="dos"
/dev/mmcblk2p1: SEC_TYPE="msdos" UUID="508C-0292" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="09501fb1-01"
/dev/mmcblk2p2: UUID="fd104305-fd92-45ce-9f7e-72f75fff5dce" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="09501fb1-02"
/dev/mmcblk2p3: UUID="e3f6787a-1434-4977-b117-700070053a8e" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="09501fb1-03"
/dev/mmcblk2p4: UUID="4e0cd664-c4fc-4391-815c-46acc72f8970" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="09501fb1-04"
root@ucm-imx8m-plus:~# mount /dev/mmcblk2p2 /media/
root@ucm-imx8m-plus:~# cat /media/etc/fstab
# stock fstab - you probably want to override this with a machine specific one

/dev/root            /                    auto       defaults              1  1
proc                 /proc                proc       defaults              0  0
devpts               /dev/pts             devpts     mode=0620,ptmxmode=0666,gid=5      0  0
tmpfs                /run                 tmpfs      mode=0755,nodev,nosuid,strictatime 0  0
tmpfs                /var/volatile        tmpfs      defaults              0  0

# uncomment this if your device has a SD/MMC/Transflash slot
#/dev/mmcblk0p1       /media/card          auto       defaults,sync,noauto  0  0

# Where the U-Boot environment resides; for devices with SD card support ONLY!
PARTUUID=09501fb1-01 /boot/efi            auto       defaults,sync         0  2
PARTUUID=09501fb1-04 /data                auto       defaults              0  2
root@ucm-imx8m-plus:~# mount /dev/mmcblk2p3 /media/
root@ucm-imx8m-plus:~# cat /media/etc/fstab
# stock fstab - you probably want to override this with a machine specific one

/dev/root            /                    auto       defaults              1  1
proc                 /proc                proc       defaults              0  0
devpts               /dev/pts             devpts     mode=0620,ptmxmode=0666,gid=5      0  0
tmpfs                /run                 tmpfs      mode=0755,nodev,nosuid,strictatime 0  0
tmpfs                /var/volatile        tmpfs      defaults              0  0

# uncomment this if your device has a SD/MMC/Transflash slot
#/dev/mmcblk0p1       /media/card          auto       defaults,sync,noauto  0  0

# Where the U-Boot environment resides; for devices with SD card support ONLY!
PARTUUID=0dd26200-01 /boot/efi            auto       defaults,sync         0  2
PARTUUID=0dd26200-04 /data                auto       defaults              0  2
vraevsky commented 3 years ago

@arrow53, in order to fix the issue please boot up the device using the sd-card, then issue: DST=/dev/mmcblk2 source /usr/local/bin/cl-deploy.mod

In the current setup please issue this command after each mender-artifact install command.

Fix will be provided latter.

arrow53 commented 3 years ago

@vraevsky ok. I think I'm all set now. I'll just leave open and once the fix is in place you can just close. That will indicate to me I don't have to issue the extra command anymore. thanks!

vraevsky commented 3 years ago

Fixed: https://github.com/compulab-yokneam/meta-mender-compulab/commit/ae616f830d6b4f13dd37a923960b007a23ab32ca

arrow53 commented 3 years ago

Tested a few times and everything looks good. Thank again for the help.