when using mender and xavier nx rootfs is too large

[arrow53]

For my image I'm running out of space

ERROR: core-image-tern-1.0-r0 do_image_mender: Size of rootfs is greater than the calculated partition space (11337203712 > 6897532928). This image won't fit on a device with the current storage configuration. Try reducing IMAGE_OVERHEAD_FACTOR if it is higher than 1.0, or raise MENDER_STORAGE_TOTAL_SIZE_MB if the device in fact has more storage.

I've tried increasing it with TEGRAFLASH_SDCARD_SIZE or MENDER_STORAGE_TOTAL_SIZE_MB but both result in parse errors that I seem unable to track down.

Does anyone have a hint what variable might work? My SD card is 256G so it's not a physical space problem ;)

[madisongh]

The ROOTFSPART_SIZE variable (from the machine conf file) controls the size of the rootfs partition. That's the value that gets written out as APPSIZE in the flash layout XML file during flashing. The mender-tegra integration should key everything off that value, too.

[arrow53]

@madisongh oh, good tip. So, xavier-nx.inc has ROOTFSPART_SIZE set to 14G and jetson-xavier-nx-devkit.conf has TEGRAFLASH_SDCARD_SIZE set to 16G.

Should I override both in my local.conf? So, if my SD card is actually 32G in my local.conf

TEGRAFLASH_SDCARD_SIZE = "32G"
ROOTFSPART_SIZE = "32212254720"

where 32212254720 is 30G

Or, do I just need ROOTFSPART_SIZE?

[madisongh]

Right, sorry, I forgot that with Mender in the mix, this gets more complicated... tegra-mender-setup.bbclass will try to calculate ROOTFSPART_SIZE for you based on TEGRAFLASH_SDCARD_SIZE and the MENDER-prefixed variables that specify the sizes of the non-rootfs partitions.

So setting TEGRAFLASH_SDCARD_SIZE to 32G should get you a more suitable rootfs partition size. Try it and check the output of bitbake -e core-image-<whatever> to see if the settings look more reasonable to you.

Taking the suggestion in the error message of changing IMAGE_OVERHEAD_FACTOR might help, too - it defaults to 1.3, where 1.1 might work better for really large rootfs partitions.

[arrow53]

haha, yeah, mender does seem to make it much harder.

Changing TEGRAFLASH_SDCARD_SIZE in the machine file to 32G or in my local.conf throws parse errors.

Changing ROOTFSPART_SIZE in the machine file or my local.conf gets me back to the same error about space not being big enough.

IMAGE_OVERHEAD_FACTOR = "1.1" gets me

Size of rootfs is greater than the calculated partition space (8109686784 > 6897532928).

[arrow53]

fwiw, here is the tail or the error when I change TEGRAFLASH_SDCARD_SIZE

| [   1.7687 ] Copying signatures
| [   1.7694 ] tegrahost_v2 --chip 0x19 0 --partitionlayout flash.xml.bin --updatesig images_list_signed.xml
| [   2.5433 ]
| [   2.5442 ] tegraparser_v2 --generategpt --pt flash.xml.bin
| [   2.5448 ] End sector for APP_b, expected at: 33554398, actual: 59203367
| [   2.5455 ]
| Error: Return value 4
| Command tegraparser_v2 --generategpt --pt flash.xml.bin
| WARNING: exit code 1 from a shell command.

[madisongh]

One thing I've done in my builds to bypass all the mender calculation is to use an override, for example:

ROOTFSPART_SIZE_jetson-xavier-nx-devkit = "8589934592"

to get an 8GiB partition size. That overrides the setting in the bbclass, and should take.

[arrow53]

hmm, similar parse error

in local.conf

ROOTFSPART_SIZE_jetson-xavier-nx-devkit = "8589934592"

then

| [   2.5133 ] tegraparser_v2 --generategpt --pt flash.xml.bin
| [   2.5139 ] End sector for APP_b, expected at: 33554398, actual: 34563879
| [   2.5149 ]
| Error: Return value 4
| Command tegraparser_v2 --generategpt --pt flash.xml.bin
| WARNING: exit code 1 from a shell command.

[arrow53]

@dwalkes ROOTFSPART_SIZE_jetson-xavier-nx-devkit works for you with tegra-demo-distro on the xavier nx?

for me I get

| [   2.5321 ] tegraparser_v2 --generategpt --pt flash.xml.bin
| [   2.5327 ] End sector for APP_b, expected at: 33554398, actual: 34563879
| [   2.5334 ]
| Error: Return value 4
| Command tegraparser_v2 --generategpt --pt flash.xml.bin
| WARNING: exit code 1 from a shell command.

[dwalkes]

@arrow53 yes I'm using ROOTFSPART_SIZE with a custom machine type, not sure what would be different about your case. Are you on the dunfell branch?

I just kicked off

source setup-env -m jetson-xavier-nx-devkit -d testdistro-mender
bitbake core-image-base

on https://github.com/OE4T/tegra-demo-distro/tree/dunfell-l4t-r32.4.3 to see if I can reproduce.

[arrow53]

wow, thanks @dwalkes I really appreciate the effort. Just knowing it works for you is very helpful.

I am using that exact workflow. I'll be curious what you find but seems I need to slowly add things from core-image-base (I have my own recipe) until it breaks.

[dwalkes]

I can reproduce, I see there's a problem here with specifying TEGRAFLASH_SDCARD_SIZE=32G in conf/local.conf with no other changes.

not sure what would be different about your case

One difference is the machine I'm using is based on the emmc version.

deploy log for kernel-bup-payload at https://drive.google.com/file/d/1KO1ueO3Lj8u_i1YYx5vSAQ6CwRPOiFOz/view?usp=sharing and flash.xml from build/tmp/work/jetson_xavier_nx_devkit-oe4t-linux/kernel-bup-payload/1.0-r0/bup-payload/flash.xml at https://drive.google.com/file/d/1a9DSxL3UbBBV9SC3s_I3PNyMKjGVE1hM/view?usp=sharing - nothing jumps out at me for either one of these at first glance.

[madisongh]

@arrow53 @dwalkes Check the flash layout XML file you are using for this line:

<device type="sdcard" instance="0" sector_size="512" num_sectors="33554432">

That num_sectors field is set for a 16GB SDcard in the stock XML file for the Xavier NX. Update that value for your larger card. You can get the number of sectors with sgdisk <DEVICENAME> --print or parted <DEVICENAME> unit s print (with sudo if needed). For my 32GB card, I get 623333952 sectors.

[dwalkes]

Ah yes that would do it. I think we should start a page about customizing root filesystem size for mender on tegra at https://github.com/OE4T/tegra-demo-distro/wiki. @arrow53 if you can verify this works and want to get it started please do.

[madisongh]

Yeah, I'd forgotten about that detail, even though I ran into the exact same problem some time back... of course, I never got around to writing it down.

We could perhaps add a build-time check to make sure that the size in the XML file matches the TEGRAFLASH_SDCARD_SIZE setting, or maybe even modify the XML based on the variable setting, at least for some common SDcard sizes.

[arrow53]

@dwalkes @madisongh I think it's fair to assume sector_size is always 512 (I think that's standard for SD/emmc) so then I think the size is just sector_size * num_sectors / 1024 / 1024 / 1024 GBs.

So num_sectors = TEGRAFLASH_SDCARD_SIZE * 1024 * 1024 * 1024 / sector_size

I get a different value than @madisongh got with partd so not sure if mine is right. Seems bad to argue with partd. Mine did build.

At any rate, the two options seem to be some awk'ing PARTITION_LAYOUT_TEMPLATE or adding a note to the wiki about creating one's own or hand touching the existing one.

Do you guys have a preference? Personally, I think just having the user patch kernel-bup-payload would be fine and making a note in the wiki about it. Lemme know if that's cool and I'll do that and close.

[madisongh]

The "G" in the SDcard size is power-of-10, not power-of-2, so use 1000 for your multiplier. There's also some overhead reserved on the card, which reduces the available number of sectors a bit further. Using what parted or sgdisk reports (which is also in sysfs) gives you an accurate number, but as long as the count isn't too far off from actual, it should work OK for SDcard builds. It's more important for the number to be accurate if you're flashing via USB.

A note on the wiki for now would be fine with me.

[arrow53]

I made a page https://github.com/OE4T/tegra-demo-distro/wiki/Customization-of-Xavier-NX

[madisongh]

Thanks @arrow53 !

[arrow53]

I wouldn't be very grateful. it's not a great page. I basically just referred back to here as honestly while it works I'm still confused a bit.

[dwalkes]

I've updated the page with a generic title and linked to the location where the flash.xml files can be customized can now be found at https://github.com/OE4T/tegra-demo-distro/wiki/Customization-of-Storage-Size-and-root-filesystem-size-on-Mender-Builds.

See similar question at https://hub.mender.io/t/nvidia-tegra-jetson-xavier-nx/2615/3

[madisongh]

Thanks @dwalkes. I added a bit more on the specifics of what needs to change in the XML file so folks won't have to click through the links.