TX2 partitioning

[lfdmn]

Hi,

I'd like to partition the eMMC a TX2 device to be able to keep data between updates.

For example have a partition mounted to /data and this partition remains untouched after FW upgrade.

I've been looking at the flash.xml and noticed the full image added to the DATA element

I also checked the NVIDIA doc for partitioning the device https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-282/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fpart_config_tx2.html

I also read about WIC it's not really clear the approach to take. Is WIC the tool to use for this? Is it compatible with tegraFlash?

Would anyone have any advice?

Thanks!

[madisongh]

The PARTITION_LAYOUT_TEMPLATE setting in the machine configuration selects the XML file to use for flash layout, so you can provide your own custom copy. What I did for the TX2-based custom carrier we had at my last job was to use the default one from the L4T kit as a starting point, then modified it as needed.

Unfortunately, NV sometimes makes changes to that file from release to release of L4T, and they are not always compatible, so with each BSP upgrade you'll have to refresh the layout. And for the TX2 in particular, be careful with any partition that the NV-provided bootloaders might need access to, since they sometimes make assumptions about size or location (e.g., #134).

As for retaining a partition across upgrades, you'll need some form of OTA update process to do that, since tegraflash will always reformat the eMMC. I had a home-grown system to do that at my last job; on the TX2 it was wrapped around the nv_update_engine provided with the L4T BSP.

[lfdmn]

Thanks again for the advises @madisongh . I'll try to make a custom layout now.

For the system update, our devices don't have access to internet. I've been thinking of using the bitbake generated ipk, zip them, upload the zip to the device, unzip in a location configured to be a local opkg repository and update all the packages in one go to have the dependencies resolved.

At a later stage I'll probably look into modifying uboot to deploy atomic images from USB drive and have it take care of partitions formatting, etc.

[lfdmn]

@madisongh sorry for the basic question but to you see an error here: tegra-bootfiles_28.2.0.bbappend

FILESEXTRAPATHS_prepend_tegra186 := "${THISDIR}/files/tegra186:"
SRC_URI_prepend_tegra186 += " \
    file://flash_l4t_t186_custom.xml \
"

do_preconfigure_prepend_tegra186() {
    mkdir -p ${S}/bootloader/${NVIDIA_BOARD}/cfg/
    cp ${WORKDIR}/flash_l4t_t186_custom.xml ${S}/bootloader/${NVIDIA_BOARD}/cfg/
}

Then PARTITION_LAYOUT_TEMPLATE is set to flash_l4t_t186_custom.xml in the machine definition.

do_preconfigure_prepend_tegra186 is never called and the flash_l4t_t186_custom.xml file never added to the cfg directory.

Also should it be possible to change the flash.xml in the image folder without rebuilding the whole image?

I modified flash.xml.in with these extra partitions at the end of the sdmmc_user device element

        <partition name="overlay" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 4294967296 </size>
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 0x8 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
        </partition>
        <partition name="calibration_data" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 1073741824 </size>
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 0x8 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
        </partition>
        <partition name="app_data" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 11811160064 </size>
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 0x8 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
        </partition>

flash.xml gets updated with the new definitions but the partitions are not created.

[dwalkes]

@lfdmn you might find this example helpful in the mender meta-tegra port: https://github.com/mendersoftware/meta-mender-community/tree/sumo/meta-mender-tegra/recipes-bsp/tegra-binaries

Or this one for warrior: https://github.com/Trellis-Logic/meta-mender-community/tree/warrior/meta-mender-tegra/recipes-bsp/tegra-binaries

Also should it be possible to change the flash.xml in the image folder without rebuilding the whole image?

When I originally was testing these files on the sumo branch the command I was using was: tegradevflash_v2 --pt flash.xml.bin --create

You should be able to extract the tegraflash installer and find the command being used, then iterate with your new partition layout.

[madisongh]

@lfdmn Your bbappend should be for the tegra-binaries recipe, since that's the one that fetches and unpacks the "sources" used by all of the BSP recipes (fetch, unpack, and patch are turned off for the other recipes). Also, you can just use "${THISDIR}/files:" in your FILESEXTRAPATHS prepend, since bitbake will automatically search below there based on other overrides (e.g. tegra186). Otherwise, your preconfigure prepend looks fine, and you shouldn't need to have a bbappend for tegra-bootfiles at all since you've copied your customized XML file into the right directory.

Unfortunately, that does mean that everything that depends on tegra-binaries will get rebuilt every time you change your XML file, which is a fair number of packages.

Make sure you insert your added partitions before the secondary_gpt entry in the XML file, and make sure the last entry has <allocation_attribute> 0x808 </allocation_attribute> on it, which tells tegradevflash to extend the partition out to the end of the available space on the MMC. I ran into some issues with some versions of NV's tools when I didn't do that for one of the custom layouts I was working with before.

As @dwalkes says, you should be able to unpack the tegraflash package and make modifications to the flash.xml.in file there. Just note that that file will have some modifications to it from the original template XML - mainly substituting in file names and some partition size information, as done by the tegraflash_create_flash_config_tegra186() function in image_types_tegra.bbclass... basically the same work that's done by NV's flash.sh script before it calls the tegraflash.py script.

[lfdmn]

Thanks a lot @madisongh and @dwalkes. I made progress

I moved my bb append to tegra-binaries and it's on for a long wait. But the file gets copied to the right place now.

Also updating flash.xml.in was working and I can see the about 30 partitions created in the device:

~ # ll /dev/mmcblk0*
brw-rw----    1 root     disk      179,   0 Jan  1 01:00 /dev/mmcblk0
brw-rw----    1 root     disk      179,   8 Jan  1 01:00 /dev/mmcblk0boot0
brw-rw----    1 root     disk      179,  16 Jan  1 01:00 /dev/mmcblk0boot1
brw-rw----    1 root     disk      179,   1 Jan  1 01:00 /dev/mmcblk0p1
brw-rw----    1 root     disk      259,   2 Jan  1 01:00 /dev/mmcblk0p10
brw-rw----    1 root     disk      259,   3 Jan  1 01:00 /dev/mmcblk0p11
brw-rw----    1 root     disk      259,   4 Jan  1 01:00 /dev/mmcblk0p12
brw-rw----    1 root     disk      259,   5 Jan  1 01:00 /dev/mmcblk0p13
brw-rw----    1 root     disk      259,   6 Jan  1 01:00 /dev/mmcblk0p14
brw-rw----    1 root     disk      259,   7 Jan  1 01:00 /dev/mmcblk0p15
brw-rw----    1 root     disk      259,   8 Jan  1 01:00 /dev/mmcblk0p16
brw-rw----    1 root     disk      259,   9 Jan  1 01:00 /dev/mmcblk0p17
brw-rw----    1 root     disk      259,  10 Jan  1 01:00 /dev/mmcblk0p18
brw-rw----    1 root     disk      259,  11 Jan  1 01:00 /dev/mmcblk0p19
brw-rw----    1 root     disk      179,   2 Jan  1 01:00 /dev/mmcblk0p2
brw-rw----    1 root     disk      259,  12 Jan  1 01:00 /dev/mmcblk0p20
brw-rw----    1 root     disk      259,  13 Jan  1 01:00 /dev/mmcblk0p21
brw-rw----    1 root     disk      259,  14 Jan  1 01:00 /dev/mmcblk0p22
brw-rw----    1 root     disk      259,  15 Jan  1 01:00 /dev/mmcblk0p23
brw-rw----    1 root     disk      259,  16 Jan  1 01:00 /dev/mmcblk0p24
brw-rw----    1 root     disk      259,  17 Jan  1 01:00 /dev/mmcblk0p25
brw-rw----    1 root     disk      259,  18 Jan  1 01:00 /dev/mmcblk0p26
brw-rw----    1 root     disk      259,  19 Jan  1 01:00 /dev/mmcblk0p27
brw-rw----    1 root     disk      259,  20 Jan  1 01:00 /dev/mmcblk0p28
brw-rw----    1 root     disk      259,  21 Jan  1 01:00 /dev/mmcblk0p29
brw-rw----    1 root     disk      179,   3 Jan  1 01:00 /dev/mmcblk0p3
brw-rw----    1 root     disk      179,   4 Jan  1 01:00 /dev/mmcblk0p4
brw-rw----    1 root     disk      179,   5 Jan  1 01:00 /dev/mmcblk0p5
brw-rw----    1 root     disk      179,   6 Jan  1 01:00 /dev/mmcblk0p6
brw-rw----    1 root     disk      179,   7 Jan  1 01:00 /dev/mmcblk0p7
brw-rw----    1 root     disk      259,   0 Jan  1 01:00 /dev/mmcblk0p8
brw-rw----    1 root     disk      259,   1 Jan  1 01:00 /dev/mmcblk0p9
brw-rw----    1 root     disk      179,  24 Jan  1 01:00 /dev/mmcblk0rpmb

And fdisk reports:

# fdisk -l
Disk /dev/mmcblk0: 31.2 GB, 31268536320 bytes
255 heads, 63 sectors/track, 3801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

        Device Boot      Start         End      Blocks  Id System
/dev/mmcblk0p1               1      267350  2147483647+ ee EFI GPT

Disk /dev/mmcblk0boot1: 4 MB, 4194304 bytes
4 heads, 16 sectors/track, 128 cylinders
Units = cylinders of 64 * 512 = 32768 bytes

Disk /dev/mmcblk0boot1 doesn't contain a valid partition table

Disk /dev/mmcblk0boot0: 4 MB, 4194304 bytes
4 heads, 16 sectors/track, 128 cylinders
Units = cylinders of 64 * 512 = 32768 bytes

Disk /dev/mmcblk0boot0 doesn't contain a valid partition table

Do you happen to know how to find the matching partition device to the new partitions added to the flash.xml? Are they allocated in the order they are declared in the xml file?

I'm wondering how partitions mounting is done with yocto / tegra.

At least the fstab doesn't tell much

~ # cat /etc/fstab 
# stock fstab - you probably want to override this with a machine specific one

/dev/root            /                    auto       defaults              1  1
proc                 /proc                proc       defaults              0  0
devpts               /dev/pts             devpts     mode=0620,gid=5       0  0
tmpfs                /run                 tmpfs      mode=0755,nodev,nosuid,strictatime 0  0
tmpfs                /var/volatile        tmpfs      defaults              0  0

# uncomment this if your device has a SD/MMC/Transflash slot
#/dev/mmcblk0p1       /media/card          auto       defaults,sync,noauto  0  0

[dwalkes]

@lfdmn

Do you happen to know how to find the matching partition device to the new partitions added to the flash.xml? Are they allocated in the order they are declared in the xml file?

Yes, although not all are there, presumably because they are hidden GPT partitions. Easiest way to find them is to cross reference size with the partition list and then just try mounting them.

I'm wondering how partitions mounting is done with yocto / tegra. At least the fstab doesn't tell much

Most of the partitions won't be mounted. You can update fstab to include any you'd like mounted. As for why fstab shows /dev/root instead of a specific partition I think it's because it was the rootfs setup by initramfs. To find the corresponding device you can use these instructions. For instance, on my system

root@jetson-tx2:~# stat /
  File: /
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: b301h/45825d    Inode: 2           Links: 20
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-10 05:34:14.000000000
Modify: 2019-07-08 16:50:23.000000000
Change: 2019-07-08 16:50:23.000000000

where b301 is device major 179 (0xb3) and minor 1 so rootfs is on /dev/mmcblk0p1

root@jetson-tx2:~# ls -la /dev/mmcblk0* | grep "179,.* 1 "
brw-rw----    1 root     disk      179,   1 Jul  9 14:18 /dev/mmcblk0p1

For more information about the content of the partitions on tegra you can see the partition configuration page.

[madisongh]

To help with checking on the partition layout, include util-linux-lsblk and either parted or sgdisk. Then include e2fsprogs so you can format the new partition (assuming you want the filesystem to be ext4), and you can add it to your fstab.

[lfdmn]

Hi,

Rehearsing an old topic. lsblk is very useful for finding the partitions

# lsblk --f --bytes --output NAME,FSTYPE,MOUNTPOINT,PARTLABEL,SIZE,STATE,TYPE
NAME         FSTYPE MOUNTPOINT PARTLABEL                SIZE STATE   TYPE
sda                                             128035676160 running disk
`-sda1       ext4   /data                       128034676224         part
mmcblk0                                          31268536320         disk
|-mmcblk0p1  ext4   /          APP               12884901888         part
|-mmcblk0p2                    mts-bootpack          4194304         part
|-mmcblk0p3                    mts-bootpack_b        4194304         part
|-mmcblk0p4                    cpu-bootloader         524288         part
|-mmcblk0p5                    cpu-bootloader_b       524288         part
|-mmcblk0p6                    bootloader-dtb         524288         part
|-mmcblk0p7                    bootloader-dtb_b       524288         part
|-mmcblk0p8                    secure-os             3145728         part
|-mmcblk0p9                    secure-os_b           3145728         part
|-mmcblk0p10                   eks                   2097152         part
|-mmcblk0p11                   adsp-fw               4194304         part
|-mmcblk0p12                   adsp-fw_b             4194304         part
|-mmcblk0p13                   bpmp-fw                618496         part
|-mmcblk0p14                   bpmp-fw_b              618496         part
|-mmcblk0p15                   bpmp-fw-dtb            512000         part
|-mmcblk0p16                   bpmp-fw-dtb_b          512000         part
|-mmcblk0p17                   sce-fw                2097152         part
|-mmcblk0p18                   sce-fw_b              2097152         part
|-mmcblk0p19                   sc7                   6291456         part
|-mmcblk0p20                   sc7_b                 6291456         part
|-mmcblk0p21                   FBNAME                2097152         part
|-mmcblk0p22                   BMP                 134217728         part
|-mmcblk0p23                   BMP_b               134217728         part
|-mmcblk0p24                   SOS                  33554432         part
|-mmcblk0p25                   SOS_b                33554432         part
|-mmcblk0p26                   kernel               67108864         part
|-mmcblk0p27                   kernel_b             67108864         part
|-mmcblk0p28                   kernel-dtb             524288         part
|-mmcblk0p29                   kernel-dtb_b           524288         part
|-mmcblk0p30                   CAC                 268435456         part
`-mmcblk0p31                   UDA               17593871872         part
mmcblk0boot0                                         4194304         disk
mmcblk0boot1                                         4194304         disk
mmcblk0rpmb                                          4194304         disk

I'm trying to mount UDA but it looks like it's not formatted after tegraflash. Any hint if/how this can be done?

I don't see how I could format it during boot if I need it as overlay FS partition.

[madisongh]

There are a couple of ways to deal with this. One would be to create a filesystem image during the build to flash onto that partition... you'd have to modify the XML file to add the name of the file containing the formatted filesystem, and add a tegraflash_custom_pre function to your image recipe to generate that file.

The second would be to format the partition at runtime. To do that automatically during boot, you'd need to have an entry in your /etc/fstab to have the partition mounted and (assuming you're using systemd) either use the x-systemd.makefs option have an x-systemd.requires= option in the entry to have the mount depend on a service you add which would handle formatting the partition if it hasn't already been formatted (or needs to be reformatted if it's become corrupted). You'll need to include e2fsprogs-e2fsck and e2fsprogs-mke2fs in your build (assuming you'll be using an ext4 filesystem).

You can also combine these approaches. A full-sized filesystem image for that partition would be very large and increase flashing time, but you could generate a minimally-sized one and add the x-systemd.growfs option to the fstab entry for it to have systemd automatically grow the size of the filesystem to fill the entire partition before mounting.

There are probably other approaches, or variations on the above, that you could take, depending on what your needs are for your application. I don't have any simple examples to point you at, unfortunately.

[lfdmn]

Thanks again!

I prefer the combined approach.

I created empty images

tegraflash_custom_pre() {
    dd if=/dev/zero of=persistent.img bs=1M count=10
    /sbin/mkfs.ext4 persistent.img

    dd if=/dev/zero of=overlay.img bs=1M count=10
    /sbin/mkfs.ext4 overlay.img
}

updated the template and now I can mount.

Tomorrow I'll make them grow =)

[ENM185]

Hi @madisongh. I'm planning on doing something very similar to what @lfdmn described in this issue. The only difference is that I don't want to have to format the partitions at all on boot.

In my dd command (which is similar to the comment above mine), will I need to update the count to be the full size of the partition?

[dwalkes]

@ENM185

The only difference is that I don't want to have to format the partitions at all on boot.

Do you mean you don't want to rely on x-systemd-growfs or similar commands like resize2fs to grow the filesystem either? If you can use x-systemd-growfs it's as simple as just placing it in your fstab line.

If you can't or don't want to do a resize then I expect you would need to set the size with dd command and deal with a longer flashing time, but maybe @madisongh has other suggestions.

[madisongh]

Right, each filesystem image would have to match the size of the partition. It could be a bit tricky to get that size just right, and once you've got that coded, any tiny change in your flash layout you make later could lead to build and/or flashing failures due to size mismatches, so tread carefully. And as @dwalkes mentions, you'll end up with longer flashing times, too.

You might want to consider a two-step approach of USB-flashing a small initramfs-based installation image that takes care of formatting and loading content into the partitions (either from a server or from bundled compressed tarballs).

[ENM185]

I actually have a sh script running in an initramfs image already (for something else). Is there a good way of getting it to format the partitions, but only doing that on the first boot?

I assume all I need to do in that case is have the partitions in flash.xml (without an associated file), and then run mkfs.ext4 in the initramfs script. Is this correct?

[madisongh]

Essentially, yes. Then just check if they're already formatted (maybe use fsck) before running mkfs, so they're only formatted once.

[ENM185]

Great, thanks for the help!