Image broken across firmware upgrade/downgrade

[joshuaspence]

# curl -fLSs https://raw.githubusercontent.com/unifi-utilities/uxg-boot/master/ins
tall.sh | sh
 _   ___  ______   ____              _
| | | \ \/ / ___| | __ )  ___   ___ | |_
| | | |\  / |  _  |  _ \ / _ \ / _ \| __|
| |_| |/  \ |_| | | |_) | (_) | (_) | |_
 \___//_/\_\____| |____/ \___/ \___/ \__|

Updating uxg-setup:default (9062591289242d300168bf7f6a88bab281a099c9aa8c6f57c1d0a35433eafc7c) to joshuaspence/uxg-setup:0.1.19
uxg-setup: Trying to pull docker.io/joshuaspence/uxg-setup:0.1.19...
uxg-setup: Getting image source signatures
uxg-setup: Copying blob sha256:1f391b1dc2338ebcd7700598c4fdfa3d4c700a0edbeead61c2c2faa80e3a4950
uxg-setup: Copying blob sha256:c41311a2282cfd734acce14be8115056e17774f8aaff2014d5b424c40bb6f66c
uxg-setup: Copying config sha256:d2adf09eeee85760b38e804fda20dd55ccc40fbb2e6f115c36c26531f654a879
uxg-setup: Writing manifest to image destination
uxg-setup: Storing signatures
uxg-setup: d2adf09eeee85760b38e804fda20dd55ccc40fbb2e6f115c36c26531f654a879
uxg-setup: Stopping uxg-setup
uxg-setup: f83b244967ad78ff5c321236dff0b57e46750c80cb6bb9532517576ad01c5612
uxg-setup: f83b244967ad78ff5c321236dff0b57e46750c80cb6bb9532517576ad01c5612
uxg-setup: Error: error parsing image data "d2adf09eeee85760b38e804fda20dd55ccc40fbb2e6f115c36c26531f654a879": readlink /mnt/data/podman/storage/overlay/l/T4JFBUV6G3YAD543QDQQERXJNN: no such file or directory
uxg-setup: Starting uxg-setup
uxg-setup: podman create -d --name uxg-setup --label version= --network=host --privileged  --stop-timeout=60 --mount type=bind,source=/usr/lib/version,target=/usr/lib/version,ro=true --mount type=bind,source=/mnt/data/ssh/id_rsa,target=/root/.ssh/id_rsa,ro=true --mount type=bind,source=/etc,target=/etc_host,ro=true --mount type=bind,source=/usr/bin/ubios-udapi-client,target=/bin/ubios-udapi-client,ro=true --mount type=bind,source=/usr/bin/mca-dump,target=/usr/bin/mca-dump,ro=true --mount type=bind,source=/sbin/ubnt-systool,target=/sbin/ubnt-systool,ro=true --mount type=bind,source=/usr/sbin/ntpd,target=/usr/sbin/ntpd,ro=true uxg-setup:default
uxg-setup: Error: readlink /mnt/data/podman/storage/overlay/l/T4JFBUV6G3YAD543QDQQERXJNN: no such file or directory
uxg-setup: Starting container from uxg-setup:default failed
uxg-setup: Error: unable to find container uxg-setup: no container with name or ID uxg-setup found: no such container
uxg-setup: Starting uxg-setup attempt 1 failed
uxg-setup: podman create -d --name uxg-setup --label version= --network=host --privileged  --stop-timeout=60 --mount type=bind,source=/usr/lib/version,target=/usr/lib/version,ro=true --mount type=bind,source=/mnt/data/ssh/id_rsa,target=/root/.ssh/id_rsa,ro=true --mount type=bind,source=/etc,target=/etc_host,ro=true --mount type=bind,source=/usr/bin/ubios-udapi-client,target=/bin/ubios-udapi-client,ro=true --mount type=bind,source=/usr/bin/mca-dump,target=/usr/bin/mca-dump,ro=true --mount type=bind,source=/sbin/ubnt-systool,target=/sbin/ubnt-systool,ro=true --mount type=bind,source=/usr/sbin/ntpd,target=/usr/sbin/ntpd,ro=true uxg-setup:default
uxg-setup: Error: readlink /mnt/data/podman/storage/overlay/l/T4JFBUV6G3YAD543QDQQERXJNN: no such file or directory
uxg-setup: Starting container from uxg-setup:default failed
uxg-setup: Error: unable to find container uxg-setup: no container with name or ID uxg-setup found: no such container
uxg-setup: Starting uxg-setup attempt 2 failed

[joshuaspence]

The uxg-setup images from 1.11.0 and 1.12.19 have different hashes, but seem to be identical internally.

# container-diff diff --type file --type history --type layer --type metadata --type node --type size --type sizelayer cache/uxgpro-1.11.0/input.tar cache/uxgpro-1.12.19/input.tar 

-----File-----

These entries have been added to cache/uxgpro-1.11.0/input.tar: None

These entries have been deleted from cache/uxgpro-1.11.0/input.tar: None

These entries have been changed between cache/uxgpro-1.11.0/input.tar and cache/uxgpro-1.12.19/input.tar: None

-----FileLayer-----

Diff for Layer 0:
These entries have been added to cache/uxgpro-1.11.0/input.tar: None

These entries have been deleted from cache/uxgpro-1.11.0/input.tar: None

These entries have been changed between cache/uxgpro-1.11.0/input.tar and cache/uxgpro-1.12.19/input.tar: None

-----History-----

Docker history lines found only in cache/uxgpro-1.11.0/input.tar: None

Docker history lines found only in cache/uxgpro-1.12.19/input.tar: None

-----Metadata-----

Image metadata differences between cache/uxgpro-1.11.0/input.tar and cache/uxgpro-1.12.19/input.tar:

cache/uxgpro-1.11.0/input.tar None

cache/uxgpro-1.12.19/input.tar None

-----Node-----

Packages found only in cache/uxgpro-1.11.0/input.tar: None

Packages found only in cache/uxgpro-1.12.19/input.tar: None

Version differences: None

-----Size-----

Image size difference between cache/uxgpro-1.11.0/input.tar and cache/uxgpro-1.12.19/input.tar: None

-----SizeLayer-----

Layer size differences between cache/uxgpro-1.11.0/input.tar and cache/uxgpro-1.12.19/input.tar: None

# skopeo inspect docker-archive:cache/uxgpro-1.11.0/input.tar
{
    "Digest": "sha256:177f2b3ea107f286872cee2a48ea2c093c0fb1a667abb9c92df0eddc87c96fc6",
    "RepoTags": [],
    "Created": "2021-12-20T17:04:16.681228372Z",
    "DockerVersion": "",
    "Labels": {
        "version": "0.1.19"
    },
    "Architecture": "arm64",
    "Os": "linux",
    "Layers": [
        "sha256:e4db13cbac2cf2c205ceb9d37b0dd9082e79b68df50a13d89bdcc8dd65c0568e"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "NODE_VERSION=14.15.1",
        "YARN_VERSION=1.22.5"
    ]
}

# skopeo inspect docker-archive:cache/uxgpro-1.12.19/input.tar
{
    "Digest": "sha256:da0d3f75775c09070f79e6830d9d6de0b488086c40827e2e8387088bd52d76aa",
    "RepoTags": [],
    "Created": "2022-04-21T17:59:15.232600482Z",
    "DockerVersion": "",
    "Labels": {
        "version": "0.1.19"
    },
    "Architecture": "arm64",
    "Os": "linux",
    "Layers": [
        "sha256:e4db13cbac2cf2c205ceb9d37b0dd9082e79b68df50a13d89bdcc8dd65c0568e"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "NODE_VERSION=14.15.1",
        "YARN_VERSION=1.22.5"
    ]
}

[joshuaspence]

cache/uxgpro-1.11.0/fs/rootfs/var/lib/containers/storage/overlay/l/UEWN7BC76ZVFILVQERLU6JDTU6 and cache/uxgpro-1.12.19/fs/rootfs/var/lib/containers/storage/overlay/l/T4JFBUV6G3YAD543QDQQERXJNN are both symlinks to ../e4db13cbac2cf2c205ceb9d37b0dd9082e79b68df50a13d89bdcc8dd65c0568e/diff

[joshuaspence]

I think this is containers/podman#5986

[boostchicken]

Is it possible to build a new image where you just touch a file somewhere so the hashes are different? The image digest is different, but the layers are identical. So add another layer to differentiate your builds.

[joshuaspence]

I'm not sure I understand what you mean but I had some success with squashing the image into a single layer. I want to experiment more though as I think something weird is happening. My suspicion is that when a firmware update happens, the system is abruptly rebooted rather than properly shutdown.

I also need to rule out whether this is caused by the scyto/multicast-relay container I am running, which seems to take a few seconds to stop.

[joshuaspence]

In any case, it seems like a newer version of Podman would fix this but I'm unsure how we can do that here since by if Podman can't start the uxg-setup container then the boot scripts (which could update Podman) won't get a chance to run

[boostchicken]

So your image manifests have different guids. However their layers are identical. There have one single layer. This means the images share the same layer and will share the same overlay space on the disk. Flattening the layers is a good call to fix it

Notice both images are comprised of this single layer, meaning that will share an overlay as well.
"Layers": [
        "sha256:e4db13cbac2cf2c205ceb9d37b0dd9082e79b68df50a13d89bdcc8dd65c0568e"
    ],

You absolutely have a chicken or egg problem there, this is why I would not lean on updating Podman as your fix. Make sure you have unique layer digests If you update the image it could impact the running one and cause weirdness.

Also, for the brave there is the udm-kernel project, it allows you to have extremely early customization. But, for every new version you have to ask Ubi for the new source code and then update the tools. I assume UXG and UDM will differ

https://github.com/fabianishere/udm-kernel

[joshuaspence]

Ah, those images I am comparing are the uxg-setup container from two different firmware versions. They wouldn't ordinarily coexist but I guess because I am basing my container off 1.12, when I attempt to downgrade to 1.11 they do coexist.

I'm unclear on why this would be a problem though. If people are running their own containers on the UXG and those containers are based off of the same base, they could share layers too

[joshuaspence]

If anyone ends up in this state, the following script will force uninstall uxg-boot.

# NOTE: This will output an error but it seems to succeed anyway.
podman image list --quiet joshuaspence/uxg-setup | sed -e 's/false\s*$//' | xargs podman image rm --force || true

uxg-setup reset