Closed joshuaspence closed 2 years ago
The uxg-setup
images from 1.11.0 and 1.12.19 have different hashes, but seem to be identical internally.
# container-diff diff --type file --type history --type layer --type metadata --type node --type size --type sizelayer cache/uxgpro-1.11.0/input.tar cache/uxgpro-1.12.19/input.tar
-----File-----
These entries have been added to cache/uxgpro-1.11.0/input.tar: None
These entries have been deleted from cache/uxgpro-1.11.0/input.tar: None
These entries have been changed between cache/uxgpro-1.11.0/input.tar and cache/uxgpro-1.12.19/input.tar: None
-----FileLayer-----
Diff for Layer 0:
These entries have been added to cache/uxgpro-1.11.0/input.tar: None
These entries have been deleted from cache/uxgpro-1.11.0/input.tar: None
These entries have been changed between cache/uxgpro-1.11.0/input.tar and cache/uxgpro-1.12.19/input.tar: None
-----History-----
Docker history lines found only in cache/uxgpro-1.11.0/input.tar: None
Docker history lines found only in cache/uxgpro-1.12.19/input.tar: None
-----Metadata-----
Image metadata differences between cache/uxgpro-1.11.0/input.tar and cache/uxgpro-1.12.19/input.tar:
cache/uxgpro-1.11.0/input.tar None
cache/uxgpro-1.12.19/input.tar None
-----Node-----
Packages found only in cache/uxgpro-1.11.0/input.tar: None
Packages found only in cache/uxgpro-1.12.19/input.tar: None
Version differences: None
-----Size-----
Image size difference between cache/uxgpro-1.11.0/input.tar and cache/uxgpro-1.12.19/input.tar: None
-----SizeLayer-----
Layer size differences between cache/uxgpro-1.11.0/input.tar and cache/uxgpro-1.12.19/input.tar: None
# skopeo inspect docker-archive:cache/uxgpro-1.11.0/input.tar
{
"Digest": "sha256:177f2b3ea107f286872cee2a48ea2c093c0fb1a667abb9c92df0eddc87c96fc6",
"RepoTags": [],
"Created": "2021-12-20T17:04:16.681228372Z",
"DockerVersion": "",
"Labels": {
"version": "0.1.19"
},
"Architecture": "arm64",
"Os": "linux",
"Layers": [
"sha256:e4db13cbac2cf2c205ceb9d37b0dd9082e79b68df50a13d89bdcc8dd65c0568e"
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NODE_VERSION=14.15.1",
"YARN_VERSION=1.22.5"
]
}
# skopeo inspect docker-archive:cache/uxgpro-1.12.19/input.tar
{
"Digest": "sha256:da0d3f75775c09070f79e6830d9d6de0b488086c40827e2e8387088bd52d76aa",
"RepoTags": [],
"Created": "2022-04-21T17:59:15.232600482Z",
"DockerVersion": "",
"Labels": {
"version": "0.1.19"
},
"Architecture": "arm64",
"Os": "linux",
"Layers": [
"sha256:e4db13cbac2cf2c205ceb9d37b0dd9082e79b68df50a13d89bdcc8dd65c0568e"
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NODE_VERSION=14.15.1",
"YARN_VERSION=1.22.5"
]
}
cache/uxgpro-1.11.0/fs/rootfs/var/lib/containers/storage/overlay/l/UEWN7BC76ZVFILVQERLU6JDTU6
and cache/uxgpro-1.12.19/fs/rootfs/var/lib/containers/storage/overlay/l/T4JFBUV6G3YAD543QDQQERXJNN
are both symlinks to ../e4db13cbac2cf2c205ceb9d37b0dd9082e79b68df50a13d89bdcc8dd65c0568e/diff
I think this is containers/podman#5986
Is it possible to build a new image where you just touch a file somewhere so the hashes are different? The image digest is different, but the layers are identical. So add another layer to differentiate your builds.
I'm not sure I understand what you mean but I had some success with squashing the image into a single layer. I want to experiment more though as I think something weird is happening. My suspicion is that when a firmware update happens, the system is abruptly rebooted rather than properly shutdown.
I also need to rule out whether this is caused by the scyto/multicast-relay
container I am running, which seems to take a few seconds to stop.
In any case, it seems like a newer version of Podman would fix this but I'm unsure how we can do that here since by if Podman can't start the uxg-setup
container then the boot scripts (which could update Podman) won't get a chance to run
So your image manifests have different guids. However their layers are identical. There have one single layer. This means the images share the same layer and will share the same overlay space on the disk. Flattening the layers is a good call to fix it
Notice both images are comprised of this single layer, meaning that will share an overlay as well.
"Layers": [
"sha256:e4db13cbac2cf2c205ceb9d37b0dd9082e79b68df50a13d89bdcc8dd65c0568e"
],
You absolutely have a chicken or egg problem there, this is why I would not lean on updating Podman as your fix. Make sure you have unique layer digests If you update the image it could impact the running one and cause weirdness.
Also, for the brave there is the udm-kernel project, it allows you to have extremely early customization. But, for every new version you have to ask Ubi for the new source code and then update the tools. I assume UXG and UDM will differ
Ah, those images I am comparing are the uxg-setup
container from two different firmware versions. They wouldn't ordinarily coexist but I guess because I am basing my container off 1.12, when I attempt to downgrade to 1.11 they do coexist.
I'm unclear on why this would be a problem though. If people are running their own containers on the UXG and those containers are based off of the same base, they could share layers too
If anyone ends up in this state, the following script will force uninstall uxg-boot
.
# NOTE: This will output an error but it seems to succeed anyway.
podman image list --quiet joshuaspence/uxg-setup | sed -e 's/false\s*$//' | xargs podman image rm --force || true
uxg-setup reset