Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.
It's confirmed that both of the tests can be successfully executed manually outside of checkbox, but they will fail with permission denied error when being executed by testing with checkbox snap.
This problem can only be reproduced when docker snap is installed on the DUT, using a deb version of docker (docker.io and docker-compose) this problem is not observed.
Execute checkbox tests
$ checkbox.checkbox-cli run com.canonical.certification::docker_resource com.canonical.certification::docker/export-and-import_arm64 com.canonical.certification::docker/save-and-load_arm64
Environment
OS: Ubuntu Classic (both of desktop and server can reproduce)
Checkbox type: snap
Checkbox version: 4.2.0.dev82
Relevant log output
Test result with docker snap
=========[ Running job 2 / 3. Estimated time left (at least): 0:00:00 ]=========
--------------[ Test docker import and export a docker container ]--------------
ID: com.canonical.certification::docker/export-and-import_arm64
Category: com.canonical.certification::docker
... 8< -------------------------------------------------------------------------
write /dev/stdout: permission denied
------------------------------------------------------------------------- >8 ---
Outcome: job failed
=========[ Running job 3 / 3. Estimated time left (at least): 0:00:00 ]=========
------------------[ Test docker save and load a docker image ]------------------
ID: com.canonical.certification::docker/save-and-load_arm64
Category: com.canonical.certification::docker
... 8< -------------------------------------------------------------------------
Using default tag: latest
latest: Pulling from library/ubuntu
Digest: sha256:dfc10878be8d8fc9c61cbff33166cb1d1fe44391539243703c72766894fa834a
Status: Image is up to date for ubuntu:latest
docker.io/library/ubuntu:latest
write /dev/stdout: permission denied
------------------------------------------------------------------------- >8 ---
Outcome: job failed
Finalizing session that hasn't been submitted anywhere: checkbox-run-2024-09-30T04.34.20
==================================[ Results ]===================================
☑ : Configure docker jobs
☑ : Enumerate available system executables
☒ : Test docker import and export a docker container
☒ : Test docker save and load a docker image
Test result with docker deb packages (docker.io and docker-compose)
=========[ Running job 2 / 3. Estimated time left (at least): 0:00:00 ]=========
--------------[ Test docker import and export a docker container ]--------------
ID: com.canonical.certification::docker/export-and-import_arm64
Category: com.canonical.certification::docker
... 8< -------------------------------------------------------------------------
sha256:115cab9d523de75aeb317fbd97de03d779fb5563124886f7d2e13f4972025338
plainbox/ubuntu v1.0 115cab9d523d Less than a second ago 101MB
102ceda73b89ef6a763b23480fced7b08a29634142d291765d264057bfc84c2e
Untagged: plainbox/ubuntu:v1.0
Deleted: sha256:115cab9d523de75aeb317fbd97de03d779fb5563124886f7d2e13f4972025338
Deleted: sha256:aaa3f20569ab06659bacb6662bad77be3e4d9ff5d38309029fb3f688893fed44
------------------------------------------------------------------------- >8 ---
Outcome: job passed
=========[ Running job 3 / 3. Estimated time left (at least): 0:00:00 ]=========
------------------[ Test docker save and load a docker image ]------------------
ID: com.canonical.certification::docker/save-and-load_arm64
Category: com.canonical.certification::docker
... 8< -------------------------------------------------------------------------
Using default tag: latest
latest: Pulling from library/ubuntu
Digest: sha256:dfc10878be8d8fc9c61cbff33166cb1d1fe44391539243703c72766894fa834a
Status: Image is up to date for ubuntu:latest
docker.io/library/ubuntu:latest
Loaded image: ubuntu:latest
ubuntu latest 2b1b17d5e5a2 4 weeks ago 101MB
localhost:5000/ubuntu latest 2b1b17d5e5a2 4 weeks ago 101MB
------------------------------------------------------------------------- >8 ---
Outcome: job passed
Finalizing session that hasn't been submitted anywhere: checkbox-run-2024-09-30T04.26.06
==================================[ Results ]===================================
☑ : Configure docker jobs
☑ : Enumerate available system executables
☑ : Test docker import and export a docker container
☑ : Test docker save and load a docker image
Additional context
This problem is reproducible on both ARM64 and AMD64 arch.
There's no such problem when running with the following scenarios:
Run the docker tests on Ubuntu Core
Installing core confinement checkbox snap on Ubuntu Classic image and execute docker tests
This problem can be reproduced both checkbox in local and remote modes
Bug Description
There are 2 docker related tests failed with permission denied error, which are: docker/export-and-import_{arch} and docker/save-and-load_{arch}
It's confirmed that both of the tests can be successfully executed manually outside of checkbox, but they will fail with permission denied error when being executed by testing with checkbox snap.
This problem can only be reproduced when docker snap is installed on the DUT, using a deb version of docker (docker.io and docker-compose) this problem is not observed.
To Reproduce
$ checkbox.checkbox-cli run com.canonical.certification::docker_resource com.canonical.certification::docker/export-and-import_arm64 com.canonical.certification::docker/save-and-load_arm64
Environment
Relevant log output
Test result with docker snap
Test result with docker deb packages (docker.io and docker-compose)
Additional context