search

containers / toolbox

Tool for interactive command line environments on Linux
https://containertoolbx.org/
Apache License 2.0
2.38k stars 208 forks source link

.zuul: Enable testing on Fedora 40 #1468

Closed debarshiray closed 2 months ago

softwarefactory-project-zuul[bot] commented 3 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/457baa233ee949baa5e852fd55e58e57

:heavy_check_mark: unit-test SUCCESS in 5m 05s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 35s :heavy_check_mark: unit-test-restricted SUCCESS in 4m 07s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 34m 03s :x: system-test-fedora-40 TIMED_OUT in 1h 00m 20s :heavy_check_mark: system-test-fedora-39 SUCCESS in 34m 35s :heavy_check_mark: system-test-fedora-38 SUCCESS in 34m 23s

debarshiray commented 3 months ago

recheck

softwarefactory-project-zuul[bot] commented 3 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/ea568dffd0ec4d4fb025df7883d85eda

:heavy_check_mark: unit-test SUCCESS in 4m 46s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 14s :heavy_check_mark: unit-test-restricted SUCCESS in 4m 45s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 33m 43s :x: system-test-fedora-40 TIMED_OUT in 1h 00m 25s :heavy_check_mark: system-test-fedora-39 SUCCESS in 34m 35s :heavy_check_mark: system-test-fedora-38 SUCCESS in 33m 22s

softwarefactory-project-zuul[bot] commented 3 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/faf6e71e3a1c4623aa7de40625142eef

:heavy_check_mark: unit-test SUCCESS in 5m 03s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 29s :heavy_check_mark: unit-test-restricted SUCCESS in 4m 15s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 35m 00s :x: system-test-fedora-40 TIMED_OUT in 1h 20m 28s :heavy_check_mark: system-test-fedora-39 SUCCESS in 35m 42s :heavy_check_mark: system-test-fedora-38 SUCCESS in 34m 41s

debarshiray commented 3 months ago

recheck

softwarefactory-project-zuul[bot] commented 3 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/4132e3e669db41dfb78c2f24e8d0faca

:heavy_check_mark: unit-test SUCCESS in 4m 51s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 38s :heavy_check_mark: unit-test-restricted SUCCESS in 4m 07s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 34m 15s :x: system-test-fedora-40 TIMED_OUT in 1h 20m 24s :heavy_check_mark: system-test-fedora-39 SUCCESS in 33m 41s :heavy_check_mark: system-test-fedora-38 SUCCESS in 33m 41s

debarshiray commented 3 months ago

recheck

softwarefactory-project-zuul[bot] commented 3 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/e4317ecd416746898f59eb190074fb78

:heavy_check_mark: unit-test SUCCESS in 5m 16s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 32s :heavy_check_mark: unit-test-restricted SUCCESS in 3m 57s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 36m 05s :x: system-test-fedora-40 TIMED_OUT in 1h 20m 26s :heavy_check_mark: system-test-fedora-39 SUCCESS in 35m 22s :heavy_check_mark: system-test-fedora-38 SUCCESS in 35m 04s

debarshiray commented 3 months ago

@danpawlik , @TristanCacqueray do you have any idea what's wrong with the cloud-fedora-40 instances? It says that it's timing out, but that's not the real problem. It has the same timeout of 4800 seconds as Fedora Rawhide, which is more than the 3600 seconds that we use for the other stable Fedoras.

It gets stuck setting up the test suite and doesn't ever progress:

2024-03-13 15:22:14.061987 | TASK [Run system tests]
2024-03-13 15:22:17.033732 | fedora-40 | 1..340
2024-03-13 15:22:17.089197 | fedora-40 | # test suite: Set up
2024-03-13 16:37:39.488832 | RUN END RESULT_TIMED_OUT: [untrusted : github.com/containers/toolbox/playbooks/system-test.yaml@main]
danpawlik commented 3 months ago

hi @debarshiray . No idea, I will try to take a look soon/tomorrow.

danpawlik commented 3 months ago

recheck

softwarefactory-project-zuul[bot] commented 3 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/e3c5ba0d9acf44afbbb0a5072df1ca88

:heavy_check_mark: unit-test SUCCESS in 5m 09s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 40s :heavy_check_mark: unit-test-restricted SUCCESS in 3m 43s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 34m 51s :x: system-test-fedora-40 FAILURE in 9m 20s :heavy_check_mark: system-test-fedora-39 SUCCESS in 34m 29s :heavy_check_mark: system-test-fedora-38 SUCCESS in 36m 00s

danpawlik commented 3 months ago

by using command:

bats --timing ./test/system -x

I spotted, that it stuck on:

$ run "$PODMAN" --root "${DOCKER_REG_ROOT}" run -d --rm --name "${DOCKER_REG_NAME}" --privileged -v "${DOCKER_REG_AUTH_DIR}":/auth -e REGISTRY_AUTH=htpasswd -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH="/auth/htpasswd" -v "${DOCKER_REG_CERTS_DIR}":/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -p 50000:443 "${IMAGES[docker-reg]}"
$ assert_success
$ run "$PODMAN" login --authfile "${TEMP_BASE_DIR}/authfile.json" --username user --password user "${DOCKER_REG_URI}"
$ assert_success
$ run "$SKOPEO" copy --dest-authfile "${TEMP_BASE_DIR}/authfile.json" dir:"${IMAGE_CACHE_DIR}"/fedora-toolbox-34 docker://"${DOCKER_REG_URI}"/fedora-toolbox:34  #### HERE ####

so it might be not working on F40. Try to bump the base image to f38

danpawlik commented 3 months ago

@debarshiray did you try with updated images?

debarshiray commented 3 months ago

@debarshiray did you try with updated images?

Sorry, I got pulled away by some other Fedora 40 and internal deadlines. Let me try now. Thanks for confirming that there's nothing wrong with Zuul.

softwarefactory-project-zuul[bot] commented 3 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/394efbc1975c4b9f9fb104fc23d5ce8c

:heavy_check_mark: unit-test SUCCESS in 5m 15s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 50s :heavy_check_mark: unit-test-restricted SUCCESS in 3m 45s :x: system-test-fedora-rawhide TIMED_OUT in 1h 20m 20s :x: system-test-fedora-40 TIMED_OUT in 1h 20m 28s :heavy_check_mark: system-test-fedora-39 SUCCESS in 34m 36s :heavy_check_mark: system-test-fedora-38 SUCCESS in 33m 37s

debarshiray commented 3 months ago

I spotted, that it stuck on:

$ run "$PODMAN" --root "${DOCKER_REG_ROOT}" run -d --rm --name "${DOCKER_REG_NAME}" --privileged -v "${DOCKER_REG_AUTH_DIR}":/auth -e REGISTRY_AUTH=htpasswd -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH="/auth/htpasswd" -v "${DOCKER_REG_CERTS_DIR}":/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -p 50000:443 "${IMAGES[docker-reg]}"
$ assert_success
$ run "$PODMAN" login --authfile "${TEMP_BASE_DIR}/authfile.json" --username user --password user "${DOCKER_REG_URI}"
$ assert_success
$ run "$SKOPEO" copy --dest-authfile "${TEMP_BASE_DIR}/authfile.json" dir:"${IMAGE_CACHE_DIR}"/fedora-toolbox-34 docker://"${DOCKER_REG_URI}"/fedora-toolbox:34  #### HERE ####

I added a debug commit to run the tests with bats --trace ..., and it shows me the same thing.

Interestingly, it also gets stuck on Fedora Rawhide now, in addition to F40.

so it might be not working on F40. Try to bump the base image to f38

It's trying to use skopeo(1) to upload a fedora-toolbox:34 image that was already downloaded into a local directory onto a temporary local Docker registry created by the test suite. So, I don't think the fedora-toolbox:34 image disappeared from registry.fedoraproject.org, because the image got downloaded. For some reason the skopeo copy ... can't upload it.

I think I will need a Fedora 40 system to debug this. Fedora 40 Beta will be released later today, so it's good timing. :)

For context:

We download a bunch of images using skopeo copy ... while setting up the test suite, and cache them in a separate directory using the dir: notation, where they don't show up in podman images, etc.. Then we use skopeo copy ... to either place them in containers-storage: to make them visible to Podman, or to upload them to the temporary local Docker registry created by the test suite, as necessary. This way each test can start with a clean slate, but we don't have to repeatedly download the images again and again.

softwarefactory-project-zuul[bot] commented 3 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/a9ebc15b1a5a47c1bfe15970c120cb3e

:heavy_check_mark: unit-test SUCCESS in 4m 53s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 38s :heavy_check_mark: unit-test-restricted SUCCESS in 3m 48s :x: system-test-fedora-rawhide FAILURE in 35m 56s :x: system-test-fedora-40 FAILURE in 34m 21s :x: system-test-fedora-39 FAILURE in 35m 39s :x: system-test-fedora-38 FAILURE in 34m 50s

softwarefactory-project-zuul[bot] commented 2 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/52cdf068d4044d36a106fdd7ca223206

:heavy_check_mark: unit-test SUCCESS in 6m 37s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 15s :heavy_check_mark: unit-test-restricted SUCCESS in 5m 53s :x: system-test-fedora-rawhide TIMED_OUT in 1h 20m 31s :x: system-test-fedora-40 TIMED_OUT in 1h 20m 26s :heavy_check_mark: system-test-fedora-39 SUCCESS in 35m 52s :heavy_check_mark: system-test-fedora-38 SUCCESS in 36m 28s

softwarefactory-project-zuul[bot] commented 2 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/3ca6f98f3a69446f910491ec2fb43f0c

:heavy_check_mark: unit-test SUCCESS in 6m 55s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 2m 48s :heavy_check_mark: unit-test-restricted SUCCESS in 5m 35s :x: system-test-fedora-rawhide TIMED_OUT in 1h 20m 21s :x: system-test-fedora-40 TIMED_OUT in 1h 20m 27s :heavy_check_mark: system-test-fedora-39 SUCCESS in 47m 56s :heavy_check_mark: system-test-fedora-38 SUCCESS in 38m 16s

danpawlik commented 2 months ago

recheck

softwarefactory-project-zuul[bot] commented 2 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/b7bd5f1282cc4637b7138984f06a7cee

:heavy_check_mark: unit-test SUCCESS in 6m 51s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 2m 54s :heavy_check_mark: unit-test-restricted SUCCESS in 5m 37s :x: system-test-fedora-rawhide TIMED_OUT in 1h 20m 27s :x: system-test-fedora-40 TIMED_OUT in 1h 20m 19s :heavy_check_mark: system-test-fedora-39 SUCCESS in 33m 30s :heavy_check_mark: system-test-fedora-38 SUCCESS in 32m 31s

debarshiray commented 2 months ago

There's this spew coming from the system tests on Fedora Rawhide:

TASK [Run system tests]
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 7: TEMP_BASE_DIR: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 8: TEMP_STORAGE_DIR: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 10: IMAGE_CACHE_DIR: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 11: ROOTLESS_PODMAN_STORE_DIR: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 12: ROOTLESS_PODMAN_RUNROOT_DIR: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 13: PODMAN_STORE_CONFIG_FILE: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 14: DOCKER_REG_ROOT: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 15: DOCKER_REG_CERTS_DIR: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 16: DOCKER_REG_AUTH_DIR: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 17: DOCKER_REG_URI: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 18: DOCKER_REG_NAME: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 21: PODMAN: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 22: TOOLBX: readonly variable
fedora-rawhide | /home/zuul-worker/src/github.com/containers/toolbox/test/system/libs/helpers.bash: line 23: SKOPEO: readonly variable
...
fedora-rawhide | 1..340

This is caused by a regression in Bats 1.11.0. So far it's harmless other than the visual noise.

debarshiray commented 2 months ago

It's trying to use skopeo(1) to upload a fedora-toolbox:34 image that was already downloaded into a local directory onto a temporary local Docker registry created by the test suite. So, I don't think the fedora-toolbox:34 image disappeared from registry.fedoraproject.org, because the image got downloaded. For some reason the skopeo copy ... can't upload it.

I think I will need a Fedora 40 system to debug this. Fedora 40 Beta will be released later today, so it's good timing. :)

For context:

We download a bunch of images using skopeo copy ... while setting up the test suite, and cache them in a separate directory using the dir: notation, where they don't show up in podman images, etc.. Then we use skopeo copy ... to either place them in containers-storage: to make them visible to Podman, or to upload them to the temporary local Docker registry created by the test suite, as necessary. This way each test can start with a clean slate, but we don't have to repeatedly download the images again and again.

I filed a Skopeo issue for this: https://github.com/containers/podman/issues/22575

debarshiray commented 2 months ago

recheck

softwarefactory-project-zuul[bot] commented 2 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/27afdd9b622c41d88c647a43f4db1b86

:heavy_check_mark: unit-test SUCCESS in 6m 38s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 7m 51s :heavy_check_mark: unit-test-restricted SUCCESS in 5m 47s :x: system-test-fedora-rawhide FAILURE in 7m 31s :x: system-test-fedora-40 FAILURE in 7m 29s :heavy_check_mark: system-test-fedora-39 SUCCESS in 35m 19s :heavy_check_mark: system-test-fedora-38 SUCCESS in 35m 16s

TristanCacqueray commented 2 months ago

Not sure if the failure is caused by a missing requirement or a test tool, but that error looks legitimate: Error: could not find slirp4netns, the network namespace can't be configured: exec: "slirp4netns": executable file not found in $PATH

debarshiray commented 2 months ago

Not sure if the failure is caused by a missing requirement or a test tool, but that error looks legitimate: Error: could not find slirp4netns, the network namespace can't be configured: exec: "slirp4netns": executable file not found in $PATH

Yeah, it's caused by slirp4nets being changed to a Suggests in containers-common-extra.

softwarefactory-project-zuul[bot] commented 2 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/6a857e23a9f34101a98f90fd8e0ad8d8

:x: unit-test POST_FAILURE in 6m 24s :x: unit-test-migration-path-for-coreos-toolbox POST_FAILURE in 3m 26s :x: unit-test-restricted POST_FAILURE in 5m 33s :x: system-test-fedora-rawhide POST_FAILURE in 38m 41s :x: system-test-fedora-40 POST_FAILURE in 36m 36s :x: system-test-fedora-39 POST_FAILURE in 36m 35s :x: system-test-fedora-38 POST_FAILURE in 36m 42s

debarshiray commented 2 months ago

recheck

softwarefactory-project-zuul[bot] commented 2 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/50d32dbf05924553abe2e47d458f2ca7

:x: unit-test POST_FAILURE in 6m 27s :x: unit-test-migration-path-for-coreos-toolbox POST_FAILURE in 3m 14s :x: unit-test-restricted POST_FAILURE in 5m 33s :x: system-test-fedora-rawhide POST_FAILURE in 37m 59s :x: system-test-fedora-40 POST_FAILURE in 36m 23s :x: system-test-fedora-39 POST_FAILURE in 36m 52s :x: system-test-fedora-38 POST_FAILURE in 36m 53s

TristanCacqueray commented 2 months ago

recheck ci outage because of no space left on logserver

softwarefactory-project-zuul[bot] commented 2 months ago

Build succeeded. https://softwarefactory-project.io/zuul/t/local/buildset/60cdb252d3a448f792e0eebe2f5cd9b5

:heavy_check_mark: unit-test SUCCESS in 6m 33s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 27s :heavy_check_mark: unit-test-restricted SUCCESS in 5m 39s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 37m 10s :heavy_check_mark: system-test-fedora-40 SUCCESS in 35m 23s :heavy_check_mark: system-test-fedora-39 SUCCESS in 34m 58s :heavy_check_mark: system-test-fedora-38 SUCCESS in 35m 00s

debarshiray commented 2 months ago

recheck ci outage because of no space left on logserver

I see. Thanks for fixing that!

softwarefactory-project-zuul[bot] commented 2 months ago

Build succeeded. https://softwarefactory-project.io/zuul/t/local/buildset/45e3f5373efd44818feaa7818614e280

:heavy_check_mark: unit-test SUCCESS in 7m 28s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 21s :heavy_check_mark: unit-test-restricted SUCCESS in 6m 13s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 44m 51s :heavy_check_mark: system-test-fedora-39 SUCCESS in 43m 55s :heavy_check_mark: system-test-fedora-38 SUCCESS in 44m 30s

softwarefactory-project-zuul[bot] commented 2 months ago

Build succeeded. https://softwarefactory-project.io/zuul/t/local/buildset/5ea1534a9f81423088dfb4b5fe52b953

:heavy_check_mark: unit-test SUCCESS in 6m 44s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 24s :heavy_check_mark: unit-test-restricted SUCCESS in 5m 52s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 39m 01s :heavy_check_mark: system-test-fedora-40 SUCCESS in 34m 19s :heavy_check_mark: system-test-fedora-39 SUCCESS in 37m 18s :heavy_check_mark: system-test-fedora-38 SUCCESS in 34m 49s

softwarefactory-project-zuul[bot] commented 2 months ago

Build succeeded. https://softwarefactory-project.io/zuul/t/local/buildset/ce725485b49c4b859723f61bcdefafc2

:heavy_check_mark: unit-test SUCCESS in 7m 02s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 18s :heavy_check_mark: unit-test-restricted SUCCESS in 5m 45s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 51m 54s :heavy_check_mark: system-test-fedora-40 SUCCESS in 50m 27s :heavy_check_mark: system-test-fedora-39 SUCCESS in 50m 48s :heavy_check_mark: system-test-fedora-38 SUCCESS in 49m 46s

debarshiray commented 2 months ago

It's trying to use skopeo(1) to upload a fedora-toolbox:34 image that was already downloaded into a local directory onto a temporary local Docker registry created by the test suite. So, I don't think the fedora-toolbox:34 image disappeared from registry.fedoraproject.org, because the image got downloaded. For some reason the skopeo copy ... can't upload it. I think I will need a Fedora 40 system to debug this. Fedora 40 Beta will be released later today, so it's good timing. :) For context: We download a bunch of images using skopeo copy ... while setting up the test suite, and cache them in a separate directory using the dir: notation, where they don't show up in podman images, etc.. Then we use skopeo copy ... to either place them in containers-storage: to make them visible to Podman, or to upload them to the temporary local Docker registry created by the test suite, as necessary. This way each test can start with a clean slate, but we don't have to repeatedly download the images again and again.

I filed a Skopeo issue for this: containers/podman#22575

This was worked around by forcing the use of slirp4nets(1).