search

Islandora-Devops / isle-buildkit

Provides a number of Docker images which can be used to build an Islandora site. See also https://github.com/Islandora-Devops/isle-dc
https://www.islandora.ca/
MIT License
13 stars 23 forks source link

Tests timeout on CI/CD server #136

Closed nigelgbanks closed 3 years ago

nigelgbanks commented 3 years ago

Bump up the default time out for tests to something larger than 30 seconds to prevent errors with GitHub actions.

nigelgbanks commented 3 years ago

It isn't a test time out issue, the issue stems from a service exiting before all services are started, and s6 overlay is overriding the exit code set by the exited service.

2021-04-13T08:16:25.0518935Z base_tests_serviceexitcode_base_1 | [services.d] starting services
2021-04-13T08:16:25.0520158Z base_tests_serviceexitcode_base_1 | [services.d] service test finish: executing...
2021-04-13T08:16:25.0522010Z base_tests_serviceexitcode_base_1 | [services.d] service test exiting with exit code: 11Creating network "_base_tests_serviceexitcode_default" with the default driver
2021-04-13T08:16:25.0523340Z Creating base_tests_serviceexitcode_base_1 ... 
2021-04-13T08:16:25.0524098Z Creating base_tests_serviceexitcode_base_1 ... done
2021-04-13T08:16:25.0524548Z 
2021-04-13T08:16:25.0525326Z base_tests_serviceexitcode_base_1 | [services.d] done.

Note that [services.d] done. occurs after the service has exited. So it overrides the exit code here:

https://github.com/just-containers/s6-overlay/blob/7893221d99d43039df3b9f20041e53e7a11b0b34/builder/overlay-rootfs/etc/s6/init/init-stage2#L184-L185

I can't reproduce this locally, though from the logs that is definitely what is happening according to the logs and from reading the code it makes sense, it's a really narrow margin.

nigelgbanks commented 3 years ago

Forcing it to run serially I cannot reproduce the problem locally either.

test.zip

docker run --rm --cpus=1 --cpuset-cpus=0 test; echo $status                                                       274ms  Tue 13 Apr 2021 10:15:05 UTC
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] service test finish: executing...
[services.d] service test exiting with exit code: 11
[services.d] done.
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
11

Though the services does exit now before init-stage2 exits.

nigelgbanks commented 3 years ago

I'm just gonna remove this test.

dannylamb commented 3 years ago

Resolved via https://github.com/Islandora-Devops/isle-buildkit/commit/44265d58c18d7eb0254e64e3ab9ca4b244e267f3