Add timeout on take_screenshot

GilbN commented 5 months ago

Added

Added dynamic timeout on the screenshot test.
New env variables:
- WEB_SCREENSHOT_TIMEOUT: Replaces WEB_SCREENSHOT_DELAY. default 120s.
- WEB_SCREENSHOT_DELAY: Time in seconds to delay before taking a screenshot after loading the web page. Default 10s
- DOCKER_LOGS_TIMEOUT: Replaces DOCKER_LOGS_DELAY. default 120.
- Added validation of the numeric class attributes.
- Added possibility to add volumes or privilege for future. Currently not in use.
- Added new log level success. Logs with green text.
- Added logging of env data

Changed

Added helper method for adding test results.
Added how long a test takes in seconds.

Deprecated

DOCKER_LOGS_DELAY
DELAY_START

Tests

Lidarr: https://gilbnlsio2.s3.us-east-1.amazonaws.com/linuxserver/lidarr/PR-70/index.html

Healthchecks: https://gilbnlsio2.s3.us-east-1.amazonaws.com/linuxserver/healthchecks/PR-99/index.html

For local testing run:

docker run --rm -i \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /mnt/c/development/docker-ci/output:/ci/output:rw \
-e META_TAG="PR-99" \
-e IMAGE="linuxserver/healthchecks" \
-e TAGS="amd64-v3.3-ls230|arm64v8-v3.3-ls230" \
-e BASE="alpine" \
-e S3_BUCKET="gilbnlsio2" \
-e S3_REGION="eu-north-1" \
-e SECRET_KEY="dummy" \
-e ACCESS_KEY="dummy" \
-e WEB_SCREENSHOT="true" \
-e WEB_SCREENSHOT_TIMEOUT="120" \
-e DOCKER_LOGS_TIMEOUT="120" \
-e SBOM_TIMEOUT="120" \
-e DELAY_START="20" \
-e PORT="8000" \
-e CI_LOG_LEVEL="INFO" \
-e DRY_RUN="true" \
-e DOCKER_ENV="TZ=US/Pacific|SITE_ROOT=http://*" \
-t dockerci:latest \
python3 test_build.py

docker run --rm -i \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /mnt/c/development/docker-ci/output:/ci/output:rw \
-e META_TAG="PR-70" \
-e IMAGE="linuxserver/lidarr" \
-e TAGS="amd64-nightly-2.3.2.4178-ls131|arm64v8-nightly-2.3.2.4178-ls131" \
-e BASE="alpine" \
-e S3_BUCKET="gilbnlsio2" \
-e S3_REGION="eu-north-1" \
-e SECRET_KEY="dummy" \
-e ACCESS_KEY="dummy" \
-e WEB_SCREENSHOT="true" \
-e WEB_SCREENSHOT_TIMEOUT="120" \
-e WEB_AUTH="" \
-e DOCKER_LOGS_TIMEOUT="120" \
-e SBOM_TIMEOUT="120" \
-e PORT="8686" \
-e CI_LOG_LEVEL="INFO" \
-e DRY_RUN="true" \
-t dockerci:latest \
python3 test_build.py

LinuxServer-CI commented 5 months ago

I am a bot, here is the pushed image/manifest for this PR:

ghcr.io/linuxserver/lspipepr-ci:d43cab19-pkg-d43cab19-dev-792527a8824809a6931fc1c608452677b7921ca8-pr-35

LinuxServer-CI commented 5 months ago

I am a bot, here is the pushed image/manifest for this PR:

ghcr.io/linuxserver/lspipepr-ci:d43cab19-pkg-d43cab19-dev-809343661ba1a319d47961b34e5e2a284b8423a3-pr-35

LinuxServer-CI commented 5 months ago

I am a bot, here is the pushed image/manifest for this PR:

ghcr.io/linuxserver/lspipepr-ci:d3aeef46-pkg-d3aeef46-dev-efbea9038858ccf0e349e3bd870beca7993b7615-pr-35

aptalca commented 5 months ago

General thoughts:

We currently have CI_DELAY set in all the repos' jenkins-vars.yml, which currently translates to DELAY_START but it will be deprecated and not used in this PR. We could potentially use the existing value for one of the new vars (perhaps DOCKER_LOGS_TIMEOUT) in the jenkins-builder
Ci container will support new vars, but they need to be added to jenkins-builder first, and then need to be added to downstream repos if they need to be customized.
Should we try to set defaults in jenkinsfiles for the new vars if they're not set in jenkins-vars.yml? If we do, we'll be overriding the defaults in the ci container and make them pointless. Alternatively, we can tentatively set the new vars by checking if they're set first.
Potential issues we had with the timeouts and their status with this PR:
- Webtop containers take a long time for SBOM due to their size and that the package builds saturate the builders. 900s timeout for SBOM should be sufficient for them.
- Some containers have failing arm64 container starts due to not liking qemu. 120s timeout for logs should be sufficient for them. Now that logs and sbom timeouts are decoupled, the qemu failure should have the effective 120s timeout unless overridden

GilbN commented 5 months ago

We currently have CI_DELAY set in all the repos' jenkins-vars.yml, which currently translates to DELAY_START but it will be deprecated and not used in this PR. We could potentially use the existing value for one of the new vars (perhaps DOCKER_LOGS_TIMEOUT) in the jenkins-builder

So something like this?

self.logs_timeout: int = os.environ.get("DOCKER_LOGS_TIMEOUT", os.environ.get("DOCKER_LOGS_DELAY", os.environ.get("DELAY_START ","900")))

DOCKER_LOGS_TIMEOUT -> DOCKER_LOGS_DELAY -> DELAY_START -> 900

aptalca commented 5 months ago

I actually did it on the jenkins-builder side so no changes necessary here: https://github.com/linuxserver/docker-jenkins-builder/pull/255/files#diff-80cf75bfcd603f93321535f40851571b49b190e57b737f22e8a31a9334e74526R1000-R1004

If CI_DOCKER_LOGS_TIMEOUT is set in the project's jenkins-vars.yml, it will use that to set DOCKER_LOGS_TIMEOUT in ci tests. If that's not set, but CI_DELAY exists, it will set DOCKER_LOGS_TIMEOUT to CI_DELAY value in ci tests.

It will provide some backwards compatibility while we update the project repos over time.

LinuxServer-CI commented 5 months ago

I am a bot, here is the pushed image/manifest for this PR:

ghcr.io/linuxserver/lspipepr-ci:d3aeef46-pkg-d3aeef46-dev-77fffdfd60c02993a35d0af86cdb318a5a2a272a-pr-35

LinuxServer-CI commented 5 months ago

I am a bot, here is the pushed image/manifest for this PR:

ghcr.io/linuxserver/lspipepr-ci:d3aeef46-pkg-d3aeef46-dev-24b729cd82601158dec2ee97179bc67effb85a5a-pr-35

linuxserver / docker-ci