[CI] Enhance params env check script

jstourac commented 3 weeks ago

[CI] let's run params-env workflow also on push
[CI] enhance the check-params-env.sh to also check uniqueness of values
[CI] check-params-env.sh prints also time of creation of the checked image

More info as description of each commit.

https://issues.redhat.com/browse/RHOAIENG-8812

Description

How Has This Been Tested?

In the root of the repository, run:

./ci/check-params-env.sh
./ci/check-runtime-images.sh

We should backport this to relevant branches upstream and downstream.

[x] The commits are squashed in a cohesive manner and have meaningful messages.
[x] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[x] The developer has manually tested the changes and verified that the changes work

openshift-ci[bot] commented 3 weeks ago

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

jstourac commented 3 weeks ago

This is expected - as it will be fixed by #565.

jstourac commented 3 weeks ago

Technically this one is ready for review. I just need to think whether we shall squash it to a one commit for easier cherry-picking to other branches or not :upside_down_face: :slightly_smiling_face:

jiridanek commented 3 weeks ago

We can always implement the tide/merge-method-squash girlthub label, and then squashing/not squashing can be done by tide, and won't require force-push to PR by author.

jstourac commented 3 weeks ago

I wouldn't do this in bash... But it's already done in bash and it checks useful things.

What would you use instead of bash?

jiridanek commented 3 weeks ago

Python; I think there is enough "logic" in the script for that. But ad I said, since it's in bash already, I'm fine with it.

jstourac commented 3 weeks ago

Since I don't change any functional code, for the record and as an exercise - review the test failures here:

ci/prow/images:

: Build image runtime-cuda-tensorflow-ubi8-python-3.8 from the repository 
{  error occurred handling build runtime-cuda-tensorflow-ubi8-python-3.8-amd64: could not get build runtime-cuda-tensorflow-ubi8-python-3.8-amd64: builds.build.openshift.io "runtime-cuda-tensorflow-ubi8-python-3.8-amd64" not found}

In build log, see:

Pushing image image-registry.openshift-image-registry.svc:5000/ci-op-1ks48214/pipeline:runtime-cuda-tensorflow-ubi8-python-3.8-amd64 ...
Getting image source signatures
Copying blob sha256:7822e944d15c45e998e88e0638073a1974246aea8fd268a925948eb2e070e048
Copying blob sha256:b7029bcb1edaf3277934b59207226ddf843bfbb348a24e5c7e84080488fa35a7
Copying blob sha256:8756f22094d074e5ea7b13b5a7cb8c5132b61a8b39d550f58e6a6053e4b3530d
Copying blob sha256:bea2a0b08f4fd7df72285c8ccf71ff0e9b76c025a0bc4dc67a4f40695feb0eca
Copying blob sha256:b82ddf37e40febb44c258077df217aef2b72f65c2c190ecd3a165ae894256e11
Copying blob sha256:f534e68b04663ce568284dc947b748f2c1793baaecb9573fd8f51a5ae25f1b1b
Copying blob sha256:21c47f62671b8ef160e5748e575f9a5c4642c7b1225c67b9ea7fc0d3417d4747
Error: received unexpected terminate signal

So the build failed at the point when it was being pushed to the image registry.

ci/prow/notebooks-e2e-tests:

: Run multi-stage test notebooks-e2e-tests - notebooks-e2e-tests-jupyter-intel-tf-ubi9-python-3.9-test-e2e container test

In test build log:

bin/kubectl port-forward svc/jupyter-intel-tensorflow-ubi9-python-3-9-notebook 8888:8888 & curl --retry 5 --retry-delay 5 --retry-connrefused http://localhost:8888/notebook/opendatahub/jovyan/api ; EXIT_CODE=$?; echo && pkill --full "^bin/kubectl.*port-forward.*"; \

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Transient problem: connection refused Will retry in 5 seconds. 5 
Warning: retries left.
Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Handling connection for 8888
E0618 22:42:48.486331      84 portforward.go:406] an error occurred forwarding 8888 -> 8888: error forwarding port 8888 to pod ea2566031bc9bfff803ae4f828369012152c6baabd2e85b9b57d28ed32768bda, uid : port forward into network namespace "/var/run/netns/71e5135a-8375-4d71-ac48-bd1227bebf53": failed to connect to localhost:8888 inside namespace ea2566031bc9bfff803ae4f828369012152c6baabd2e85b9b57d28ed32768bda: dial tcp [::1]:8888: connect: connection refused

0     0    0     0    0     0      0      0 --:E0618 22:42:48.486853      84 portforward.go:234] lost connection to pod
--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server

Looks like the expected service isn't listening on the image at expected URL and port.

: Run multi-stage test test phase

Two tests failed - one is same as above, the second one is related to the previous one:

{"component":"entrypoint","error":"wrapped process failed: exit status 2","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2024-06-18T22:42:48Z"}
error: failed to execute wrapped command: exit status 2

openshift-ci[bot] commented 3 weeks ago

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: jiridanek

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/opendatahub-io/notebooks/blob/main/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

jstourac commented 3 weeks ago

Python; I think there is enough "logic" in the script for that. But ad I said, since it's in bash already, I'm fine with it.

Yeah, truth is that this is still 100times better readable for me and easier to maintain than a Python script; but I agree, it's just my problem. We can rewrite this to Python someday if we feel so.

jiridanek commented 3 weeks ago

Nah, would not rewrite, not worth it.

atheo89 commented 3 weeks ago

I will override the tests since are not related with the pr content, to continue with the merging. Thanks Jan for this work!

/override ci/prow/notebooks-e2e-tests /override ci/prow/images

openshift-ci[bot] commented 3 weeks ago

@atheo89: Overrode contexts on behalf of atheo89: ci/prow/images, ci/prow/notebooks-e2e-tests

In response to [this](https://github.com/opendatahub-io/notebooks/pull/567#issuecomment-2178329534): >I will override the tests since are not related with the pr content, to continue with the merging. >Thanks Jan for this work! > >/override ci/prow/notebooks-e2e-tests >/override ci/prow/images Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

jiridanek commented 3 weeks ago

/cherrypick 2024a 2023b 2023a

openshift-cherrypick-robot commented 3 weeks ago

@jiridanek: new pull request created: #575

In response to [this](https://github.com/opendatahub-io/notebooks/pull/567#issuecomment-2179350864): >/cherrypick 2024a 2023b 2023a Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

jiridanek commented 2 weeks ago

/cherrypick 2023b 2023a

openshift-cherrypick-robot commented 2 weeks ago

@jiridanek: #567 failed to apply on top of branch "2023b":

Applying: let's run params-env workflow also on push
Using index info to reconstruct a base tree...
A   .github/workflows/params-env.yaml
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): .github/workflows/params-env.yaml deleted in HEAD and modified in let's run params-env workflow also on push. Version let's run params-env workflow also on push of .github/workflows/params-env.yaml left in tree.
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 let's run params-env workflow also on push
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to [this](https://github.com/opendatahub-io/notebooks/pull/567#issuecomment-2183033873): >/cherrypick 2023b 2023a Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

jiridanek commented 2 weeks ago

/cherrypick 2023a

openshift-cherrypick-robot commented 2 weeks ago

@jiridanek: #567 failed to apply on top of branch "2023a":

Applying: let's run params-env workflow also on push
Using index info to reconstruct a base tree...
A   .github/workflows/params-env.yaml
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): .github/workflows/params-env.yaml deleted in HEAD and modified in let's run params-env workflow also on push. Version let's run params-env workflow also on push of .github/workflows/params-env.yaml left in tree.
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 let's run params-env workflow also on push
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to [this](https://github.com/opendatahub-io/notebooks/pull/567#issuecomment-2183044603): >/cherrypick 2023a Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

opendatahub-io / notebooks

[CI] Enhance params env check script #567

Description

How Has This Been Tested?