[BUG] Not able to install tools from OpenShift Mirror

[divyansh42]

All the nightly checks are failing as none of the tools can be installed from the OpenShift Mirror. We are investigating this problem and will update here as soon as we fix the bug.

[divyansh42]

If your use case satisfies, please feel free to use input source: github to download and install binaries from GitHub instead of OpenShift Mirror.

[tetchel]

it looks like there's been a redirect added:

curl -i https://mirror.openshift.com/pub/openshift-v4/clients/serverless/0.15.2
HTTP/2 302
content-type: application/xml
content-length: 0
date: Thu, 16 Dec 2021 01:07:44 GMT
server: AmazonS3
location: /pub/openshift-v4/x86_64/clients/serverless/0.15.2/
x-cache: Hit from cloudfront
via: 1.1 50faaaa196a6b0875217ef7827f97d7c.cloudfront.net (CloudFront)
x-amz-cf-pop: SEA19-C3
x-amz-cf-id: IjSgDyxzZbDbKEjIhRkSiU33zm1IwdFhHkfqk2OFOD1KPlNZ8Ae6aw==
age: 12813

if I add the -L flag (which follows redirects) then it works

Notice the location: header (which adds x86_64) pasted in the above output matches the "invalid URL" in the thrown error - and the error is correct, it is indeed an invalid URL - it needs to be appended to https://mirror.openshift.com/ to get the right URL

It seems that curl handles the invalid URL correctly by treating it as a path relative to the original URL - but the HTTP client used in this action does not handle it correctly.

The mirror team should probably be setting this header correctly to the full URL and not just a relative path. Or maybe this should be considered a bug in the actions/http-client - I'm not sure what the specification is for relative redirects. Either way this can be worked around on the tools-installer side if you tweak how it handles redirects or just use a different HTTP client that handles this better.

https://github.com/redhat-actions/openshift-tools-installer/blob/main/src/mirror-client-finder/directory-finder.ts#L58 (also that Content-Type header should be an Accept header - oops) https://github.com/redhat-actions/openshift-tools-installer/blob/main/src/util/utils.ts#L12

Hope that helps

[tetchel]

If you browse the mirror it's also received a facelift so I suppose now they've moved to the new mirror we were warned about - and it ended up breaking this action after all! :exploding_head:

[urswiss]

source: github is not working for us:

[milindkejriwal]

oc and helm are not available on source: github :face_with_head_bandage:

[divyansh42]

Yes, you can find the list of tools that are available on GitHub in the following list https://github.com/redhat-actions/openshift-tools-installer#supported-tools.

[divyansh42]

@tetchel I noticed a strange thing if you put / at the end of the URL. Then everything works fine.

curl -i https://mirror.openshift.com/pub/openshift-v4/clients/serverless/0.15.2/ 
HTTP/2 200 
content-type: text/html
content-length: 13967
server: CloudFront
date: Thu, 16 Dec 2021 07:27:53 GMT
vary: Accept-Encoding
cache-control: max-age=0
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 5e51c2cb85f3832b4e4037f8dff6904d.cloudfront.net (CloudFront)
x-amz-cf-pop: TLV50-C2
x-amz-cf-id: 6pEogNMCl23rE28YVp5bT2hyGjPsL7K9jF-4LEv8a-7289ifeYG5Dw==

I think this solution will work for us.

But ideally, I don't think this should happen as the OpenShift Mirror team told us that transition won't be a breaking change.

[urswiss]

@divyansh42 it doesn't look like redhat-actions/openshift-tools-installer@v1 would take an url so we can also not pass the additional slash. Does somebody from Redhat know how long it takes until this issue is resolved? Our development team can not merge any PR's at the moment and deployments are no longer possible through github actions.

[divyansh42]

We are working on a fix for this. Hopefully, by the end of the day, we'll come up with the patch.

[divyansh42]

@urswiss We have fixed this issue, could you please re-run your workflows and confirm.

[urswiss]

@divyansh42 I can confirm that it is working again. Thanks for fixing this! However, the action redhat-actions/openshift-tools-installer@v1 runs about 3 minutes now, compared to < 20 seconds yesterday.

[divyansh42]

@urswiss I am wondering if caching is the case here. With the updated OpenShift mirror, probably older cached versioned is not matched with the new version, hence it has to download the binaries once again. Could you please re-run the workflow to check the time difference?

[milindkejriwal]

Thanks a lot for getting on top of this!

[urswiss]

@divyansh42 I ran it a couple of times now, and it's always around 3 minutes. But I can't rule out that it is something on our end (we have a Openshift inside the company). We will monitor the situation in the next few days.

Current situation (3 minutes)

Yesterday morning (17 seconds)

[divyansh42]

@urswiss You are running on GitHub enterprise, so cache can't be a reason for this time difference. We'll also try to investigate this on our end.

[jduimovich]

As a data point, I ran a bare bones install in this test repo https://github.com/jduimovich/bench-install which installed oc on github in cached and not-cached modes. Both run in small numbers of seconds.

This is the uncached run (5s) and the cached (not shown) is in 2s.

[divyansh42]

Resolved in https://github.com/redhat-actions/openshift-tools-installer/pull/59