Testing registries for release 0.2.0 #147

Open vsoch opened 5 months ago

The current main branch has the TBA 0.2.0 release. I was waiting on getting a bit more testing of the new auth backend across registries of interest. Here is the list:

[ ] ghcr.io
[ ] GitLab packages
[ ] Docker Hub (distribution)
[ ] Quay.io
[ ] AWS
[ ] Azure
[ ] Google (gcr.io)
[ ] Zot
[ ] Quay

I would like to contribute some testings, and some questions, please?

Distribution, Basic-on-http

I believe I was able to test Basic-on-http using "distribution" (ghcr.io/oras-project/registry:latest) as demonstrated by this example notebook, which uses this example configuration with local "distribution" container.

Zot, Basic-on-http

I believe I was able to test Basic-on-http using "Zot" (ghcr.io/project-zot/zot-linux-arm64:latest) as demonstrated by this example notebook, which uses this example configuration with local container.

Distribution/Zot, using Token

I'm not sure if/how possible to test Token-on-http with either Distribution or Zot.

Using Token

I'm having problem with Token-on-http/-https. I've tried:

localhost-quay using http: https://github.com/tarilabs/demo20240629-oras147-testregistries/blob/main/demo-localhost-quay-basic.ipynb
quay.io using https: https://github.com/tarilabs/demo20240629-oras147-testregistries/blob/main/demo-quayio-token.ipynb

The error looks similar and getting Unauthorized when allegedly the token get used to upload blob.

To check the server should answer as expected, I've used with the oras CLI (from Homebrew) and I'm able to successfully login, then push, to both localhost-quay using http (using --insecure on the command line) and quay.io using https (proof).

I don't recall this was a problem pre <0.2.0.

I'm running short of ideas how to resolve this issue, does anyone has any suggestion here, please?

`tls_verify=False` ignored for Token, and workaround

I believe when Token-on-https there are still 2 calls which would ignore the self-signed certs opt-in of tls_verify=False. Here: https://github.com/oras-project/oras-py/blob/caf8db5b279382335fbb1f6d7402ed9b73618d37/oras/auth/token.py#L151 and here: https://github.com/oras-project/oras-py/blob/caf8db5b279382335fbb1f6d7402ed9b73618d37/oras/auth/base.py#L141

I believe an effective workaround might be to do as follows:

import oras.client
client = oras.client.OrasClient(auth_backend="token", tls_verify=False)
client.session.verify = False

If I implement this workaround, I get back into the previous case (unable to actually use Token to push).

This is great - thank you! For each of the cases that don't work, it's a bit arduous but what I do is test in an interactive IPython terminal, so when it fails I have a catch in the code that does:

import IPython
IPython.embed()

And then it's possible to manually inspect and see what went wrong, and figure out how we might update an auth backend or (more reasonably) create a custom one to support the behavior.

Thanks for the feedback; for better context, my question is about Token backend, is there a OCI registry servr which was modeled that after and for me attempt replicate a successful test, please?

The reason I'm asking:

if I try with Oras-py 0.1.30 and quay.io, authentication works, per this example (note it's not the main branch)
if I try with Oras-py 0.1.30 and local quay (insecure), authentication works, per this example (note it's not the main branch)

But instead:

if I try with Oras-py 0.2.0 and quay.io, authentication do not work per previous comment
if I try with Oras-py 0.2.0 and local quay (insecure), authentication do not work per previous comment
I've also tried dockerhub, with Oras-py 0.2.0, authentication do not work, per this example

I'm trying to understand which OCI registry, using Token based auth of Oras-py 0.2.0 would showcase a successful run, because to me that at present feels a bit of a regression based on the above reporting? But it could be very likely I'm doing something wrong! In other words, I'm trying to understand better how the current auth backend works :) so then I can move on to understand better why it's failing for the cases provided?

That's the reason we need testing and someone to closely inspect (interactively inspect) what is going on - the new auth setup likely needs development work so those cases work. I expected them to need some work.

I see. Thanks for the feedback, I'll try to investigate further/report back/contribute.

Thank you! And if there is a specific registry I can access as well, I can definitely help to reproduce (and then see if I can do a fix).

Maybe we could add harbor to the registries. Maybe we could create automated tests against at least some registries. Local tests for registries which can be bootstrap via podman/docker and CICD tests on the cloud registries triggert by pull requests.

I did a little research: For Azure oras-py could apply for Credits for Opensource Project Link - if accepted and with enough credits there could be some non Azure Registries be started For AWS ECR -> there is a free tier Link GCP is supporting the cncf wich could provide the through the oras-project quay.io access via Redhat developer subscriptions Link ghcr.io -> available for all github users/teams/orgs the selfhosted registries could be run either in Cloud(AWS,GCP,Azure) or could be bootstrapped on Pull Requests (Maybe exceeds with actions usage quota )

quay.io access via Redhat developer subscriptions Link

For Quay (in general, not necessarily .io) we can use a boiled down version of K8s deployment without operators, like we're doing here.

In general, I would be happy to propose similarly to these:

example of CNCF distribution registry (with no user/password, but can be modified althought to my understand in this repo we have that scenario covered already)
example of Zot (with no user/password, but can be modified to require user/password)
example of using Quay "lite"

If we agree with the approach in general of deploy some of the registries which can be self hosted by leveraging KinD in a GitHub action, I'm happy to contribute similarly here to oras-py. wdyt?

@tarilabs and @mariusbertram I don't have bandwidth at the moment to work on these components, but I can fully support you and provide review for what you are able to contribute. Thank you!

oras-project / oras-py