Test login, push (and authenticated pull) against docker hub

apostasie commented 1 month ago

What is the problem you're trying to solve

Right now, we do test most of our registry operations against a locally started distribution registry + cesanta token auth server.

This is good, but has limits. Specifically, Docker Hub resolution mechanisms are "special", be it the way the credentials are stored in the store, or how short names resolve to fully qualified url - and also their token auth server is proprietary IIRC, which might entail some specific behaviors.

We do not currently test that, and have seen a couple of regression recently (#3484, #3485) that could have been caught if we did.

Testing against Hub comes with challenges:

we would need a dummy account, and a pair of access tokens
it is doubtful we could keep these secrets (as obviously PRs would have access to it) if that was enabled on PR testing
hub is notorious for 429-ing when things heat up a little bit - failing the build because of that is a royal PITA

A possible solution would be to run these tests against Hub solely on master (after merge).

This ticket is related to #3257 - and also testing against Gitlab.

Describe the solution you'd like

na

Additional context

No response

AkihiroSuda commented 1 month ago

Probably those tests should be only executed on a laptop outside the CI

apostasie commented 1 month ago

Probably those tests should be only executed on a laptop outside the CI

That would be a good start - and we can re-evaluate later.

fahedouch commented 1 month ago

Good point but as open source product our first target is to fill the oci distribution requirement. Hard to satisfy all oci distribution implemention (Hub, harbor..). Otherwise fork nerdctl and statisfy specific distribution implem is a nice solution ( this is what private organization does)

apostasie commented 1 month ago

Good point but as open source product our first target is to fill the oci distribution requirement. Hard to satisfy all oci distribution implemention (Hub, harbor..). Otherwise fork nerdctl and statisfy specific distribution implem is a nice solution ( this is what private organization does)

Fair. I like Akihiro suggestion: we could have some tests in there that can only be ran if a certain token env var is set and we do not run it on the CI

apostasie commented 1 month ago

Good point but as open source product our first target is to fill the oci distribution requirement. Hard to satisfy all oci distribution implemention (Hub, harbor..). Otherwise fork nerdctl and statisfy specific distribution implem is a nice solution ( this is what private organization does)

As I see it, (one of) the problem with open-source is that people do reap the benefits and do not contribute back. I have seen so many projects wither and die, because they told people they should just "fork and satisfy [their use case]". Then people do that - and before you know it, these forks are so far removed from upstream that it makes it impossible to contribute back - further reducing an incentive that was already low to start with.

My point being: we should make nerdctl such a compelling proposal that people should feel enticed to contribute here, and we should encourage that by giving them the means to do so.

I hope I am making sense :-).

Anyhow, philosophizing here...

containerd / nerdctl