[Design discussion] docker credentials and hosts.toml

apostasie commented 1 month ago

What is the problem you're trying to solve

Currently, we retrieve credentials from the traditional docker location. Docker has no notion of "hosts/mirrors", and credentials are stored per host/port.

On the other hand, we support containerd hosts.toml (to some extent).

Currently, when we are faced with things like that:

server = "http://foo.com"

[host."http://localhost:5000"]
  capabilities = ["pull", "resolve"]

We will:

first retrieve credentials associated with foo.com, if any
then get the hosts list from hosts.toml
try the host, REFUSE to send the credentials as the host does not match (this is good!) and fail

So, it is impossible to actually use hosts.toml with authentication.

The fundamental problem here is that we are trying to reconcile conceptually different worlds (the docker creds world and the hosts toml world).

My suggestion here would be the following:

nerdctl login foo.com
- retrieve docker credentials for foo.com
- do NOT try to contact any of the configured hosts - just contact foo.com
- if there are any host configured for foo.com
  - IF contacting foo.com FAILED (airgap scenario for example), provide some meaningful explanation and suggest them that they should try login against these hosts (localhost:5000 in this example)
  - IF login in foo.com did SUCCEED, still output a gentle message to the user that foo.com is currently configured to resolve to localhost:5000, and that we are NOT authenticated against it (and if they want to do so, they should nerctl login localhost:5000)
nerdctl pull/push foo.com/bla
- use the current flow (resolve hosts.toml, try the hosts in order)
- for each host, do retrieve the credentials for that host

I appreciate this is a tricky change, with wide ranging impact, but it seems to me that currently, we simply do not have a correct solution for mirrors requiring authentication.

Maybe I am missing something here... please point it out if so :-)

Thanks a lot folks!

Taggin @fahedouch @AkihiroSuda and any of the other good folks here.

Describe the solution you'd like

na

Additional context

No response

apostasie commented 1 month ago

I now believe it is not possible to reconcile docker credentials and hosts.toml spec. Specifically, a given server or host section may be used multiple times for different registry namespaces. These hosts will get used with the appended ?ns=NAMESPACE query parameter, which allows the server to understand for which namespace they are being queried.

Docker credentials only saves credentials per host+port, so, we cannot have multiple credentials sets per host (eg: along namespaces).

What I am thinking we can do:

Plan A.

Have nerdctl still save credentials inside the docker config file WITH the ns query parameter:

docker will be OK reading them - although on successful login docker will then re-save them WITHOUT the ns param - although said credentials may not work when used by docker, as the registry would not receive the param

Plan Abis.

Like plan A, except we would encode / obfuscate namespaced credentials (only) so that docker would not be able to use them.

Plan B.

Have a secondary credentials storage, where we can save namespaced credentials, that we would ALSO lookup on top of the traditional docker location.

Plan C.

Divorce entirely from docker credentials management, and use only our own system.

Thoughts?

apostasie commented 1 month ago

Another important question is UX.

Given myregistry.com hosts.toml file:

server = "https://server.com"

[host."https://host.com"]

What is the UX to login?

A:

nerdctl login --registry-namespace=myregistry.com server.com
nerdctl login --registry-namespace=myregistry.com host.com

B:

nerdctl login server.com/?ns=myregistry.com
nerdctl login host.com/?ns=myregistry.com

C:

nerdctl login endpoint://myregistry.com/server.com
nerdctl login endpoint://myregistry.com/host.com

D: other?

apostasie commented 1 week ago

Closing this in favor of the PR with the design doc.

containerd / nerdctl