Cannot pull images from a private registry

[richardcase]

What happened: I am trying to pull an image from a private registry for use as the root volume. I am using this command:

fl microvm create --host 127.0.0.1:9090 --name fctest --metadata-hostname fctest --network-interface eth1:tap  --metadata-ssh-key-file ~/.ssh/id_ed25519.pub --memory 4096 --root-image ghcr.io/richardcase/private-images/flintlock-ubuntu-perftest:7a248f6 --vcpu 2

The ghcr.io/richardcase/private-images/flintlock-ubuntu-perftest:7a248f6 image is in a private package in GitHub. It requires a PAT with repo:read to get this.

Currently, there is no way to supply credentials and it was assumed we could supply this using the CRI section of the containerd config file like this:

  [plugins."io.containerd.grpc.v1.cri".registry]
    [plugins."io.containerd.grpc.v1.cri".registry.configs]
      [plugins."io.containerd.grpc.v1.cri".registry.configs."ghcr.io".auth]
        username = "YYYYYYYYYYY"
        password = "XXXXXXXXX"

However, this does not work as we aren't using the CRI.....doh!

What did you expect to happen: I would expect there to be a way to supply credentials so that i can use a image from a private registry

How to reproduce it: Try and use a privat eimage using the fl command shown above

Anything else you would like to add: This came via the community slack from Cory & Paul.

Paul suggested that we need to include a resolver here: https://github.com/weaveworks-liquidmetal/flintlock/blob/main/infrastructure/containerd/image_service.go#L146

Environment:

• flintlock version: v0.1.1
• containerd version: 1.5.9

[richardcase]

We can use the "docker resolver" from containerd. Something similar to this:

    opts := []containerd.RemoteOpt{}
    if im.config.HostsDir != "" {
        resolver, err := im.getResolver(ctx, im.config.HostsDir)
        if err != nil {
            return nil, fmt.Errorf("getting containerd resolver: %w", err)
        }
        opts = append(opts, containerd.WithResolver(resolver))
    }

    image, err := im.client.Pull(leaseCtx, imageName, opts...)
    if err != nil {
        return nil, fmt.Errorf("pulling image using containerd: %w", err)
    }

func (im *imageService) getResolver(ctx context.Context, hostsDir string) (remotes.Resolver, error) {

    hostOptions := config.HostOptions{
        HostDir: config.HostDirFromRoot(hostsDir),
        Credentials: func(host string) (string, string, error) {
            return "yyyyyy", "xxxxxx", nil
        },
    }

    options := docker.ResolverOptions{
        Hosts: config.ConfigureHosts(ctx, hostOptions),
    }

    return docker.NewResolver(options), nil
}

But we need a configurable way to lookup the username and password (and maybe a auth header) on the hosts machines.

Some initial suggestions on how we might specify/store the credentials on a host machine:

1. Stored in a file on disk, in the file a registry host is mapped to username/pass
2. We lookup in containerd's content store a content item with specific labels(s) that denotes the content item is credentials for a specific registry host. (we have the content service that we can use to do this)
3. We submit a patch to containerd so that username/password can be specified in the hosts.toml file. This will need enhancement to loadHosrtDir

[paul-england]

Not something I've dug too much into so am reluctant to post, but I'm curious if just sending a protobuf for a requested image a la that found in crictl (sans authentication) would get the job done. That CRI config section is in containerd for a reason, one would hope.

[richardcase]

@paul-england - to give some extra context on option 2. Containerd has a number of different services that you can take advantage of. One of them is the "content store" and this is where we currently save the vm spec (its also where the image manifests / layers are stored): https://github.com/weaveworks-liquidmetal/flintlock/blob/main/infrastructure/containerd/repo.go#L86:L109. We could store credentials as a json blob in the content store and maybe retrieve them via a label that matches the domain of the registry?

Perhaps we could implement this feature where the backend store for the credentials is pluggable? Initially the content store with the option to load from filesystem in the future.

[richardcase]

If you want to look at the content store you can use ctr. For example:

ctr -n flintlock content ls

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity.

[github-actions[bot]]

This issue was closed because it has been stalled for 365 days with no activity.