How can I start a container from a private registry

[sophokles73]

I am using the version 0.0.6 leda x86_64 QEMU image, trying to start an additional container from a private container registry. I have created a /data/var/containers/manifests/my-container.json file:

{
    "container_id": "my-container",
    "container_name": "my-container",
    "image": {
        "name": "ghcr.io/my-repo/my-container:latest"
    },
    "host_config": {
        "devices": [],
        "network_mode": "bridge",
        "privileged": false,
        "restart_policy": {
            "maximum_retry_count": 0,
            "retry_timeout": 0,
            "type": "unless-stopped"
        },
        "runtime": "io.containerd.runc.v2",
        "extra_hosts": [
            "databroker:container_databroker-host"
        ],
        "port_mappings": [
        ],
        "log_config": {
            "driver_config": {
                "type": "json-file",
                "max_files": 2,
                "max_size": "1M",
                "root_dir": ""
            },
            "mode_config": {
                "mode": "blocking",
                "max_buffer_size": ""
            }
        },
        "resources": null
    },
    "io_config": {
        "open_stdin": false,
        "tty": false
    },
    "config": {
        "env": [
            "RUST_LOG=info",
            "KUKSA_DATA_BROKER_URI=http://databroker:55555"
        ],
        "cmd": []
    }
}

However, the repository on GitHub is private so, not unexpectedly, starting the container fails with an error being logge to /var/log/container-management/container-management.log:

time="2023-04-12T13:15:35.697926488Z" level=warning msg="[container-management][ctrd_client_internal.go:44][pkg:github][func:com/eclipse-kanto/container-management/containerm/ctr] the default resolver by containerd will be used for image ghcr.io/my-repo/my-container:latest" time="2023-04-12T13:15:37.511585085Z" level=error msg="[container-management][ctrd_client.go:94][pkg:github][func:com/eclipse-kanto/container-management/containerm/ctr] error while trying to get container image with ID = ghcr.io/my-repo/my-container:latest for container ID = my-container \n\t Error: failed to resolve reference \"ghcr.io/my-repo/my-container:latest\": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized "

Is there any way to configure leda to use certain credentials for authenticating to a container registry?

[mikehaller]

Yes, you can configure the Eclipse Kanto Container Manager with credentials for container registries. Refer to the Kanto Reference for details.

In Leda (starting 0.0.6), there is a convenient utility to modify the Kanto configuration:

sdv-kanto-ctl add-registry -h ghcr.io -u my_username -p my_password

It will automatically restart the container-management service so that the changes take effect. You may need to remove / recreate the container

[sophokles73]

That's good to know. It is, however, unclear to me under which circumstances I need to remove/recreate a container manually. I am currently using the init-dir approach to specify a manifest file for my container. When I make a change to that file, when will the change be picked up?

[mikehaller]

I think that would be a question to the Kanto project.

But due to this code, the init_dir approach will only run once on "first start of Kanto", but never again e.g. reboots or restarts of the container-management service will not trigger any kind of redeployments by Kanto itself.

So, to answer your question:

• When you use init_dir: You need to remove the container and recreate your container manually. Alternatively, stop Kanto, delete all its internal files and restart the service.
• When you use kanto-auto-deployer: Modifying the container json manifest (or even touching it) and saving it is enough for kanto-auto-deployer.service to pick up the change automatically, remove the existing container and recreate a new container.

As the init_dir approach is not as "convenient" as we had wished for, we decided to go for the kanto-auto-deployer approach in Leda for now. Starting with 0.0.6, the kanto-auto-deployer service will also watch for file change events in the manifests folder, so when you change a .json file in there, kanto-auto-deployer will pick it up and tries to redeploy the changed container.

Btw, we've added the "private container registry" documentation: https://eclipse-leda.github.io/leda/docs/device-provisioning/container-management/container-registries/

[sophokles73]

Now I am even more confused :-) I was trying to follow the approach described in the Leda documentation. My understanding is that the init_dir is the /data/var/containers/manifests folder, right? The last paragraph states

Important: The manifests in the kanto-cm init-dir are checked only ONCE - on service start-up. If you add/change a manifest you would need to restart the whole “container-management” service for your changes to take effect.

Now you are saying that a change will not be picked up unless the corresponding container gets removed (manually using kanto-cm remove -n container-id?). If so, why would I need to recreate the container manually? Isn't that what should be done automatically based on the manifest file(s)?

When you use kanto-auto-deployer: Modifying the container json manifest (or even touching it) and saving it is enough for kanto-auto-deployer.service to pick up the change automatically, remove the existing container and recreate a new container.

The Leda documentation seems to suggest that I would use the auto-deployer simply be means of putting (auto-deployer specific) template files to folder /data/var/containers/manifests_dev. However, that folder does not seem to exist in Leda 0.0.6 images.

Starting with 0.0.6, the kanto-auto-deployer service will also watch for file change events in the manifests folder, so when you change a .json file in there, kanto-auto-deployer will pick it up and tries to redeploy the changed container.

Does this mean that I do not need to use the auto-deployer specific template file format? Given that the init_dir approach also seems to expect manifest files to reside in that folder, which mechanism for deploying containers is now in effect, init_dir, auto-deployer, both?

[mikehaller]

Indeed the Leda documentation requires update, as the kanto-auto-deployer has been updated recently:

• it now also supports the kanto init-dir manifest format
• that allowed us to merge both folders into a single one. Hence the manifests_dev path is no longer necessary.

I think the sentence about "the containers are only deployed once - on service start" is not quite correct.

It's true for first boot, but not true for subsequent reboots. I think this has never been tested thorougly and we made the assumption it would be doing it on service (re)start.

[mikehaller]

The current mechanism is kanto-auto-deployer.

It takes the same manifests than kanto, but kanto init_dir is disabled in the container-management config.

We will keep using kanto-auto-deployer for the Leda quickstart images, as for development reasons, it's more convenient to have automatic redeployments as soon as the container manifest files change on disk.

[mikehaller]

And btw, kanto has an open pull request to also support updating containers after the "initial deployment", so the implementation there is ongoing...

[sophokles73]

ok, got it, thanks for explaining and IMHO you should really adapt the docs accordingly ;-)