Configure custom registry

fragolinux commented 3 years ago

hi what's the correct way to add a custom registry.yaml file to RD k3s? in k3d i do something like this: --volume "$cfgDir/registries.yaml:/etc/rancher/k3s/registries.yaml" thanks in advance

jandubois commented 3 years ago

There is no "correct" way (as in "supported" way) to do this right now, but you could try to add the mount manually. Edit ~/Library/Application\ Support/rancher-desktop/lima/0/lima.yaml after you have been running Rancher Desktop at least once, but while the app is stopped.

Locate the mounts section and add an additional entry at the end:

mounts:
  - location: ~/Library/Caches/rancher-desktop/k3s
    writable: false
  - location: "~"
    writable: false
  - location: /tmp/rancher-desktop
    writable: true
  - location: "/etc/rancher/k3s"
    writable: false

It is important that the entry is added at the end; otherwise it will be overwritten by Rancher Desktop on the next start. And you cannot map directories to a different location, so you will have to create /etc/rancher/k3s on your host as root, and put the registries.yaml file in there (it is not possible to just mount the file; you have to mount the directory).

After you saved the lima.yaml file, restart Rancher Desktop. It should now use your configured registries. Please report back if this actually works, as I haven't tested it. 😸

Eventually we will add proper support for this functionality in the UI, but for now this ~~hack~~workaround should give you the functionality you need.

jandubois commented 3 years ago

I assumed this was about macOS; if you are running on Windows, this would need a different approach.

fragolinux commented 3 years ago

I assumed this was about macOS; if you are running on Windows, this would need a different approach.

yes, sorry for the missing information: talking about macos, indeed... thanks :)

fragolinux commented 3 years ago

after closed RD, added those 2 lines at the end of the mounts section of lima.yaml, and run:

sudo mkdir -p /etc/rancher/k3s
sudo nano /etc/rancher/k3s/registries.yaml

put my registries in there, saved, i have now:

sudo ls -la /etc/rancher/k3s
total 8
drwxr-xr-x  3 root  wheel    96  4 Ott 18:05 .
drwxr-xr-x  3 root  wheel    96  4 Ott 18:04 ..
-rw-r--r--  1 root  wheel  2111  4 Ott 18:05 registries.yaml

now RD does not start anymore... about 5 minutes stuck in STARTING KUBERNETES...

jandubois commented 3 years ago

I'm sorry, I forgot that k3s needs to be able to write to that directory to create k3s.yaml, so this approach won't work.

I tried to copy the file in via a provisioning script, but that doesn't work either because at that point the directories have not yet been mounted.

What did work was embedding the registries.yaml inside the lima.yaml file as a provisioning script:

provision:
...
  - mode: system
    script: |
      #!/bin/sh
      set -eux
      mkdir -p /etc/rancher/k3s
      cat <<EOF >/etc/rancher/k3s/registries.yaml
      mirrors:
        "my.company.registry:5000":
          endpoint:
          - http://my.company.registry:5000
      EOF

As before, this has to be added at the end. And if Rancher Desktop during an upgrade adds another provisioning script, your script will have to be added back after the upgrade.

fragolinux commented 3 years ago

yes, this way it worked, i could download and run an image from a private ecr registry on aws, thanks!

jandubois commented 3 years ago

Reference (for implementing this in the UI): private registry configuration.

A minimal implementation could just maintain a registry.yaml internally and make it editable in a text field.

Longer term a full UI (maybe driven by a schema definition) would be nice, but it should not delay exposing the functionality through the UI.

himslm01 commented 3 years ago

What did work was embedding the registries.yaml inside the lima.yaml file as a provisioning script:

On Linux should I edit $HOME/.local/share/rancher-desktop/lima/0/lima.yaml to insert the suggested object into the provision array? That file seems to be over-written once the VM Rancher Desktop starts.

  - mode: system
    script: |
      #!/bin/sh
      set -eux
      mkdir -p /etc/rancher/k3s
      cat <<EOF >/etc/rancher/k3s/registries.yaml
      mirrors:
        docker.io:
          endpoint:
            - "http://nexus.lan:8082"
        nexus.lan:
          endpoint:
            - "http://nexus.lan:8082"
      EOF

jandubois commented 3 years ago

That file seems to be over-written once the VM Rancher Desktop starts.

Yes, it is being overwritten, but it should be merged with the file on disk. So if you add your changes at the end of lists, they should be retained.

Of course during a Rancher Desktop update they might still get lost, as we might add another provisioning script or volume mount, or whatever.

himslm01 commented 3 years ago

Ah - putting it at the end of the array of the provision array is the bit I missed.

So - I have the registries.yaml file in place. Should I expect the "pull image" feature in Rancher Desktop or nerdctl in the VM to honour it?

$ cat /etc/rancher/k3s/registries.yaml 
mirrors:
  docker.io:
    endpoint:
      - "http://nexus.lan:8082"
  nexus.lan:
    endpoint:
      - "http://nexus.lan:8082"

$ nerdctl --namespace "k8s.io" image pull "nexus.lan/jvmoptions:latest"
INFO[0000] trying next host                              error="failed to do request: Head \"https://nexus.lan/v2/jvmoptions/manifests/latest\": dial tcp 10.64.0.115:443: connect: connection refused" host=nexus.lan
FATA[0000] failed to resolve reference "nexus.lan/jvmoptions:latest": failed to do request: Head "https://nexus.lan/v2/jvmoptions/manifests/latest": dial tcp 10.64.0.115:443: connect: connection refused

dirk-olmes commented 3 years ago

I assumed this was about macOS; if you are running on Windows, this would need a different approach.

What would be the approach for configuring a custom registry in Windows? I cannot find a lima.yaml anywhere in my account.

jandubois commented 3 years ago

What would be the approach for configuring a custom registry in Windows? I cannot find a lima.yaml anywhere in my account.

RD on Windows isn't using Lima, but WSL2, so there is no way to run a provisioning script. But you can copy the file into the distro manually:

wsl -d rancher-desktop mkdir -p /etc/rancher/k3s
wsl -d rancher-desktop cp registries.yaml /etc/rancher/k3s

The first command shouldn't be necessary, but I found that I didn't have that directory in my distro. It is probably because I was testing something with a fresh install, and it failed before running k3s successfully.

Note that for the second command, the path to the local registries.yaml must use Linux path names, so should be ./data/registries.yaml, or /mnt/c/Users/Jan/data/registries.yaml and not the Windows path name.

Please let me know if this worked for you!

dirk-olmes commented 3 years ago

RD on Windows isn't using Lima, but WSL2, so there is no way to run a provisioning script. But you can copy the file into the distro manually:
wsl -d rancher-desktop mkdir -p /etc/rancher/k3s
wsl -d rancher-desktop cp registries.yaml /etc/rancher/k3s
... Please let me know if this worked for you!

@jandubois Thank you for the workaround - it works. Luckily I played around with k3s/k3d earlier and figured out the registries.yaml part already. Here's an example of what worked for me:

mirrors:
  "docker.io":
    endpoint:
      - https://my-private-registry:5000

Since the my-private-registry uses a self signed certificate I had to copy that into the rancher-desktop WSL image to /etc/ssl/certs.

pulberg commented 2 years ago

I have tried several different things with the yaml but i can't get my private reg recognized -

macOS Big Sur 11.6.1, RD 0.6.0

- mode: system
    script: |
      #!/bin/sh
      set -eux
      mkdir -p /etc/rancher/k3s
      cat <<EOF >/etc/rancher/k3s/registries.yaml
      mirrors:
        "myregurl:80":
          endpoint:
          - http://myregurl:80
      EOF

I can build/tag the image in RD but the push keeps trying to use https to the reg -

Error trying to push myregurl/eskibana/eskibana:es7.15.1-node14.17.6-32gb:

time="2021-11-01T21:40:32Z" level=info msg="pushing as a single-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:c5b041f767e5d5c4377f44efc2dc9c023ffc60b06dfda6833177572aede6b8e4)"
time="2021-11-01T21:41:02Z" level=fatal msg="failed to do request: Head \"https://myregurl/v2/eskibana/eskibana/blobs/sha256:4ea40d27a2cfcec3d38b2a0ebe5ca77633d27a394541c449b500fce4639516d4\": dial tcp 10.32.58.90:443: i/o timeout"

B-Paluch commented 2 years ago

@jandubois I might be doing it wrongly, or the workaround for windows doesn't seem to work anymore in RD 0.7.0 version. Has anything changed? Is there any special formatting i have to give the file?

himslm01 commented 2 years ago

For v0.7.0 will we need to create both /etc/rancher/k3s/registries.yaml for k3s and /etc/docker/daemon.json for dockerd?

himslm01 commented 2 years ago

Copying here from Slack...

With Rancher Desktop v0.7.0, for using local insecure registries:

When the Container Runtime is containerd you need to make /etc/rancher/k3s/registries.yaml with mirrors, as shown below. Here the mirrors act like aliases, so I can have a deployments spec.template.spec.containers[].image refer to nexus.lan/image:tag and containerd will pull the image from http://nexus.lan:8082
When the Container Runtime is dockerd (moby) you need to make /etc/docker/daemon.json with insecure-registries and registry-mirrors as shown below. But there are two problems as far as I can see.
- there is no possibility of having an alias - so in my example deployment the spec.template.spec.containers[].image MUST refer to nexus.lan:8082/image:tag - making the deployment yaml potentially incompatible between containerd and dockerd (moby).
- kubernetes MUST have imagePullSecrets pointing to a secret of type kubernetes.io/dockerconfigjson defined in the deployment's spec.imagePullSecrets or in the imagePullSecrets of the serviceaccount called default in the namespace of your deployment.

You can define both /etc/rancher/k3s/registries.yaml and /etc/docker/daemon.json in one go by creating override.yaml in the following location:

Linux: $HOME/.local/share/rancher-desktop/lima/_config/override.yaml
Mac: $HOME/Library/Application Support/rancher-desktop/lima/_config/override.yaml
Windows: ???

Here's my override.yaml

provision:
  - mode: system
    script: |
      #!/bin/sh
      set -eux
      mkdir -p /etc/rancher/k3s
      cat <<EOF >/etc/rancher/k3s/registries.yaml
      mirrors:
        docker.io:
          endpoint:
            - "http://nexus.lan:8082"
        nexus.lan:
          endpoint:
            - "http://nexus.lan:8082"
      EOF
      mkdir -p /etc/docker
      cat <<EOF >/etc/docker/daemon.json
      {
        "insecure-registries" : ["nexus.lan:8082"],
        "registry-mirrors": ["http://nexus.lan:8082"],
        "experimental": true
      }
      EOF

TomLevvv commented 1 year ago

HI,

I am trying to use Rancher desktop on windows 10, rancher desktop version is 1.7.0, k8s version is:v1.21.4 But I can't pull images from the local registry - I get unauthorized error:

eu.gcr.io/tos-ci/verifier:23-1-prc1.0.0: resolving      |--------------------------------------| 
elapsed: 1.4 s                           total:   0.0 B (0.0 B/s)                                         
time="2023-01-18T13:21:55Z" level=fatal msg="failed to resolve reference \"---23-1-prc1.0.0\": unexpected status from HEAD request to https://---/---/---/manifests/23-1-prc1.0.0: 401 Unauthorized"

I tried to fix it with the following ways:

Run the k3s with containerd, and copy the registries.yaml into the rancher:

wsl -d rancher-desktop mkdir -p /etc/rancher/k3s
wsl -d rancher-desktop cp registries.yaml /etc/rancher/k3s

I didn't know how to restart the k3s from the rancher desktop- when I reset the Kubernetes from the troubleshooting page, the registries file is deleted, so I changed the k3s to dockerd, and then to containerd- instead of restart, Is it enough? because after that - I still get the unauthorized error. The registries.yaml content is used in a real Linux machine- and works there.

Run the rancher desktop with dockerd and login to docker manually, but get the following error:

#  docker login -u _json_key --password-stdin https://eu.gcr.io/aaa-ci < read_only.json
/usr/local/bin/docker-credential-rancher-desktop: line 12: CREDFWD_AUTH: parameter not set
Error saving credentials: error storing credentials - err: exit status 2, out: ``

Run the rancher desktop with containerd and login to nerdctl manually, but get the following error: (sane error from wsl and from powershell )

nerdctl login -u _json_key --password-stdin https://eu.gcr.io/aaa-ci < read_only.json
/usr/local/bin/docker-credential-rancher-desktop: line 12: CREDFWD_AUTH: parameter not set
FATA[0000] error saving credentials: error storing credentials - err: exit status 2, out: ``

rancher-sandbox / rancher-desktop

Configure custom registry #721