search

k3d-io / k3d

Little helper to run CNCF's k3s in Docker
https://k3d.io/
MIT License
5.39k stars 458 forks source link

[HELP/QUESTION] Docker registry cache not working for private registries #238

Open giulioprovasi opened 4 years ago

giulioprovasi commented 4 years ago

Your Question/Help Request

I have tried to setup a docker container for caching registry image pulls but without any success, the local registry only cache docker.io images and nothing else

Information for Helpers

What did you do?

registries.yaml

---
mirrors:
  "docker.io":
    endpoint:
      - "https://registry-1.docker.io"
  "quay.io":
    endpoint:
      - "https://quay.io/repository"
  "registry.acme.com:443":
    endpoint:
      - "https://registry.acme.com:443"

configs:
  "registry.acme.com:443":
    auth:
      username: "user"
      password: "passwd"
    tls:
      ca_file: "/etc/ssl/certs/acme_Root_CA.pem"

then tried to run an image on the cluster and check the logs of the docker container caching the image pulls:

kubectl run -it --rm --image alpine:3.11 test-pull sh

and the docker logs -f k3d-registry output :

time="2020-05-10T19:43:10.182010618Z" level=info msg="Adding new scheduler entry for library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb9a54 with ttl=167h59m59.999994463s" go.version=go1.11.2 instance.id=84e49f71-e19b-41b4-b477-aeabdcd3c103 service=registry version=v2.7.1
...

same with kubectl run -it --rm --image registry.acme.com:443/xxx/cli:devel test-pull2 sh

and have zero logs on the registry container.

Moreover, here the ls -lisa output on the registry cache container

$ docker exec -ti k3d-registry ls -lisa /var/lib/registry/docker/registry/v2/repositories/                                                                                                                 total 28
2626451      4 drwxr-xr-x    7 root     root          4096 May  9 21:05 .
2626450      4 drwxr-xr-x    4 root     root          4096 May  9 20:47 ..
3018135      4 drwxr-xr-x    4 root     root          4096 May  9 20:58 bitnami
3018120      4 drwxr-xr-x    6 root     root          4096 May 10 19:43 library
3018121      4 drwxr-xr-x    3 root     root          4096 May  9 20:58 minio
2626452      4 drwxr-xr-x    6 root     root          4096 May  9 20:47 rancher
3018231      4 drwxr-xr-x    3 root     root          4096 May  9 21:05 rediscommander

Nothing is going on about acme registry (quay.io not working too)

Which OS & Architecture?

$ uname -a
Linux gpr 4.19.10-arch1-1-ARCH #1 SMP PREEMPT Tue Dec 18 00:04:08 UTC 2018 x86_64 GNU/Linux

Which version of k3d?

$ k3d --version
k3d version dev

Which version of docker?


$ docker version                                                                                                                                                                                           Client:
 Version:           19.03.8-ce
 API version:       1.40
 Go version:        go1.14.1
 Git commit:        afacb8b7f0
 Built:             Thu Apr  2 00:04:36 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.14.1
  Git commit:       afacb8b7f0
  Built:            Thu Apr  2 00:04:16 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.3.3.m
  GitCommit:        d76c121f76a5fc8a462dc64594aea72fe18e1178.m
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
iwilltry42 commented 4 years ago

Hi @giulioprovasi , thanks for opening this issue. I might be wrong, but I think this is how --enable-registry --enable-registry-cache was setup: only cache DockerHub images. The functionality was introduced by PR #207 by @inercia to have the option that...

the local k3d registry can act as a pull-through cache for the Docker hub

giulioprovasi commented 4 years ago

thanks for your answer @iwilltry42 , indeed after digging up some code I found out that only DockerHub is cached, but I come out with a solution that works, here it is (in case someone is looking for the same thing):

generate a cert with mkcert in order to get SSL on localhost mkcert -cert-file cert.pem -key-file key.pem *.local

docker-compose.yaml: mount ssl certificate into traefik (you can also just skip traefik and go with a simple registry2 as a frontend, I've used traefik as I use multiple cache registries and expose them with a single ingress service)

version: "3.8"

services:

  ingress:
    image: "traefik:v2.2"
    restart: "always"
    ports:
      - "8080:80"
      - "8443:443"
      - "9090:9090"
    networks:
      - "k3d-dev" # only this service need to be exposed
      - "backend"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./services/traefik/traefik:/etc/traefik:ro"
      - "./services/traefik/certs:/etc/certs:ro"

  registry-acme:
    image: "registry:2"
    restart: "always"
    volumes:
      - "registry:/var/lib/registry"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.docker-registry.rule=Host(`registry-acme.local`)"
      - "traefik.http.routers.docker-registry-cache-acme.entrypoints=https"
      - "traefik.http.routers.docker-registry-cache-acme.tls=true"
    networks:
      - "backend"
    environment:
      REGISTRY_PROXY_REMOTEURL: "https://registry.acme.com:443"
      REGISTRY_PROXY_USERNAME: "giulio.provasi@acme.com"
      REGISTRY_PROXY_PASSWORD: "securepassword"

note that traefik exposes a custom port 8443, as I am using the 443 for k3d cluster

optional: edit /etc/docker/daemon.json in order to share cache between your local docker daemon and k3d daemon: 

"registry-mirrors": [
    "https://registry-acme.local:8443"
  ]

create the cluster: note that the cluster name matters, as it will influence the network that traefik has to join (see docker-compose.yaml)

k3d create \ 
    --name dev \
    --api-port "6551" \
    --registries-file $HOME/.k3d/registries.yaml \
    --volume /$HOME/.local/share/mkcert/rootCA.pem:/etc/ssl/certs/Registry_Root_CA.pem

Now I am using mkcert, so we need to provide the mkcert Root CA to k3d, then again create the registries.yaml file with the following:

---
mirrors:
  "registry.acme.com:443":
    endpoint:
      - "https://registry-acme.local:8443"

configs:
  "registry-acme.local:8443":
    tls:
      ca_file: "/etc/ssl/certs/Registry_Root_CA.pem"

Wrap-up:

giulioprovasi commented 4 years ago

It would be cool to have that functionality without going through all that trouble, in the other way, I like this way as we can share cache between local daemon and k3d, my use case was that work docker images are heavy (some > 3gb) and while at office is it not an issue, but due to a lot of remote lately, it became a pain.

iwilltry42 commented 4 years ago

This is indeed pretty cool @giulioprovasi :) I will create a Feature Request out of this :+1:

inercia commented 4 years ago

Hi guys,

I was wondering if this could be solved by just setting the right environment variables. It seems to me that that the main change is that this solution establishes values for REGISTRY_PROXY_REMOTEURL, REGISTRY_PROXY_USERNAME and REGISTRY_PROXY_PASSWORD. Could that be solved just with that?

I would also skip the registry through the Ingress as it adds some unnecessary complexity, as well as the certificates (not really necessary if we assume the main use case for k3d is for local clusters).

giulioprovasi commented 4 years ago

@inercia I agree with you for both traefik and ssl, it is my usecase but may be simplified by skipping those steps indeed.

For the environment variables I am not sure about that, as if I take a look at cli/registry.go (excuse me if I misunderstand, I am not a go guy) but it seems to me that the cache only works with default docker.io, although if I misunderstand I'll be glad to simplify my solution.

What I'd need is to being able to setup multiple registry cache pullthrough:

so it seems to me that currently that is not possible

inercia commented 4 years ago

@giulioprovasi Then maybe the solution would be to start N (one for each upstream registry) local registries containers, all of them using image registry:2, but each one with

Not sure this would work (I guess we would need to do a PoC), but maybe this could be possible...

giulioprovasi commented 4 years ago

it is exactly what I did here (stripped out traefik data, assume registry is https on port 8443):

docker-compose.yaml

version: "3.8"

services:

  registry-acme:
    image: "registry:2"
    restart: "always"
    volumes:
      - "registry:/var/lib/registry"
    networks:
      - "backend"
    environment:
      REGISTRY_PROXY_REMOTEURL: "https://registry.acme.com:443"
      REGISTRY_PROXY_USERNAME: "giulio.provasi@acme.com"
      REGISTRY_PROXY_PASSWORD: "mysecurepassword"

  registry-quayio:
    image: "registry:2"
    restart: "always"
    volumes:
      - "registry:/var/lib/registry"
    networks:
      - "backend"
    environment:
      REGISTRY_PROXY_REMOTEURL: "https://quay.io/repository"

  registry-dockerio:
    image: "registry:2"
    restart: "always"
    volumes:
      - "registry:/var/lib/registry"
    networks:
      - "backend"
    environment:
      REGISTRY_PROXY_REMOTEURL: "http://registry-1.docker.io"

volumes:
  registry:
    labels:
      - "gpr.keep=true"

daemon.json

{
  "storage-driver": "overlay2",
  "features": {
    "buildkit": true 
  },
  "debug": true,
  "experimental": true,
  "registry-mirrors": [
    "https://registry-dockerio.local:8443",
    "https://registry-quayio.local:8443",
    "https://registry-infomaniak.local:8443"
  ]
}

registries.yaml

---
mirrors:
  "docker.io":
    endpoint:
      - "https://registry-dockerio.local:8443"
  "registry.infomaniak.com:443":
    endpoint:
      - "https://registry-infomaniak.local:8443"
  "quay.io":
    endpoint:
      - "https://registry-quayio.local:8443"

configs:
  "registry-dockerio.local:8443":
    tls:
      ca_file: "/etc/ssl/certs/Registry_Root_CA.pem"
  "registry-quayio.local:8443":
    tls:
      ca_file: "/etc/ssl/certs/Registry_Root_CA.pem"
  "registry-infomaniak.local:8443":
    tls:
      ca_file: "/etc/ssl/certs/Registry_Root_CA.pem"
mcanevet commented 4 years ago

@giulioprovasi I tried to mimic your setup (in pure http without reverse proxy in front of it), but I couldn't get it work for quay.io. I have the error "manifest schema v1 unsupported" while it works fine for docker.io. Does it really work for you?

mcanevet commented 4 years ago

OK I found it. Since registry v2.7 you have to add REGISTRY_COMPATIBILITY_SCHEMA1_ENABLED=true for registry that are still using the old schema version (quay.io at least)

frimik commented 3 years ago

it is exactly what I did here (stripped out traefik data, assume registry is https on port 8443) ...

@giulioprovasi inspired by you and my VPN being too slow to be bareable while iterating on cluster teardown and creation I gathered all sorts of practices from around the docker hub docs and other places and set this up for a local caching registry proxy setup, hope you don't mind: https://github.com/frimik/local-docker-registry-proxy

For months and months now I had been running with a plain docker run registry:2 local proxy and finally I did the real thing.

kmhughes commented 3 years ago

In the docker-compose above, there is a volume label gpr.keep=true. What is this for?

giulioprovasi commented 3 years ago

@kmhughes nothing important, gpr stands for @giulioprovasi, I have a rule in the docker client config : 

 ➜  ~ cat ~/.docker/config.json 
{
    "pruneFilters": [
        "label!=gpr.keep=true"
    ]
}

to avoid dropping unwanted stuffs

kmhughes commented 3 years ago

Ahh, OK, thanks!