Support for private registry mirrors for multiple repositories

jgallucci32 commented 3 years ago

Is your feature request related to a problem? Please describe. As of v2.1 Harbor can now act as a proxy_cache and a private registry. It also has the ability to act as a repository mirror but it is currently not possible to combine all those features to support private registry mirrors for multiple repositories. The reason for this all the container clients need an endpoint and are not path-aware of the top-level directory of the repository. Products like JFrog solve this problem by creating separate DNS endpoints (or IP addresses) which map to different registry projects. This is discussed in more detail in the comments of https://github.com/goharbor/harbor/issues/8082

Describe the solution you'd like Harbor should be configurable to allow for 1 or more repository mirrors to either private or proxy_cache projects to allow multiple mirrors to be configured. It should be done in a way which is easy to setup through Helm and not require extensive tweaking or changes to the infrastructure to support repositories on-demand.

Describe the main design/architecture of your solution Create a DNS alias to Harbor to be used as a repository mirror for container clients. By creating a DNS alias to the harbor endpoint, you can use rules in the nginx proxy to redirect traffic to various projects for a repository mirror. By configuring a wildcard DNS, the subdomain can be used to dynamically point to different mirrors with minimal configuration on the server side.

Describe the development plan you've considered Add the following variables to the Helm chart for configuration

registryMirror.endpoint - The DNS endpoint used for the default repository mirror
registryMirror.defaultProject - The default Harbor project used for repository mirror. Defaults to docker.io and assumes the user will configure the project as a proxy_cache for pulling images from Docker Hub.

Configure Harbor to act as both a transparent proxy and Private Registry for multiple repositories:

Create a Proxy Cache project (i.e. docker.io) as pull-through cache to Docker Hub
Create DNS alias (i.e. docker-mirror.domain.local) which points to the DNS name of Harbor
Create wildcard DNS alias (i.e. *.docker-mirror.domainlocal) which points to the DNS name of Harbor
Modify the nginx.conf file for the harbor-nginx pod to rewrite the URLs for /v2 and /service when using the DNS alias
Configure Docker daemon to use DNS alias as registry-mirror (or use DNS subdomain of Harbor project) (NOTE: You must specify "insecure-registries" if your SSL cert is not configured for wildcard DNS subdomains)

Modifications to nginx.conf

Put in http { block

# Map for repository mirroring
map $host $harbor_project {
default 0;
~^docker-mirror.domain.local$ docker.io;
~^(?<project>.+).docker-mirror.domain.local$ $project;
}
map $request_uri $new_uri {
~^/v2/(.+)$ /v2/$harbor_project/$1;
}
map $args $new_args {
~^(?<prefix2>.*scope=repository%3A)(?<suffix2>.*)$     ${prefix2}${harbor_project}%2F$suffix2;
}
map $upstream_http_www_authenticate $new_header {
~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
}

Put in second server { block

# Rewrite for repository mirroring
if ($harbor_project != 0) {
  rewrite ^/v2/(.+)$ $new_uri;
  set $args $new_args;
}

Put in location /v2/ { block
```
  # Modify headers for repository mirroring
  proxy_hide_header Www-Authenticate;
  add_header Www-Authenticate $new_header always;
```
Additional context The goal is to incorporate this directly into the Helm chart and existing nginx configurations to avoid adding additional services and endpoints which will add complexity to the deployment.

james-d-elliott commented 3 years ago

For docker official images they exist as library/traefik (you can pull traefik directly, it refers to library/traefik) for example. Not sure if this is the best way to handle it but if you add this above the section in the server block pasted here:

    if ($request_uri ~* ^/v2/([a-zA-Z0-9]+)/(manifests|blobs|tags)/?(.*)$) {
        set $dockerio_extra_project "/library";
    }

Then use this map instead:

  map $request_uri $new_uri {
    ~^/v2/(.+)$ /v2/$harbor_project$dockerio_extra_project/$1;
  }

Everything works as expected.

DASXCE commented 3 years ago

@jgallucci32 thank you for the solution! Just one comment... This:

Put in second server { block

confused me as the second server block is the one for the 8080 port which does the redirect to the first server block listening on HTTPS, so i put the if block in the first server block as HTTPS is my entry point.

@james-d-elliott

For docker official images they exist as library/traefik (you can pull traefik directly, it refers to library/traefik) for example. Not sure if this is the best way to handle it but if you add this above the section in the server block pasted here:
    if ($request_uri ~* ^/v2/([a-zA-Z0-9]+)/(manifests|blobs|tags)/?(.*)$) {
        set $dockerio_extra_project "/library";
    }
Then use this map instead:
  map $request_uri $new_uri {
    ~^/v2/(.+)$ /v2/$harbor_project$dockerio_extra_project/$1;
  }
Everything works as expected.

This is not needed if you use the docker cli. For example doing a :

docker pull traefik:latest

Docker will generate these HTTP requests:

GET /service/token?account=robot%24library&scope=repository%3Alibrary%2Ftraefik%3Apull&service=harbor-registry
HEAD /v2/library/traefik/manifests/latest
GET /v2/library/traefik/manifests/sha256:xyz...

DASXCE commented 3 years ago

One other note which would have helped me... The Proxy Cache project, or any other non proxy cache project must be public! Unfortunately docker itself does not support auth for registry-mirrors. The GET token request always sets the account to robot$library instead of selecting proper credentials.

GET /service/token?account=robot%24library&scope=repository...

More info here https://github.com/moby/moby/issues/30880

dm0610 commented 3 years ago

Hi. We had a problem with the url sended from containerd to harbor-nginx. Containerd add the key ?ns=docker.io host: harbor.vk.com mirror: dockerhub.harbor.vk.com url: https://harbor.vk.com/v2/dockerhub/library/nginx/manifests/stable?ns=docker.io. Server return error (no artefact) to fix this problem add the following configs to your nginx.conf

# to http {
# it's already exists
  map $request_uri $new_uri {
    ~^/v2/(.+)$ /v2/$harbor_project/$1;
  }
#add it below
  map $new_uri $new_new_uri {
    default $new_uri
    ~^/v2/(.+)\?ns=docker\.io(.*)$ /v2/$1$2;
  }
 #to server
     # Rewrite for repository mirroring
    if ($harbor_project != 0) {
      rewrite ^/v2/(.+)$ $new_new_uri;
      set $args $new_args;
    }

ricardojdsilva87 commented 3 years ago

Hi. We had a problem with the url sended from containerd to harbor-nginx. Containerd add the key ?ns=docker.io host: harbor.vk.com mirror: dockerhub.harbor.vk.com url: https://harbor.vk.com/v2/dockerhub/library/nginx/manifests/stable?ns=docker.io. Server return error (no artefact) to fix this problem add the following configs to your nginx.conf
# to http {
# it's already exists
  map $request_uri $new_uri {
    ~^/v2/(.+)$ /v2/$harbor_project/$1;
  }
#add it below
  map $new_uri $new_new_uri {
    default $new_uri
    ~^/v2/(.+)\?ns=docker\.io(.*)$ /v2/$1$2;
  }
 #to server
     # Rewrite for repository mirroring
    if ($harbor_project != 0) {
      rewrite ^/v2/(.+)$ $new_new_uri;
      set $args $new_args;
    }

Hello all, This extra setting works perfectly for crictl and containerd. We are using kops with a kubernetes version 1.19.13 with the containerd as the engine.

Thanks!

ricardojdsilva87 commented 3 years ago

Hello all, just an update with the code for nginx that enabled us to use Harbor as a dockerproxy with containerd and docker.

We are using NGINX helm chart, and added the following in the http:

    http-snippet: |
      # Map for docker.io proxy cache
      map $host $harbor_project {
        default 0;
        ~^dockerproxy(.*)xxxxxx$ proxy;
      }
      map $request_uri $new_harbor_uri {
        ~^/v2/(.+)$ /v2/$harbor_project/$1;
      }
      map $new_harbor_uri $new_containerd_uri {
        default $new_harbor_uri;
        ~^/v2/(.+)\?ns=docker\.io(.*)$ /v2/$1$2;
      }
      map $args $new_harbor_args {
        ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>.*)$     ${prefix2}${harbor_project}%2F$suffix2;
      }
      map $upstream_http_www_authenticate $new_harbor_header {
        ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$http_host$suffix1;
      }

Added after the following annotations on the proxy ingress:

  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      # Rewrite for docker.io proxy cache
      if ($harbor_project != 0) {
        rewrite ^/v2/(.+)$ $new_harbor_uri;
        rewrite ^/v2/(.+)$ $new_containerd_uri;
        set $args $new_harbor_args;
      }
    nginx.ingress.kubernetes.io/configuration-snippet: |
      # Modify headers for proxy cache  
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $new_harbor_header always;

Hope it helps

github-actions[bot] commented 2 years ago

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

goharbor / harbor

Support for private registry mirrors for multiple repositories #13579