private registry in gcloud

[sdorosz]

I need some advice on how I can setup watchtower to monitor changes in images in my private gcloud registry.

i don't know how to setup the authentification for pulling.

at the moment I log into the gcloudregistry every time i pull an image.

thank you for your advice

[nomukaiki]

This might help you: https://cloud.google.com/container-registry/docs/advanced-authentication

[Jacobh2]

Following the steps for JSON file, and updating my compose file to: (Having the service account's JSON file one one line for the password)

  watchtower:
    image: v2tec/watchtower
    environment:
      - REPO_USER=_json_key
      - REPO_PASS={"type":"service_account", ..... 

yields a problem with watchtower when it is (what I presume) checking itself:

watchtower_1  | time="2018-07-26T09:11:04Z" level=info msg="Unable to update container /watchtower_watchtower_1, err='Error response from daemon: Get https://registry-1.docker.io/v2/v2tec/watchtower/manifests/latest: unauthorized: incorrect username or password'. Proceeding to next."

It is able to successfully check my private repo at gcloud and pull down and update my container, but not itself.

Is it possible for watchtower to still use the default cred when checking itself? Or is the solution that I host a version of watchtower in my container repo also?

EDIT: Seems like #55 is related to this?

[lokeshh]

@Jacobh2 Following your solution, I'm getting following error

ERROR: yaml.parser.ParserError: while parsing a block mapping
  in "./docker-compose.yml", line 32, column 9
expected <block end>, but found ','
  in "./docker-compose.yml", line 32, column 45

Could you help?

[Jacobh2]

@lokeshh Could you post the content of your docker-compose.yaml file? Just make sure to remove the actual content of the REPO_PASS value 😉

My first guess is that there is something wrong with the REPO_PASS env value. Make sure you have the json content on one line!

[lokeshh]

@Jacobh2 Following are the contents of my docker-compose.yml:

version: '3.2'
services:
  watchtower:
    image: v2tec/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --interval 10
    environment:
      - REPO_USER=_json_key
      - REPO_PASS={  "type": "service_account",  "project_id": "xxx",  "private_key_id": "xxx",  "private_key": "-----BEGIN PRIVATE KEY-----\nxxx==\n-----END PRIVATE KEY-----\n",  "client_email": "xxx",  "client_id": "xxx",  "auth_uri": "https://accounts.google.com/o/oauth2/auth",  "token_uri": "https://oauth2.googleapis.com/token",  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",  "client_x509_cert_url": "xxx"}

I get the following error when I run docker-compose up:

ERROR: yaml.parser.ParserError: while parsing a block mapping
  in "./docker-compose.yml", line 10, column 9
expected <block end>, but found ','
  in "./docker-compose.yml", line 10, column 47

Line 10, column 9 is the letter R of REPO_PASS and column 47 is first comma (,).

[lokeshh]

@Jacobh2 I removed all the spaces and it worked. Many thanks :+1:

[leandrocruz]

Do you have to replace "_json_key" by some value or should it be used literally?

[lokeshh]

@leandrocruz It has to be used literally.

[leandrocruz]

Thanks @lokeshh . I wonder if this authentication method still works. I created a service account with all privileges, but watchtower still can't pull the images. I have pasted the complete content of the json key (no new lines and no spaces) into the REPO_PASS environment variable, but it didn't work.

[leandrocruz]

I have also tried to share the "config.json" , but no luck.

[Jacobh2]

@leandrocruz I have actually updated the way I use this to instead mount my .docker/config.json file directly!

My docker-compose file now looks something like this:

version: '3'
services:
  watchtower:
    image: <image>
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /home/client/.docker/config.json:/config.json
    command: --interval 10
    restart: always

where /home/client/.docker/config.json is the path to your user's config.json file. The config.json file was created by running

docker login -u _json_key --password-stdin https://eu.gcr.io < gcloudauth.json

where gcloudauth.json is the gcloud service-account's json file. More info here

[simskij]

Closing due to lack of activity

[greenspray]

I'm still having issue with gcloud. I used docker-credential-gcr configure-docker to successfully login in my server but I get authentication error.

[polaroidkidd]

I am facing the same issue as well. Is there any way to supply you with more information?

[Jacobh2]

Are you using a gcp user with the correct privileges? Are you able to sign in with that user manually and try to do a docker pull to verify that it should have access?

[polaroidkidd]

Currently I run the docker-compose pull command to get the latest version of my images without a hitch (when I'm in the directory containing the docker-compose.yml file). I have not checked weather I have the correct privileges or not yet. Let me check and get back to you.

[greenspray]

I can definitely do a normal docker pull

[polaroidkidd]

Right, so I've tried going through the various steps (stop all docker services, wipe system, restart docker service, authenticate using the keyfile and boot everything up again). Unfortunately I keep on getting the same error message.

time="2020-02-06T21:22:51Z" level=info msg="Unable to update container /dle-cloud, err='Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication'. Proceeding to next." 

for every private container. The docker-compose pull command ran through without a hitch. I'm not sure what other information I could give you at this point which would help debug this. Is there a verbose logs option for watchtower?

EDIT:

I realised I wasn't using the containerrrr version. I've updated the docker-compose file to match @Jacobh2 but am now getting the error

dle-watchtower     | time="2020-02-06T21:30:48Z" level=info msg="Unable to update container /dle-site. Proceeding to next."

for all the containers which use my private registry. The containers in public registries never pop up in the logs.

[simskij]

@polaroidkidd, could you please upload the docker-compose config for watchtower and at least one of the private containers you want to keep updated? We're kind of at a loss here due to the lack of knowledge about your system.

This part of the documentation should be sufficient to get you up and running with private registries, but we might have missed something for gcloud in particular.

Thanks 🙏🏼 Simon

[polaroidkidd]

Hi

Here's my complete configuration:

version: '3.7'

services:
###################################################
  ## BEGIN REVERSE PROXY AND CERTS ##
  dle-nginx:
    image: eu.gcr.io/dle-dev/nginx:latest
    container_name: dle-nginx
    restart: unless-stopped
    volumes:
      - ./volumes/proxy-certs:/etc/letsencrypt
      - ./volumes/proxy-logs:/var/log/nginx
      - ./volumes/proxy-acme:/etc/certbot
    ports:
      - "80:80"
      - "443:443"
      - "32400:32400"
    depends_on:
      - dle-certbot
      - dle-site
      - dle-cloud
      - dle-cloud-db
      - dle-qbt
      - dle-plex
      - dle-cloud-redis
      - dle-jackett
      - dle-radarr
      - dle-sonarr
      - dle-watchtower
    environment:
      - VPS_REDICRECT_HTTPS=${VPS_REDICRECT_HTTPS}

  dle-certbot:
    image: eu.gcr.io/dle-dev/certbot-dns-digitalocean:v0.37.2
    container_name: dle-certbot
    restart: unless-stopped
    volumes:
      - ./volumes/proxy-certs:/etc/letsencrypt
      - ./volumes/proxy-acme:/etc/certbot
    environment:
      - VPS_REDICRECT_HTTPS=${VPS_REDICRECT_HTTPS}

###################################################
  ## BEGINE WEBSITE dle.dev ##
  dle-site:
    image: eu.gcr.io/dle-dev/frontend:1.1.0
    container_name: dle-site
    restart: unless-stopped
    ports:
      - 80

###################################################
  ## BEGIN NEXTCLOUD ##
  dle-cloud:
    image: eu.gcr.io/dle-dev/linuxserver-nextcloud:latest
    container_name: dle-cloud
    depends_on:
      - dle-cloud-db
    volumes:
      - ./volumes/cloud/html:/var/www/html
      - ./volumes/cloud/config:/var/www/html/config
      - ./volumes/cloud/custom-apps:/var/www/html/custom_apps
      - ./volumes/cloud/data:/var/www/html/data
      - ./volumes/cloud/themes:/var/www/html/themes
      - /etc/localtime:/etc/localtime:ro
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_HOST=dle-cloud-db
    restart: unless-stopped
    ports:
      - 5678:80

  dle-cloud-db:
    image: mariadb
    container_name: dle-cloud-db
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    volumes:
      - cloud-db:/var/lib/mysql
      - /etc/localtime:/etc/localtime:ro
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
    restart: unless-stopped

  dle-cloud-redis:
    image: redis
    container_name: dle-redis
    restart: unless-stopped
    ports:
      - 6379

###################################################
## BEGIN qBitTorrent ##
  dle-qbt:
    image: linuxserver/qbittorrent:4.2.0201911300248-6791-7e0d642ubuntu18.04.1-ls14
    container_name: "dle-qbt"
    volumes:
      - ./volumes/qbt/config:/config
      - /home/dle/Media/downloads:/downloads
      - /home/dle/Media:/Media
      - ./volumes/qbt/shared:/shared
    expose:
      - "8082"
    ports:
      - "6881:6881"
      - "6881:6881/udp"
    restart: unless-stopped
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
      - UMASK_SET=002
      - WEBUI_PORT=8082

###################################################
    ## BEGIN PLEX ##
  dle-plex:
    image: plexinc/pms-docker:latest
    container_name: dle-plex
    environment:
      - ADVERTISE_IP=${ADVERTISE_IP}
      - TZ=Europe/Zurich
      - PLEX_CLAIM=claim-${CLAIM_TOKEN}
      - PUID=1000
      - PGID=1000
      - VERSION=docker
    volumes:
      - ./volumes/plex/config:/config
      - ./volumes/plex/transcode:/transcode
      - /home/dle/Media:/data
    restart: unless-stopped
    expose:
      - 32400/tcp
      - 3005/tcp
      - 8324/tcp
      - 32469/tcp
      - 1900/udp
      - 32410/udp
      - 32412/udp
      - 32413/udp
      - 32414/udp

#################################################
### BEGIN RADARR ###
  dle-radarr:
    image: linuxserver/radarr:latest
    container_name: dle-radarr
    volumes:
      - ./volumes/radarr/config:/config
      - /home/dle/Media/downloads:/downloads
      - /home/dle/Media/Movies:/Media/Movies
      - "/etc/localtime:/etc/localtime:ro"
      - ./volumes/radarr/shared:/shared
    ports:
      - 7878
    restart: unless-stopped
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London

#################################################
### BEGIN SONARR ###
  dle-sonarr:
    image: linuxserver/sonarr:latest
    container_name: dle-sonarr
    volumes:
      - ./volumes/sonarr/config:/config
      - /home/dle/Media/downloads:/downloads
      - /home/dle/Media/Series:/Media/Series
      - "/etc/localtime:/etc/localtime:ro"
      - ./volumes/sonarr/shared:/shared
    expose:
      - 8989
    restart: unless-stopped
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London

#################################################
### BEGIN JACKETT ###
  dle-jackett:
    image: linuxserver/jackett:latest
    container_name: dle-jackett
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
    volumes:
      - ./volumes/jackett/config:/config
      - /home/dle/Media/downloads:/downloads
    expose:
      - 9117
    restart: unless-stopped

###################################################
#### BEGIN WATCHTOWER ###

###################################################
#### BEGIN WATCHTOWER ###
  dle-watchtower:
    image:  containrrr/watchtower:latest
    container_name: dle-watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /home/dle/.docker/config.json:/config.json
        # command: --interval 10    
    command: --schedule "0 0 4 * * *" --cleanup

volumes:
  cloud-db:

actually, give me a minute. I think I got it.

Edit:

Nope. Trying to figure out how to load the google credential helper into a separate docker file so i can mount it as per the example now. This will probably be it. I'll close it for now and should i figure this out, post my solution here.

[greenspray]

Any updates yet on this issue?

[polaroidkidd]

@greenspray I'm afraid not. I tried and failed to follow the documentation and adapting it for the google private docker registry.

If it's any help, I've been using the gcloud credential helper, but couldn't get that one to work in the docker container. I'll try Standalone Credential Helper next and work my way down the list here https://cloud.google.com/container-registry/docs/advanced-authentication

If I managed to get it working, should I submit at PR for an update to the documentation or just post it here?

[simskij]

Yes please, a PR would be most welcome!

One thing to note is that you, if you build it from source, likely need to disable CGO, or the resulting binary wont be executable in the watchtower containrrr (which uses a scratch image rather than a full os)

[seirius]

I don't know if anyone figured out this yet but I found out how to make it work. I used the manual file configuration as the watchtower documentation points out https://containrrr.github.io/watchtower/private-registries/ Create a base64 string with echo -n 'username:password' | base64 For username use _json_key and for password the gcloudauth.json file's content. Add the string to config.json file as value of auth key for your domain, in my case:

{
    "auths": {
        "eu.gcr.io": {
                    "auth": "yourBase64String"
                 }
    },
    "HttpHeaders": {
        "User-Agent": "Docker-Client/19.03.5 (darwin)"
    },
    "credsStore": "desktop",
    "credHelpers": {
        "eu.gcr.io": "gcloud"
    },
    "experimental": "disabled",
    "stackOrchestrator": "swarm"
}

[simskij]

I don't know if anyone figured out this yet but I found out how to make it work. I used the manual file configuration as the watchtower documentation points out https://containrrr.github.io/watchtower/private-registries/ Create a base64 string with echo -n 'username:password' | base64 For username use _json_key and for password the gcloudauth.json file's content. Add the string to config.json file as value of auth key for your domain, in my case:

{
  "auths": {
      "eu.gcr.io": {
                    "auth": "yourBase64String"
                 }
  },
  "HttpHeaders": {
      "User-Agent": "Docker-Client/19.03.5 (darwin)"
  },
  "credsStore": "desktop",
  "credHelpers": {
      "eu.gcr.io": "gcloud"
  },
  "experimental": "disabled",
  "stackOrchestrator": "swarm"
}

Thank you so much! 🙏🏼

Would you mind providing a PR that adds this to the docs?

[simskij]

I added a little something to the docs. Hope it helps.