Docker registry configuration is ignored by Kolla

[jonglezb]

The docker registry configuration in reservation.yaml is no longer taken into account by Kolla.

For instance, with the following configuration in Grid'5000, the docker cache is never actually used:

registry:
  ip: docker-cache.grid5000.fr
  port: 80
  type: external

From https://intranet.grid5000.fr/bugzilla/show_bug.cgi?id=13121#c2 this is what happens:

TASK [baremetal : Merge custom docker config] **********************************
ok: [grisou-25-kavlan-7.nancy.grid5000.fr]
ok: [grisou-24-kavlan-7.nancy.grid5000.fr]
ok: [grisou-20-kavlan-7.nancy.grid5000.fr]

TASK [baremetal : Write docker config] *****************************************
--- before: /etc/docker/daemon.json
+++ after: /home/ajenkins/.ansible/tmp/ansible-local-8216xd3w41m0/tmptugzcdoc
@@ -1,5 +1,6 @@
 {
-  "debug": true,
-  "registry-mirrors": ["[http://docker-cache.lille.grid5000.fr:80](http://docker-cache.lille.grid5000.fr/)"],
-  "insecure-registries": ["[http://docker-cache.lille.grid5000.fr:80](http://docker-cache.lille.grid5000.fr/)"]
+    "log-opts": {
+        "max-file": "5",
+        "max-size": "50m"
+    }
 }
\ No newline at end of file

It's caused by this change in Kolla : https://opendev.org/openstack/kolla-ansible/commit/a5808ad8ba8e0c9cd02d5f4ec3e1f073c332a7d3

[jonglezb]

In the Grid'5000 bug report above, @msimonin suggested the following workaround, which works:

kolla:
  docker_custom_config:
    registry-mirrors:
      - "http://docker-cache.grid5000.fr"
    insecure-registries:
      - "http://docker-cache.grid5000.fr"

Either we need to document this workaround, or fix the bug and make sure that Kolla takes our configuration into account.

[rcherrueau]

Thanks @jonglezb for reporting the problem.

I confirm that the Docker daemon configuration file /etc/docker/daemon.json, that is setup by enoslib at lines 120-137 [0], is then overwritten by the call of kolla-ansible bootstrap-servers at lines 156-160 [1].

The root cause is a task in the post-install.yml file [2].

This affects internal and external Docker registry mirrors.

I am currently testing a fix.

[0] https://github.com/BeyondTheClouds/enos/blob/bdf4ae2ed4356d70cc2b7fdda31e5accfe55739a/enos/tasks/up.py#L120-L137 [1] https://github.com/BeyondTheClouds/enos/blob/bdf4ae2ed4356d70cc2b7fdda31e5accfe55739a/enos/tasks/up.py#L158-L162 [2] https://github.com/openstack/kolla-ansible/blob/stable/ussuri/ansible/roles/baremetal/tasks/post-install.yml#L135-L145

[rcherrueau]

@jonglezb, I have pushed a fix. I checked it on top of libvirt with an internal docker registry mirror. On your side, would you be okay to test on top of G5k with the external G5k mirror?

To test it (you will need poetry[0])

git clone https://github.com/rcherrueau/enos.git
cd enos
poetry install
poetry run enos new --provider=g5k
poetry run enos up

At the end of the UP phase, make sure that the docker daemon configuration file (i.e., /etc/docker/daemon.json) contains the G5k mirror. Something like:

poetry run enos info -o json | jq -r ".rsc.all[].address" |\
   xargs -I{} ssh -lroot {} cat /etc/docker/daemon.json

[0] https://python-poetry.org/docs/#installation

[rcherrueau]

@msimonin would you mind trying this on G5k?

[msimonin]

Hi @rcherrueau, long time :) and thanks for this update. Can you believe it, there are still some people using EnOS on G5K !