Building second set of overcloud container images fails

If we build a second set of images using a new tag we come up against a docker overlay storage driver limitation:

ERROR:kolla.image.build.nova-libvirt:Error'd with the following message
ERROR:kolla.image.build.nova-libvirt:link /var/lib/docker/overlay/e1251f91bd1949691006a20cf47c8ae73dbe02d230f1fd4864b37c664907f880/root/var/lib/yum/yumdb/h/6e65f96e8fd137ed
72729a1e39f219905779eb91-hardlink-1.0-19.el7-x86_64/checksum_type /var/lib/docker/overlay/0d9d6d7a3d39471dff211b82d24ffce4f3e756fe9e9464d60cee8a1bb3f243fa/tmproot290834309/
var/lib/yum/yumdb/h/6e65f96e8fd137ed72729a1e39f219905779eb91-hardlink-1.0-19.el7-x86_64/checksum_type: too many links

The current workaround is delete the old set of images. A better solution would be to use another storage driver. I briefly tested overlay2:

(kayobe) [root@seed kayobe]# cat /etc/docker/daemon.json 
{
  "storage-driver": "overlay2",
   "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}

but there is an issue with the version of docker kolla-ansible installs:

Docker version 1.12.0

When the container images are built with this storage engine, you get spurious file permission errors when attempting to run the containers:

RUNNING HANDLER [common : Initializing toolbox container using normal user] ************************************************************************************************
fatal: [control-0]: FAILED! => {"changed": false, "cmd": ["docker", "exec", "-t", "kolla_toolbox", "ansible", "--version"], "delta": "0:00:00.150262", "end": "2018-09-14 18:21:35.929873", "failed": true, "rc": 1, "start": "2018-09-14 18:21:35.779611", "stderr": "", "stderr_lines": [], "stdout": "Traceback (most recent call last):\r\n File \"/opt/ansible/bin/ansible\", line 43, in <module>\r\n    import ansible.constants as C\r\nImportError: No module named ansible.constants", "stdout_lines": ["Traceback (most recent call last):", "  File \"/opt/ansible/bin/ansible\", line 43, in <module>", " import ansible.constants as C", "ImportError: No module named ansible.constants"]

The solution may be to upgrade docker (which kolla-ansible doesn't actually pin).

stackhpc / sib-kayobe-config

Building second set of overcloud container images fails #9