docker-1.13.1-35.git8fd0ebb not respecting 'insecure_registries' in /etc/containers/registries.conf

[miabbott]

On the latest CAHC version (7.2017.839), it's been observed that configuring insecure_registries in /etc/containers/registries.conf is not being respected.

# rpm-ostree status
State: idle
Deployments:
● centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
                   Version: 7.2017.839 (2017-11-07 16:11:56)
                    Commit: ced0f451cfb6cbbd1ac84599311dbc2c248a61eca50504c3325736f2558d2b73

# rpm -q atomic-registries docker
atomic-registries-1.19.1-5.git48c224b.el7.centos.x86_64
docker-1.13.1-35.git8fd0ebb.el7.x86_64

# cat /etc/containers/registries.conf 
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to YAML format and does not support recursive
# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: 'registries', 'insecure_registies', 
# and 'block_registries'.

#registries:
#  - registry.access.redhat.com

# If you need to access insecure registries, uncomment the section below
# and add the registries fully-qualified name. An insecure registry is one
# that does not have a valid SSL certificate or only does HTTP.
insecure_registries:
  - 172.17.0.1:5000

# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#block_registries:
#  - 

# docker info | grep -A 3 Insecure
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

# ps -ww 12930
  PID TTY      STAT   TIME COMMAND
12930 ?        Ssl    0:00 /usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-current --default-runtime=oci --authorization-plugin=rhel-push-plugin --containerd /run/containerd.sock --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --log-driver=journald --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/atomicos-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true

# journalctl -b -u docker
Nov 07 16:41:59 micah-cahc-vm1107a systemd[1]: Starting Docker Application Container Engine...
Nov 07 16:41:59 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:41:59Z" level=info msg="SUSE:secrets :: enabled"
Nov 07 16:41:59 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:41:59.928276717Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 07 16:41:59 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:41:59.928927062Z" level=info msg="Loading containers: start."
Nov 07 16:41:59 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:41:59.936669471Z" level=info msg="Firewalld running: false"
Nov 07 16:42:00 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:42:00.007398951Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be 
Nov 07 16:42:00 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:42:00.029950185Z" level=info msg="Loading containers: done."
Nov 07 16:42:00 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:42:00.037354988Z" level=warning msg="failed to retrieve docker-init version"
Nov 07 16:42:00 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:42:00.037708093Z" level=info msg="Daemon has completed initialization"
Nov 07 16:42:00 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:42:00.037733925Z" level=info msg="Docker daemon" commit=1378b5f-unsupported graphdriver=devicemapper version=1.13.1
Nov 07 16:42:00 micah-cahc-vm1107a systemd[1]: Started Docker Application Container Engine.
Nov 07 16:42:00 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:42:00.060926588Z" level=info msg="API listen on /var/run/docker.sock"
Nov 07 16:42:01 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:42:01.888217816Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=13024}"
Nov 07 16:42:01 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:42:01.895790842Z" level=warning msg="failed to retrieve docker-init version"
Nov 07 16:56:04 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:56:04.669586519Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=13069}"
Nov 07 16:56:04 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:56:04.683185684Z" level=warning msg="failed to retrieve docker-init version"
Nov 07 16:56:11 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:56:11.025890953Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=13077}"
Nov 07 16:56:11 micah-cahc-vm1107a dockerd-current[12930]: time="2017-11-07T16:56:11.032608056Z" level=warning msg="failed to retrieve docker-init version"
-- Logs begin at Tue 2017-11-07 16:37:28 UTC, end at Tue 2017-11-07 16:56:11 UTC. --
Nov 07 16:38:33 micah-cahc-vm1107a systemd[1]: Starting Docker Application Container Engine...
Nov 07 16:38:33 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:38:33Z" level=info msg="SUSE:secrets :: enabled"
Nov 07 16:38:33 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:38:33.894335172Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 07 16:38:33 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:38:33.894991468Z" level=info msg="Loading containers: start."
Nov 07 16:38:33 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:38:33.973593177Z" level=info msg="Firewalld running: false"
Nov 07 16:38:34 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:38:34.197270760Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be u
Nov 07 16:38:34 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:38:34.244452323Z" level=info msg="Loading containers: done."
Nov 07 16:38:34 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:38:34.256796746Z" level=warning msg="failed to retrieve docker-init version"
Nov 07 16:38:34 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:38:34.257335554Z" level=info msg="Daemon has completed initialization"
Nov 07 16:38:34 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:38:34.257353186Z" level=info msg="Docker daemon" commit=1378b5f-unsupported graphdriver=devicemapper version=1.13.1
Nov 07 16:38:34 micah-cahc-vm1107a systemd[1]: Started Docker Application Container Engine.
Nov 07 16:38:34 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:38:34.305263604Z" level=info msg="API listen on /var/run/docker.sock"
Nov 07 16:41:20 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:41:20.515877659Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=12781}"
Nov 07 16:41:20 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:41:20.527457992Z" level=warning msg="failed to retrieve docker-init version"
Nov 07 16:41:59 micah-cahc-vm1107a systemd[1]: Stopping Docker Application Container Engine...
Nov 07 16:41:59 micah-cahc-vm1107a dockerd-current[1095]: time="2017-11-07T16:41:59.689278461Z" level=info msg="Processing signal 'terminated'"

This doesn't appear to be happening on Fedora or RHEL, but specific to CentOS.

The docker build is coming from the virt7-containers-common-candidate repo.

http://cbs.centos.org/repos/virt7-container-common-candidate/

And this is the specific build:

http://cbs.centos.org/koji/buildinfo?buildID=20385

[miabbott]

FYI @lsm5 @mike-nguyen @ashcrow

[runcom]

@bbaude ptal

[runcom]

@baude rather I guess

[mike-nguyen]

This is also happening on the latest CentOS 7 Atomic Host (qcow here: http://cloud.centos.org/centos/7/atomic/images/CentOS-Atomic-Host-7-GenericCloud.qcow2) which is running an older version of docker.

[cloud-user@centos ~]$ rpm-ostree status
State: idle
Deployments:
● centos-atomic-host:centos-atomic-host/7/x86_64/standard
                Version: 7.1708 (2017-09-15 15:32:30)
                 Commit: 33b4f0442242a06096ffeffadcd9655905a41fbd11f36cd6f33ee0d974fdb2a8
           GPGSignature: 1 signature
                         Signature made Fri 15 Sep 2017 05:17:39 PM UTC using RSA key ID F17E745691BA8335
                         Good signature from "CentOS Atomic SIG <security@centos.org>"

[cloud-user@centos ~]$ rpm -qa docker*
docker-client-latest-1.13.1-21.1.gitcd75c68.el7.centos.x86_64
docker-1.12.6-48.git0fdc778.el7.centos.x86_64
docker-common-1.12.6-48.git0fdc778.el7.centos.x86_64
docker-latest-1.13.1-21.1.gitcd75c68.el7.centos.x86_64
docker-client-1.12.6-48.git0fdc778.el7.centos.x86_64
docker-lvm-plugin-1.12.6-48.git0fdc778.el7.centos.x86_64
docker-novolume-plugin-1.12.6-48.git0fdc778.el7.centos.x86_64

[ashcrow]

bump

[ashcrow]

/cc @jwhonce

[baude]

i'm on it

[baude]

In /usr/lib/systemd/system/docker.service, we need to add EnvironmentFile=/run/containers/registries.conf so that the registries get added properly. You can look at the Fedora or RH service file as an example.

[miabbott]

Based on IRC conversations with @lsm5 yesterday, he informed me that the docker package in virt7-containers-common-candidate is a rebuild of the Fedora Rawhide version from here:

http://pkgs.fedoraproject.org/cgit/rpms/docker.git

Lokesh said he would update the CentOS service file there and initiate a rebuild for CentOS afterwards.

[lsm5]

this got fixed few mins back.

[miabbott]

This got fixed here - http://pkgs.fedoraproject.org/cgit/rpms/docker.git/commit/docker.service.centos?id=957fa278aa5f152da2a588a1a28f650ba7fe807d