Reprovisioning of a running container fails if network_mode is host

[horkhe]

Cookbook version: v2.3.5

When a running container that has network_mode host is reprovisioned the following error is reported:

==> default: * docker_container[kafka] action run
==> default: 
==> default:     - stopping kafka 
==> default: 
==> default:     - deleting kafka
==> default: 
==> default:     
==> default: ================================================================================
==> default:     Error executing action `run` on resource 'docker_container[kafka]'
==> default:     ================================================================================
==> default:     
==> default:     Docker::Error::ConflictError
==> default:     ----------------------------
==> default:     Conflicting options: -h and the network mode (--net)
==> default:     
==> default:     Cookbook Trace:
==> default:     ---------------
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/files/default/vendor/gems/docker-api-1.24.1/lib/docker/connection.rb:48:in `rescue in request'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/files/default/vendor/gems/docker-api-1.24.1/lib/docker/connection.rb:38:in `request'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/files/default/vendor/gems/docker-api-1.24.1/lib/docker/connection.rb:65:in `block (2 levels) in <class:Connection>'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/files/default/vendor/gems/docker-api-1.24.1/lib/docker/container.rb:246:in `create'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/libraries/docker_container.rb:181:in `block (3 levels) in <class:DockerContainer>'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/libraries/helpers_base.rb:57:in `call'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/libraries/helpers_base.rb:57:in `with_retries'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/libraries/docker_container.rb:180:in `block (2 levels) in <class:DockerContainer>'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/compat_resource/files/lib/chef_compat/copied_from_chef/chef/provider.rb:50:in `converge_if_changed'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/libraries/docker_container.rb:177:in `block in <class:DockerContainer>'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/libraries/docker_container.rb:159:in `call_action'
==> default:     /tmp/vagrant-chef/910d0cdeee32b5308e0471021b3346d5/cookbooks/docker/libraries/docker_container.rb:272:in `block in <class:DockerContainer>'
==> default:     
==> default:     Resource Declaration:
==> default:     ---------------------
==> default:     # In /tmp/vagrant-chef/1d5ee6bed7140dd82057d90f338645b1/cookbooks/mg-kafka/recipes/default.rb
==> default:     
==> default:      64: docker_container "kafka" do
==> default:      65:   action :run
==> default:      66:   detach true
==> default:      67:   container_name "kafka"
==> default:     
==> default:  68:   network_mode "host"
==> default: 
==> default:     
==> default:  69:   entrypoint "/opt/kafka/bin/kafka-server-start.sh"
==> default: 
==> default:     
==> default:  70:   command "/opt/kafka/config/server.properties"
==> default: 
==> default:      71:   binds [
==> default:      72:     "#{node["mg-kafka"]["config_dir"]}/server.properties:/opt/kafka/config/server.properties",
==> default:      73:     "#{node["mg-kafka"]["config_dir"]}/log4j.properties:/opt/kafka/config/log4j.properties",
==> default:      74:     "#{node["mg-kafka"]["data_dir"]}:/var/kafka",
==> default:      75:     "#{node["mg-kafka"]["log_dir"]}:/var/log/kafka"
==> default:      76:   ]
==> default:      77:   repo "registry.postgun.com:5000/mailgun/kafka"
==> default:      78:   tag node["mg-kafka"]["image_tag"]
==> default:      79: end
==> default:      80: 
==> default:     
==> default:     Compiled Resource:
==> default:     ------------------
==> default:     # Declared in /tmp/vagrant-chef/1d5ee6bed7140dd82057d90f338645b1/cookbooks/mg-kafka/recipes/default.rb:64:in `from_file'
==> default:     
==> default:     docker_container("kafka") do
==> default:       action [:run]
==> default:       retries 0
==> default:       retry_delay 2
==> default:       default_guard_interpreter :default
==> default:       declared_type :docker_container
==> default:       cookbook_name :"mg-kafka"
==> default:       recipe_name "default"
==> default:       detach true
==> default:       container_name "kafka"
==> default:       entrypoint "/opt/kafka/bin/kafka-server-start.sh"
==> default:       command "/opt/kafka/config/server.properties"
==> default:       binds ["/etc/kafka/server.properties:/opt/kafka/config/server.properties", "/etc/kafka/log4j.properties:/opt/kafka/config/log4j.properties", "/var/mailgun/kafka:/var/kafka", "/var/log/kafka:/var/log/kafka"]
==> default:       repo "registry.postgun.com:5000/mailgun/kafka"
==> default:       tag "v0.8.2.2"
==> default:       network_mode "host"
==> default:       connection #<Docker::Connection:0x0000000640e2b8 @url="unix:///", @options={:socket=>"/var/run/docker.sock", "read_timeout"=>60}>
==> default:     end
==> default:     
==> default: [2015-12-09T23:46:07+00:00] INFO: Running queued delayed notifications before re-raising exception
==> default: 
==> default: Running handlers:
==> default: [2015-12-09T23:46:07+00:00] ERROR: Running exception handlers
==> default: Running handlers complete
==> default: [2015-12-09T23:46:07+00:00] ERROR: Exception handlers complete
==> default: Chef Client failed. 15 resources updated in 01 minutes 18 seconds
==> default: [2015-12-09T23:46:07+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
==> default: [2015-12-09T23:46:07+00:00] ERROR: docker_container[kafka] (mg-kafka::default line 64) had an error: Docker::Error::ConflictError: Conflicting options: -h and the network mode (--net)
==> default: [2015-12-09T23:46:07+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Chef never successfully completed! Any errors should be visible in the
output above. Please fix your recipes so that they properly complete.

[chasebolt]

this is a dupe of #542 and referenced to be fixed in #557

[someara]

I'm not seeing any host related things on the resource in the stack trace... this is probably a real bug

[someara]

can you paste the output of "docker info" ?

[chasebolt]

@someara I dealt with this in the past and going off of memory here, this is an issue with docker 1.9.0.

I think it is because we pass 'Hostname' => hostname to the docker create, prior to 1.9.0 docker would ignore it because it is nil. Now docker will actually complain about it because we are passing it during the create method.

[horkhe]

@someara here you are:

$ docker info
Containers: 2
Images: 51
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-8:1-530890-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: 
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.925 GB
 Data Space Total: 107.4 GB
 Data Space Available: 26.47 GB
 Metadata Space Used: 4.039 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.143 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-71-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 1
Total Memory: 2.939 GiB
Name: mailgun-dev
ID: XTCU:ZBIE:2ISX:CRJY:23XC:YAPY:37ZJ:737S:BNVL:4IMT:BGXS:CPWX
WARNING: No swap limit support

[horkhe]

It is still reproducible in 2.3.19

[someara]

Just ran into this myself =)

Investigating...

-s

On Wed, Dec 30, 2015 at 2:53 PM, Maxim Vladimirskiy < notifications@github.com> wrote:

It is still reproducible in 2.3.19

— Reply to this email directly or view it on GitHub https://github.com/chef-cookbooks/docker/issues/572#issuecomment-168063989 .

[chasebolt]

@someara take a look over at #542. I have a feeling these two are related.

[someara]

Verified with pry - https://gist.github.com/someara/2a7a06e2459a75f3612d#file-gistfile1-txt-L25 Adding coerce_hostname in a12066c58bbe6ab921db54754565b447018f1a9e Released as 2.3.20

[someara]

had to revert 2.3.20... this is still broken as of 2.3.21.

[someara]

Okay, we should be good to go in v2.3.22

[chasebolt]

Leave #542 open and I'll fix that when I return. There is a bigger issue with load_current_value and the properties of a container. Fixing this hostname issue as you did is more of just a bandaid. All the properties exhibit this same behavior.

[someara]

I guess this would apply to all the network_mode validation properties as well.. dns, dns_search, mac_address, and extra_hosts

On Sat, Jan 2, 2016 at 7:12 PM, Chase Bolt notifications@github.com wrote:

Leave #542 https://github.com/chef-cookbooks/docker/issues/542 open and I'll fix that when I return. There is a bigger issue with load_current_value and the properties of a container. Fixing this hostname issue as you did is more of just a bandaid. All the properties exhibit this same behavior.

— Reply to this email directly or view it on GitHub https://github.com/chef-cookbooks/docker/issues/572#issuecomment-168441858 .

[someara]

Let's track this in #542. Closing here.