Unexpected behaviour when running from Cloudbees CI

[ninjatux]

When we execute packer build locally with chef_solo everything works as expected, the amazon-ebs builder uses chef to provision the machine and then packer creates the image.

However when we run the same code on Jenkins (hosted on Cloudbees) we have problems as you can see in the logs:

amazon-ebs: Recipe: nginx::default
    amazon-ebs: * service[nginx] action reload
    amazon-ebs: - reload service service[nginx]
    amazon-ebs: * service[nginx] action restart
    amazon-ebs:
    amazon-ebs: ================================================================================
    amazon-ebs: Error executing action `restart` on resource 'service[nginx]'
    amazon-ebs: ================================================================================
    amazon-ebs:
    amazon-ebs: Mixlib::ShellOut::ShellCommandFailed
    amazon-ebs: ------------------------------------
    amazon-ebs: Expected process to exit with [0], but received '1'
    amazon-ebs: ---- Begin output of /sbin/service nginx restart ----
    amazon-ebs: STDOUT: Stopping nginx:
    amazon-ebs: Starting nginx:
    amazon-ebs: STDERR: nginx: [emerg] bind() to 0.0.0.0:9090 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:8080 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:9090 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:8080 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:9090 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:8080 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:9090 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:8080 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:9090 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:8080 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] still could not bind()
    amazon-ebs: ---- End output of /sbin/service nginx restart ----
    amazon-ebs: Ran /sbin/service nginx restart returned 1
    amazon-ebs:
    amazon-ebs: Resource Declaration:
    amazon-ebs: ---------------------
    amazon-ebs: # In /tmp/packer-chef-solo/cookbooks-0/nginx/recipes/default.rb
    amazon-ebs:
    amazon-ebs: 24: service 'nginx' do
    amazon-ebs: 25:   supports :status => true, :restart => true, :reload => true
    amazon-ebs: 26:   action   :start
    amazon-ebs: 27: end
    amazon-ebs: 28:
    amazon-ebs:
    amazon-ebs: Compiled Resource:
    amazon-ebs: ------------------
    amazon-ebs: # Declared in /tmp/packer-chef-solo/cookbooks-0/nginx/recipes/default.rb:24:in `from_file'
    amazon-ebs:
    amazon-ebs: service("nginx") do
    amazon-ebs: action [:start]
    amazon-ebs: updated true
    amazon-ebs: supports {:status=>true, :restart=>true, :reload=>true}
    amazon-ebs: retries 0
    amazon-ebs: retry_delay 2
    amazon-ebs: default_guard_interpreter :default
    amazon-ebs: service_name "nginx"
    amazon-ebs: enabled true
    amazon-ebs: running true
    amazon-ebs: pattern "nginx"
    amazon-ebs: declared_type :service
    amazon-ebs: cookbook_name :nginx
    amazon-ebs: recipe_name "default"
    amazon-ebs: end
    amazon-ebs:
    amazon-ebs: Recipe: monit-ng::config
    amazon-ebs: * service[monit] action restart
    amazon-ebs: - restart service service[monit]
    amazon-ebs: * ruby_block[reload-monit] action run
    amazon-ebs: * service[monit] action reload
    amazon-ebs: - reload service service[monit]
    amazon-ebs: - execute the ruby block reload-monit
    amazon-ebs:
    amazon-ebs: Running handlers:
    amazon-ebs: [2015-01-05T17:07:08+00:00] ERROR: Running exception handlers
    amazon-ebs: Running handlers complete
    amazon-ebs: [2015-01-05T17:07:08+00:00] ERROR: Exception handlers complete
    amazon-ebs: [2015-01-05T17:07:08+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
    amazon-ebs: Chef Client failed. 85 resources updated in 459.135194302 seconds
    amazon-ebs: [2015-01-05T17:07:08+00:00] ERROR: service[nginx] (nginx::default line 24) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
    amazon-ebs: ---- Begin output of /sbin/service nginx restart ----
    amazon-ebs: STDOUT: Stopping nginx:
    amazon-ebs: Starting nginx:
    amazon-ebs: STDERR: nginx: [emerg] bind() to 0.0.0.0:9090 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:8080 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:9090 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:8080 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:9090 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:8080 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:9090 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:8080 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:9090 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] bind() to 0.0.0.0:8080 failed (98: Address already in use)
    amazon-ebs: nginx: [emerg] still could not bind()
    amazon-ebs: ---- End output of /sbin/service nginx restart ----
    amazon-ebs: Ran /sbin/service nginx restart returned 1
    amazon-ebs: [2015-01-05T17:07:08+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored: Error executing Chef: Non-zero exit status: 1

Does anyone experienced the same issue? How I can achieve an automate image generation otherwise using Jenkins?

[sethvargo]

Hi @NinjaTux

It appears that cloudbees is already running nginx or the base image you are using already has the port bound for some reason (perhaps cloudbees is using it for something else). Unfortunately we are not experts in Jenkins or Chef, so it is difficult to reproduce this issue. If you are able to identify a specific Packer issue, we would be happy to fix it :smile:.

You might also try asking this question on the mailing list to see if anyone from the Packer community has experienced the same issue.

[ninjatux]

Thanks @sethvargo, for now we solved this with a workaround :)