chrisroberts / vagabond

Advocating idleness and work-shyness
Other
233 stars 25 forks source link

Upstart daemon watcher stop working after instalation dev utils on lxc Ubuntu 12.04LTS #35

Closed alff closed 11 years ago

alff commented 11 years ago

Hi Chris! I ran into the problem with Vagabond instalation on Ubuntu VM. How to reproduce my error: -create by kvm VM Ubuntu 12.04 LTS -install and setup Vagabond chef-server -up the new vagabond node (with same os: Ubuntu 12.04LTS) with empty run_list and ssh on it -according with this page (https://github.com/BrightcoveOS/Diamond/wiki/Installation) do this step: * apt-get install make pbuilder python-mock python-configobj python-support cdbs

After that if i'm trying to install diamond package (any version and/or builded on this node or on any other) this new service doesn't recognize as uptstart service (service: Unknown job: diamond). But if i build this package remotely and i install on Vagabond container only require packages (python-configobj and python-support) diamond package installs and works correctly. Any suggestions?

chrisroberts commented 11 years ago

Hi @alff. The issue you are running into is with upstart. It uses inotify on /etc/init to reload the configuration files when they change. The overlayfs does not properly support inotify so upstart will fail to see the new conf file you dropped in. I'm actually starting to mess around with having vagabond provide alternatives via the Vagabondfile but it's still a little ways out.

Since you are running on precise, there's a good chance that aufs is still hanging around. You can attempt to use that instead, and if it's there, upstart will work as expected. Your Vagabondfile would look like this:

{
  :nodes => {
    :my_node => {
      :template => 'ubuntu_1204',
      :run_list => [],
      :union => 'aufs'
    }
  }
}
alff commented 11 years ago

Thanks for reply, Chris. I added these lines to my Vagabond config file and now i see that chef unable to create templates. The first one fails with '403' Exception.

.... Error executing action create on resource 'template[/etc/default/sysstat]' Net::HTTPServerException 403 "Forbidden" Resource Declaration: In /var/chef/cache/cookbooks/sysstat/recipes/default.rb .....

my vagabond file (without controller stuff) is: { :nodes => { :ops => { :template => "ubuntu_1204", :environment => "vagabond", :ipaddress => "10.0.3.11", :union => "aufs", :run_list => [ "role[vagabond_ops]" ] } }, :clusters => { :simple => [ "ops" ] }, :local_chef_server => { :zero => false, :berkshelf => true, :librarian => false, :enabled => true, :auto_upload => true }, :sudo => true }

chrisroberts commented 11 years ago

@alff Give it a run again and it should pick it up okay. Is the chef run long? Looks like some custom configs aren't making it over from the server bootstrap, so I'll have a look and see that it's getting it correctly.

alff commented 11 years ago

Yeah, it looks like a bug as described here: http://tickets.opscode.com/browse/CHEF-4253

alff commented 11 years ago

I've recreated vagabond chef server with nodes and applied the patch with extended timeout for er_chef (as workaround). After that templates were created correctly.

alff commented 11 years ago

Hi Chris, not sure how's it corresponding with current issue but now chef_run can exit in any time with same ERR output: ... STDERR: ---- End output of sudo knife bootstrap 10.0.3.11 -d chef-full -N ops -i /opt/hw-lxc-config/id_rsa --no-host-key-verify --run-list "role[vagabond_ops]" -E vagabond --server-url https://10.0.3.127 ---- ...

in chef-stacktrace.out i see: Errno::EIO: Input/output error -

I guess it happens that host VM has insufficient resources, but before i haven't seen this error.

chrisroberts commented 11 years ago

Can you paste in the chef-stacktrace.out

alff commented 11 years ago

And at last. I've recived this error again. :( I connected to the ops node and i saw that couple of seconds chef-client procsess was in proccess list.

256 ? Ssl 0:12 /opt/chef/embedded/bin/ruby /usr/bin/chef-client -j /etc/chef/first-boot.json -E vagabond 552 ? Zs 0:00 [chef-client]

but then disappeared.

chef-stacktrace.out

enerated at 2013-07-17 23:41:18 +0000 Errno::EIO: Input/output error - /opt/chef/embedded/lib/ruby/1.9.1/logger.rb:574:in write' /opt/chef/embedded/lib/ruby/1.9.1/logger.rb:574:inwarn' /opt/chef/embedded/lib/ruby/1.9.1/logger.rb:574:in rescue in write' /opt/chef/embedded/lib/ruby/1.9.1/logger.rb:558:inwrite' /opt/chef/embedded/lib/ruby/1.9.1/logger.rb:377:in add' /opt/chef/embedded/lib/ruby/1.9.1/logger.rb:452:inerror' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-log-1.6.0/lib/mixlib/log.rb:113:in block in error' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-log-1.6.0/lib/mixlib/log.rb:113:ineach' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-log-1.6.0/lib/mixlib/log.rb:113:in error' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:116:inrun_exception_handlers' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:127:in block in <class:Handler>' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:120:incall' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:120:in block in run_failed' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:119:ineach' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:119:in run_failed' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:485:inrescue in do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:494:in do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:200:inrun' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:190:in run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:297:inblock in run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:290:in loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:290:inrun_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:73:in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/bin/chef-client:26:in<top (required)>' /usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in

'

alff commented 11 years ago

Hi, yesterday i've figured out in this problem. It happens when chef run goes so slow. In provision.rb we have timeout interval '2000' seconds. When i'm extending this it works fine. Now i'm thinking about how to parameterize this attribute.