elkstack does not work with chef-solo

[NickMRamirez]

Using Vagrant with chef-solo provisioner, I have elkstack in my Berksfile. When I call vagrant up, I get the following error. I shouldn't need a client.pem with chef-solo, should I? Only with chef server.

==> logstash_server: [2015-05-10T19:18:52+00:00] WARN: Failed to read the privat e key /etc/chef/client.pem: #<Errno::ENOENT: No such file or directory @ rb_syso pen - /etc/chef/client.pem>

==> logstash_server: ===========================================================

==> logstash_server: Recipe Compile Error in /tmp/vagrant-chef/00bb6c0612f789951 811ebc4d41b5940/cookbooks/elkstack/recipes/logstash.rb

==> logstash_server: ===========================================================

==> logstash_server: Chef::Exceptions::PrivateKeyMissing ==> logstash_server: ----------------------------------- ==> logstash_server: I cannot read /etc/chef/client.pem, which you told me to us e to sign requests! ==> logstash_server: Cookbook Trace: ==> logstash_server: --------------- ==> logstash_server: /tmp/vagrant-chef/00bb6c0612f789951811ebc4d41b5940/cookbo oks/elasticsearch/libraries/search_for_nodes.rb:44:in find_matching_nodes' ==> logstash_server: /tmp/vagrant-chef/00bb6c0612f789951811ebc4d41b5940/cookbo oks/elasticsearch/libraries/search_for_nodes.rb:34:insearch_for_nodes' ==> logstash_server: /tmp/vagrant-chef/00bb6c0612f789951811ebc4d41b5940/cookbo oks/elasticsearch/recipes/search_discovery.rb:21:in from_file' ==> logstash_server: /tmp/vagrant-chef/00bb6c0612f789951811ebc4d41b5940/cookbo oks/elkstack/recipes/acl.rb:17:infrom_file' ==> logstash_server: /tmp/vagrant-chef/00bb6c0612f789951811ebc4d41b5940/cookbo oks/elkstack/recipes/_server.rb:57:in from_file' ==> logstash_server: /tmp/vagrant-chef/00bb6c0612f789951811ebc4d41b5940/cookbo oks/elkstack/recipes/logstash.rb:10:infrom_file' ==> logstash_server: Relevant File Content: ==> logstash_server: ---------------------- ==> logstash_server: /tmp/vagrant-chef/00bb6c0612f789951811ebc4d41b5940/cookbook s/elasticsearch/libraries/search_for_nodes.rb: ==> logstash_server: 37: end.sort ==> logstash_server: 38: end

Vagrantfile:

Vagrant.configure(2) do |config|
  config.vm.define 'logstash_server' do |server|
    server.vm.box = 'larryli/utopic64'
    server.vm.provision 'chef_solo' do |chef|
      chef.add_recipe 'elkstack::logstash'
    end
  end
end

[brint]

Hi @NickMRamirez!

Based on the traceback, the recipe is including the private recipe _server:

https://github.com/rackspace-cookbooks/elkstack/blob/master/recipes/logstash.rb#L10

And at the bottom of _server, it's hitting this block:

# if iptables toggle is set, include host based firewall rules
iptables_enabled = node.deep_fetch('elkstack', 'config', 'iptables')
if !iptables_enabled.nil? && iptables_enabled
  include_recipe 'elkstack::acl'
end

If this attribute is flipped to false, it will bypass the elkstack::acl recipe, which is calling elasticsearch::search_discovery, that uses this library, which leverages Chef Search. This explains the error.

Adding the following to your provision block should fix it by bypassing the acl recipe:

    chef.json = {
      "elkstack" => {
        "config" => {
          "iptables" => false
        }
      }
    }

[NickMRamirez]

@brint That seems to fix the logstash recipe, which is good. I then discovered that the elasticsearch recipe also relies on Chef Search and so cannot be used with chef-solo. I am looking through that cookbook to find a way to turn that feature off, but haven't found a way yet.

[NickMRamirez]

I wonder if this second line in elkstack/recipes/elasticsearch.rb should be conditional?

node.override['elasticsearch']['discovery']['zen']['ping']['multicast']['enabled'] = false include_recipe 'elasticsearch::search_discovery'

[martinb3]

@NickMRamirez, I've added some guards/checks around various things, to make it more chef-solo friendly, as part of #146. I'm releasing a new version now -- could you try that and let us know if things are a bit smoother with the new version? Thanks!

[patcon]

haha wow. was just spinning this up for the first time today. How timely. Thanks guys!

[patcon]

Any thoughts on whether advising the search cookbooks or chef_zero usage would be a better approach than baking in checks?

https://github.com/edelight/chef-solo-search https://github.com/opscode-cookbooks/partial_search (oops. irrelevant) https://docs.vagrantup.com/v2/provisioning/chef_zero.html

(Will be investigating in the meantime.)

[NickMRamirez]

With the new changes the logstash recipe works now. I was getting a timeout when running it--at the point where it tries to restart the logstash service. But then I added the java cookbook and it seems to have fixed it. I will check out the other recipes too.

Test project: https://github.com/NickMRamirez/ElkstackProject

UPDATE: I checked it with the default recipe and everything works fine now. Thanks!

[patcon]

ok cool, but fyi chef_solo is essentially deprecated and chef_zero is the new future, so might be worth migrating (just took a sed -i'' 's/solo/zero/g' Vagrantfile, to be honest :). After that, it all worked fine for me with just the iptables=false