search

rackspace-cookbooks / platformstack

Rackspace Base Stack via Chef
Other
4 stars 6 forks source link

add logstash agent last #168

Closed lmunro closed 10 years ago

lmunro commented 10 years ago

Adding platformstack to a default recipe causes problems with elkstack as the logstash agent is already installed, and then conflicts with logstash server due to syslog port conflict.

We need to be able to include the logstash agent recipe in wrapper cookbooks, so it needs to run last in platformstack.

prometheanfire commented 10 years ago

which should go last? logstash or iptables (I'm thinking iptables)

martinb3 commented 10 years ago

I deliberately took the logging recipe out of running that way, since there wasn't any major need for it to be run last. Can you point us at what the change is intended to fix?

martinb3 commented 10 years ago

Sounds like this was a result of a search finding the wrong node and installing the agent. We're going to add an enhancement for the other problem, which appears to be single not being discoverable.

prometheanfire commented 10 years ago

delete branch?

lmunro commented 10 years ago

So the search in platformstack::logging doesn't install the agent on the nodes it finds, it just uses the nodes for the targets in the config file.

It will install the agent on all nodes, and since platform stack should and is included in the default/base recipe for customers, platformstack::logging runs before elkstack:logstash is applied to the node, so the unless in this line will always return true: https://github.com/rackspace-cookbooks/platformstack/blob/master/recipes/logging.rb#L32

We can get around it by not having a default recipe run first in the customer repo, and setting logging_enabled = true on all role recipes except elkstack, however we shouldn't have to do that as it's a messy workaround.

martinb3 commented 10 years ago

platformstack::logging doesn't install the agent on the nodes it finds Just for clarification, platformstack::logging doesn't install anything -- it only makes calls into elkstack if it finds any elk stack server nodes.

It will install the agent on all nodes, and since platform stack should and is included in the default/base recipe for customers, platformstack::logging runs before elkstack:logstash is applied to the node, so the unless in this line will always return true

Actually, it does a search first, and will not do anything unless an ELK stack cluster nodes are already found. So unless someone explicitly puts elkstack::cluster or elkstack::single on the runlist, it will simply exit until ELK nodes are deployed somewhere else: https://github.com/rackspace-cookbooks/platformstack/blob/master/recipes/logging.rb#L24

Could we get a full description of the problem? I feel like we're going round and round describing what different recipes, but I'd like to be able to reproduce it explicitly, with a test-kitchen case. If it's breaking customer deployments, I'd like to really chase it down and fix it once & for all.