Relating Logstash with other services for log collection

[aalvrz]

I am trying to use the Logstash Charm to test some log collection/forwarding for specific services.

However I can't seem to be able to connect services to Logstash. For example, when I try:

juju add-relation logstash wordpress

I get this error message:

ERROR no relations found

I had read somewhere that this Logstash charm is a subordinate charm/service. From what I understand, this means that it uses implicit relations (relations not defined in metadata.yaml by charm authors) to connect the services.

In my deployment, Zulu8 has already been deployed and connected with Logstash.

So I am at a loss on how exactly log collection for specific services is supposed to work. Some help and guidance would be greatly appreciated.

My juju version: 1.25.3-trusty-amd64

[lazypower]

Greetings @BigChief45

Thanks for taking a look at the logstash layer. What you're encountering is probably some misleading information somewhere, or blatanly missing information. I'll take a todo from this bug to update the docs across the bundles/charms to illustrate the proposed BELK stack.

What you're going to need to do is deploy one of the beats charms to your model, and relate that to logstash.

eg:

juju deploy wordpress
juju deploy logstash
juju deploy filebeat
juju add-relation wordpress filebeat:beats-host
juju add-relation filebeat logstash

The filebeat agent has a pretty broad scope of log file collection under /var/log/* and if you need additional paths to be monitored for data, simply adjust the filebeat configuration.

Additionally you will need to have elasticsearch and kibana related if you wish to store/warehouse the data and then visualize it.

[aalvrz]

@chuckbutler Thank you for the quick reply.

Indeed the information is a bit scattered and misleading. I began playing with logstash-agent, logstash-indexer, kibana, and elasticsearch in this way:

juju deploy --repository=logstash local:precise/elasticsearch
juju add-unit elasticsearch
juju deploy --repository=logstash local:precise/logstash-indexer
juju add-relation elasticsearch:cluster logstash-indexer
juju deploy --repository=logstash local:precise/kibana
juju add-relation elasticsearch:rest kibana
juju expose kibana
juju deploy --repository=logstash local:precise/logstash-agent
juju add-relation logstash-agent logstash-indexer:input

Everything seems to relate fine and when I try something like juju add-relation wordpress logstash-agent, the relation is actually created. The agent seems to be configured to get logs from /var/log/syslog by default, but it doesn't seem to be really collecting them. At least, I don't see anything in the Kibana dashboard. Then again it could be a Redis problem, but the documentation doesn't say anything about deploying Redis...

Additionally there is also the ELK bundle (which doesn't use Filebeat?). However when trying to deploy it with juju deploy ~containers/bundle/elk-stack, I get the following error:

ERROR expected a charm URL, got bundle URL "cs:~containers/bundle/elk-stack-0"

I am not sure if the error above is related to my Juju version.

In conclusion, I am just not sure which approach is the correct approach. I will try your suggestion and report back.

[aalvrz]

So I tried the following as per using the Filebeat charm approach:

juju deploy wordpress
juju deploy elasticsearch
juju deploy kibana

juju add-relation elasticsearch kibana

juju deploy logstash
juju deploy zulu8
juju add-relation logstash zulu8
juju add-relation logstash elasticsearch

juju deploy filebeat
juju add-relation wordpress filebeat:beats-host
juju add-relation logstash filebeat

Nothing seems to appear in the Kibana dashboard. And I get the following error in Kibana:

Index Patterns: Please specify a default index pattern

I am trying to use the logstash-* which I think is the default one, if logstash is forwarding the logs correctly.

Also I tried logging a new log to syslog using logger from inside the Wordpress unit, but nothing as well.

Am I still missing something?

[lazypower]

The index pattern would be filebeat-*

the beats all ship and store their information under the beats pattern hash in elastic search.

eg:

juju deploy topbeat
juju add-relation topbeat elasticsearch

You will see it posts a topbeat-{DATE} index, and starts aggregating data under that key. If that's not the behavior you're seeing I'll need to dig in deeper to see if I can root out the cause. I apologize for the difficulty in getting started i've been swamped with the kubernetes release but should be able to circle back to this in the next week.

[aalvrz]

I tried using filebeat-* as the index pattern in Kibana, but I get the following message:

Unable to fetch mapping. Do you have indices matching the pattern?

Also I tried relating Filebeat to Elasticsearch similar to how you did it with Topbeat, but still the same result.