Comments on ELK-stack post

[cteyton]

Hi, Thanks for this nice post.

When you say "As Docker has started to support direct syslog logging the need for Logspout may not be there any more. I need to try it out and will update the blog based on the findings."

Can we imagine the scenario where running container with "docker run log-driver=syslog" is enough to redirect containers logs to the logstash container ? In that case, does it mean that we have to modifiy something on the host machine (the rsyslog configuration ?) to make this redirection concrete ? If my suggestion is correct, this will create a drawback as it will make the deployment a bit more intrusive on the host machine.

How do you imagine this context ?

[cteyton]

I just discovered this syslog driver can be configured : https://docs.docker.com/reference/logging/overview/

This means we need to run a container with "docker run log-driver=syslog syslog-address=tcp://HOST_OF_LOGSTASH:PORT_OF_LOGSTASH".

[SirIle]

As I try to keep things as agnostic as possible I want to limit the need to touch the underlying OS for the Docker node. I was under the impression that I needed to configure rsyslog to get the logs to an external server, but if things are as you say and you can directly get the logs over tcp or udp to the remote logstash server then this sounds viable. Good one!

In that article the syntax seems to be "docker run --log-driver=syslog --log-opt syslog-address=tcp://HOST:PORT" so you need that --log-opt there. I'll give it a spin.

[cteyton]

Great, thank you. As the logging driver "fluentd" (a solution similar to Logstash) was integrated in Docker V1.8.0, maybe we can fairly assume that a logstash driver will appear soon.

[Mojo-Lee]

I tried to run the ELK and logspout on Docker in my local with Vagrant, the http://localhost:9200 works fine which means Elasticsearch is up and returns the result as following: { "status" : 200, "name" : "Scarlet Scarab", "cluster_name" : "elasticsearch", "version" : { "number" : "1.7.0", "build_hash" : "929b9739cae115e73c346cb5f9a6f24ba735a743", "build_timestamp" : "2015-07-16T14:31:07Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }

But http://localhost:5601 was not working well and the Kibanabox log always show: {"name":"Kibana","hostname":"kibanabox","pid":1,"level":40,"msg":"Unable to revi ve connection: http://localhost:9200/","time":"2015-11-10T13:35:09.343Z","v":0} {"name":"Kibana","hostname":"kibanabox","pid":1,"level":40,"msg":"No living conn ections","time":"2015-11-10T13:35:09.344Z","v":0} {"name":"Kibana","hostname":"kibanabox","pid":1,"level":30,"msg":"Unable to conn ect to elasticsearch at localhost:9200. Retrying in 2.5 seconds.","time":"2015-1 1-10T13:35:09.344Z","v":0}

I tried to set verify_ssl to false, it was not working. Please help advice, Thanks!

[SirIle]

@Mojo-Lee Hi, I usually give the Docker node an own IP with Vagrant and access the services from there instead of localhost. My guess is that Kibanabox is trying to access ElasticSearch from the localhost that it knows which is the container itself and is unable to connect to it.

[Mojo-Lee]

@SirIle Exactly, that's the root cause I've already found, thank you very much anyway!

[barbarello]

Hi @SirIle , Terrific post. I tried to replicate your set up in a multi host AWS environment rather than boot2docker.

1. logbox = ELK stack + logspout; by passing SERVICE_9200_NAME and SERVICE_5000_NAME , 'elasticstack' and 'logstash' are represented in Consul by the individual serviecenames rather than 'minilogbox_9200' or 'minilogbox_5000'.
2. 3 additional nodes , running a 3-member consul server cluster and logspout on each of the server. consul containers all run with -p 172.17.0.1:53:53/udp for DNS , and logspout is run --dns 172.17.0.1 --dns-search service.consul http://logstash:5000 options.
   I can ping 172.17.0.1 from within logspout container, but can not seem to resolve logstash hostname. I then attached to a running busy box created with the same dns options as logspout: ifconfig eth0 Link encap:Ethernet HWaddr 02:42:AC:11:00:05
   inet addr:172.17.0.5 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr: fe80::42:acff:fe11:5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:648 (648.0 B) TX bytes:648 (648.0 B)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

/ # nslookup logstash Server: 172.17.0.1 Address 1: 172.17.0.1

nslookup: can't resolve 'logstash' / # nslookup logstash.service.consul Server: 172.17.0.1 Address 1: 172.17.0.1

nslookup: can't resolve 'logstash.service.consul' / # nslookup consul.service.consul Server: 172.17.0.1 Address 1: 172.17.0.1

nslookup: can't resolve 'consul.service.consul' / # nslookup consul Server: 172.17.0.1 Address 1: 172.17.0.1

nslookup: can't resolve 'consul'

Any clues ??? Thanks.