ELK stack container crashes consistently in Docker-Compose

[iMerica]

Hi, I'm trying to run the entire ELK stack as per the directions and I'm consistently seeing two crashes. One relating to "Faraday" not connecting and another relating to "ThreadError: current thread not owner".

Here's my docker-compose.yml:

elk:
  image: pblittle/docker-logstash
  ports: 
    - "9292:9292"
    - "9201:9200"

and here is the output I'm seeing when running docker-compose up:

elk_1 | + logstash_start_agent
elk_1 | + local binary=/opt/logstash/bin/logstash
elk_1 | + local 'config_path=/opt/logstash/conf.d/**/*.conf'
elk_1 | + local log_file=/var/log/logstash/logstash.log
elk_1 | + case "$1" in
elk_1 | + exec /opt/logstash/bin/logstash agent --config '/opt/logstash/conf.d/**/*.conf' --log /var/log/logstash/logstash.log -- web
elk_1 | Sending logstash logs to /var/log/logstash/logstash.log.
elk_1 | ThreadError: current thread not owner
elk_1 |   mon_check_owner at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/monitor.rb:246
elk_1 |          mon_exit at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/monitor.rb:195
elk_1 |           require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:143
elk_1 |           require at /opt/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.4/lib/polyglot.rb:65
elk_1 |            (root) at /opt/logstash/lib/logstash/kibana.rb:6
elk_1 |           require at org/jruby/RubyKernel.java:1085
elk_1 |           require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
elk_1 |           require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:53
elk_1 |               run at /opt/logstash/lib/logstash/runner.rb:123
elk_1 |              call at org/jruby/RubyProc.java:271
elk_1 |               run at /opt/logstash/lib/logstash/runner.rb:175
elk_1 |              main at /opt/logstash/lib/logstash/runner.rb:92
elk_1 |            (root) at /opt/logstash/lib/logstash/runner.rb:215

Environment:

❯ docker --version && docker-compose --version                                                                                    local_dev_plus/git/master !
Docker version 1.6.2, build 7c8fca2
docker-compose 1.2.0

Thanks,

Michael

[pblittle]

@iMerica thanks for creating the issue. I haven't seen this problem. Do you mind running the elasticsearch-embedded test suite? It should mirror what you're doing fairly closely.

$ cd ./1.4/test
$ make test-elasticsearch-embedded

That will bring up the container for testing. You then need to docker exec to get inside of it.

$ docker exec -it elasticsearchembedded_logstash_1 /bin/bash

Then fire up the bats test suite:

$ ./test.bats
 ✓ Logstash is running
 ✓ Logstash.config contains '{ elasticsearch { embedded => true }'
 ✓ Elasticsearch.yml contains 'script.disable_dynamic: true'
 ✓ Kibana's elasticsearch server is 'http://+window.location.hostname+:9200'
 ✓ Elasticsearch is listening on port '9200'
 ✓ Elasticsearch is listening on port '9300'
 ✓ Kibana is listening on port '9292'
 ✓ Elasticsearch is reachable at '/_status'
 ✓ Kibana dashboard reachable at '/index.html'

9 tests, 0 failures

Please let me know if you don't have time to debug. I should be able to take a look at it tomorrow.

[iMerica]

Okay, I'm doing this now. Interestingly, this issue appears to only surface in Docker-Compose, not when when starting the container using docker run.

[iMerica]

A couple of issues with testing:

• make test-elasticsearch-embedded doesn't detach after running.
• elasticsearchembedded_logstash_1 as a name doesn't exist as a result of running the above make command. I had to ssh into the container by it's ID.
• After ssh'ing into the container, ./test.bats is not there. Here's what I see in ls: bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var

[pblittle]

@iMerica are you testing against the master branch?

make test-elasticsearch-embedded doesn't detach after running.

You're correct about the container not detaching. I should fix that. :)

elasticsearchembedded_logstash_1 as a name doesn't exist as a result of running the above make command. I had to ssh into the container by it's ID.

I definitely see the elasticsearchembedded_logstash_1 name though:

$ docker ps -a
CONTAINER ID        IMAGE                                   COMMAND                CREATED             STATUS              PORTS                                                                    NAMES
ac7cedc54a97        elasticsearchembedded_logstash:latest   "/app/bin/boot /bin/   6 minutes ago       Up 6 minutes        0.0.0.0:9292->9292/tcp, 0.0.0.0:9300->9300/tcp, 0.0.0.0:9201->9200/tcp   elasticsearchembedded_logstash_1

After ssh'ing into the container, ./test.bats is not there. Here's what I see in ls: bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var

I also see test.bats when I exec into the container:

$ docker exec -it elasticsearchembedded_logstash_1 /bin/bash
root@ac7cedc54a97:/app# ls -al
total 44
drwxr-xr-x  3 root root 4096 Jun  8 02:36 .
drwxr-xr-x 60 root root 4096 Jun  8 02:36 ..
-rw-r--r--  1 root root  797 May 14 14:14 Dockerfile
drwxr-xr-x  2 root root 4096 May 14 14:32 bin
-rwxr-xr-x  1 root root 1452 May 14 14:14 elasticsearch.sh
-rw-r--r--  1 root root   33 Jun  8 02:36 elasticsearch.yml
-rwxr-xr-x  1 root root 1150 May 14 14:14 kibana.sh
-rw-r--r--  1 root root 1025 May 14 14:14 logstash-forwarder.sh
-rwxr-xr-x  1 root root 4365 May 14 14:32 logstash.sh
-rw-r--r--  1 root root    0 May 14 14:20 redis.sh
-rwxr-xr-x  1 root root 1225 May 14 14:14 test.bats

And my environment:

$ docker --version && docker-compose --version
Docker version 1.6.2, build 7c8fca2
docker-compose 1.2.0

I would say the first step is to figure out why we aren't able to create similar test environments.

[pblittle]

Closing due to inactivity. @iMerica please reopen if I can do anything to help.