search

logsearch / logsearch-boshrelease

A BOSH-scalable Elasticsearch+Logstash+Kibana release
http://www.logsearch.io
Apache License 2.0
57 stars 46 forks source link

Parser job on v23 crashes on start (with rabbitmq input configured) #178

Closed axelaris closed 9 years ago

axelaris commented 9 years ago

Hi guys, I've just updated my deployment to v.23.0.0 (release) and it's parser job now fails to start. Here is output of parser.stderr.log:

LoadError: no such file to load -- ext/rabbitmq-client
       require at org/jruby/RubyKernel.java:1072
       require at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65
        (root) at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.11.0-java/lib/march_hare.rb:5
       require at org/jruby/RubyKernel.java:1072
       require at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65
        (root) at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-rabbitmq-1.1.0-java/lib/logstash/inputs/rabbitmq/march_hare.rb:1
      register at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-rabbitmq-1.1.0-java/lib/logstash/inputs/rabbitmq/march_hare.rb:6
          each at org/jruby/RubyArray.java:1613
  start_inputs at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:149
  start_inputs at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:148
   synchronize at org/jruby/ext/thread/Mutex.java:149
           run at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:81
           run at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:81
       execute at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/agent.rb:155
          call at org/jruby/RubyProc.java:271
           run at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/runner.rb:90
          call at org/jruby/RubyProc.java:271
           run at /var/vcap/packages/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/runner.rb:95

To reproduce this issue, I've tried to run Logstash manually, and realised that there are no such issue if I run Logstash w/out chpst: bin/logstash agent -f config/logstash.conf. When I try to run a full call: chpst -u vcap bin/logstash agent -f config/logstash.conf the issue comes. It seems it's related to permissions somehow.

mrdavidlaing commented 9 years ago

@axelaris - is it possible that one of the files in the logstash /var/vcap/packages/logstash/ package is not owned by the vcap user?

axelaris commented 9 years ago

Hi @mrdavidlaing, as I can see, all files under /var/vcap/packages/*/ are owned by root user:

root@node:/var/vcap/packages# ll logstash/
total 16
drwxr-xr-x 3 root root 4096 Sep 10 13:01 ./
drwxr-xr-x 3 root root 4096 Sep 10 14:17 ../
drwxr-xr-x 5 root root 4096 Sep 10 13:01 logstash/
-rw-r--r-- 1 root root 2438 Sep 10 13:02 plugins-default.list

I've recreated a job from scratch to make sure my experiment is clean. So these permissions is seems by design.

mrdavidlaing commented 9 years ago

@axelaris we made an explicit decision to run things as vcap a few releases ago as part of https://github.com/logsearch/logsearch-boshrelease/issues/116 - (see https://github.com/logsearch/logsearch-boshrelease/commit/dac0f77b0ee4f3b772636b841a54d36a1cdc1178) although as you can see at the time we were a little unsure of the consequences of doing this with logstash.

From a security perspective I'd prefer not to run logstash as root.

Can you help me figure out how to make things work with logstash running as vcap please.

Please could you test:

  1. Make everything under /var/vcap/packages/logstash world readable, ie: 

         chmod -R +r /var/vcap/packages/logstash

  2. Make everything under /var/vcap/packages/logstash owned by vcap, ie:

        chown -R vcap:vcap /var/vcap/packages/logstash

and let me know which resolves your issue.

If either of those work, we can patch the parser start script to ensure these permissions are in place before launching logstash.

axelaris commented 9 years ago

Thank you for suggestion, @mrdavidlaing. I found the root cause - logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.11.0-java/lib/ext/rabbitmq-client.jar had a 640 permission. Also described here: https://github.com/elastic/logstash/issues/3781

mrdavidlaing commented 9 years ago

@axelaris How do you suggest we proceed? Do we just need to wait for the next logstash release?

axelaris commented 9 years ago

I see two possible solutions here:

  1. Add chmod 644 vendor/bundle/jruby/1.9/gems/march_hare-2.11.0-java/lib/ext/rabbitmq-client.jar after this line: https://github.com/logsearch/logsearch-boshrelease/blob/develop/packages/logstash/packaging#L9
  2. Add bin/plugin update logstash-output-rabbitmq logstash-input-rabbitmq before (or after) this line: https://github.com/logsearch/logsearch-boshrelease/blob/develop/packages/logstash/packaging#L17

Anyway, these extra lines are temporary by it's nature and should be removed after upgrading to next logsearch release. And since we can't fix that issue in the current V23 release itself, it would be nice to mention it as known issues near the notice about upgrading to logstash 1.5.4

mrdavidlaing commented 9 years ago

@axelaris - could you submit a PR to implement 1 please

Add chmod 644 vendor/bundle/jruby/1.9/gems/march_hare-2.11.0-java/lib/ext/rabbitmq-client.jar after this line: https://github.com/logsearch/logsearch-boshrelease/blob/develop/packages/logstash/packaging#L9

I'll update the V23 release notes

axelaris commented 9 years ago

https://github.com/logsearch/logsearch-boshrelease/pull/180