Logstash crashes in Docker during startup when trying to initialize plugins/filters

[runninbutt]

After changes to logstash geoip plugin in >7.14, I can't seem to get logstash up in Docker with previous configuration. When I have geoip filter mentioned in my configuration (.conf file), the container will crash few seconds after startup.

Logstash version: 7.14 < Runtime: Docker from elastics docker repo

Error in logstash log:

[2021-10-07T13:15:17,363][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Errno::EACCES: Permission denied - /usr/share/logstash/data/plugins/filters>, :backtrace=>["org/jruby/RubyDir.java:615:in `mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:235:in `fu_mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:213:in `block in mkdir_p'", "org/jruby/RubyArray.java:1902:in `reverse_each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:211:in `block in mkdir_p'", "org/jruby/RubyArray.java:1820:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:196:in `mkdir_p'", "/usr/share/logstash/x-pack/lib/filters/geoip/database_manager.rb:72:in `prepare_cc_db'", "/usr/share/logstash/x-pack/lib/filters/geoip/database_manager.rb:46:in `setup'", "/usr/share/logstash/x-pack/lib/filters/geoip/database_manager.rb:219:in `block in trigger_download'", "org/jruby/ext/thread/Mutex.java:164:in `synchronize'", "/usr/share/logstash/x-pack/lib/filters/geoip/database_manager.rb:217:in `trigger_download'", "/usr/share/logstash/x-pack/lib/filters/geoip/database_manager.rb:276:in `subscribe_database_path'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.2.2-java/lib/logstash/filters/geoip.rb:181:in `select_database_path'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.2.2-java/lib/logstash/filters/geoip.rb:109:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1820:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:586:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/example.conf"], :thread=>"#<Thread:0x414de683 run>"}

Troubleshooting

I can confirm that this worked without any problems in 7.13. It seems to crash due to permission issues while trying to create a directory. I tested to shell into the container with docker exec and tried to create the mentioned folder as the logstash container's user and it fails to do so.

If I add the mentioned directory as an empty folder to logstash in docker-compose.yaml under volumes, i.e.:

- ./test:/usr/share/logstash/data/plugins/filters

then it works.

Find all the rest of the details below.

example.conf: If i comment out the geoip part, it won't crash when starting the container.

filter {
...
    geoip {
      source => "ClientIP"
    }
}

docker-compose.yaml logstash:

  logstash:
    image: docker.elastic.co/logstash/logstash:${ELK_VERSION}
    container_name: logstash
    ports:
      - 127.0.0.1:5044:5044
      - 127.0.0.1:5000:5000/tcp
      - 127.0.0.1:5000:5000/udp
      - 127.0.0.1:9600:9600
    environment:
      LS_JAVA_OPTS: "-Xmx2056m -Xms2056m"
    volumes:
      - ./mounts/logstash:/etc/logstash/conf.d
      - ./mounts/logstash/logstash.yaml:/usr/share/logstash/config/logstash.yml:ro

logstash.yaml:

http.host: "0.0.0.0"
path.config: /etc/logstash/conf.d/*.conf
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]

[kaisecheng]

@runninbutt Thank you for reporting the issue. Can you tell me what is your OS version? uname -a I cannot reproduce the problem in macOS docker, logstash 7.14.0

[runninbutt]

I've tried this on both Docker Desktop on Mac and Windows. Macos machine: uname -a:

Darwin xxxx.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64