search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.11k stars 3.48k forks source link

Regression: Run as user nobody:nogroup was broken in logstash:7.17.8 (logstash:7.17.7 works as expected) #14836

Open lioncubs opened 1 year ago

lioncubs commented 1 year ago

Logstash information:

  1. docker.elastic.co/logstash/logstash:7.17.8
  2. Run directly from the default docker containers
  3. Run in Kubernetes with appropriate config - can reproduce with no config as well
root@kind:~# docker run -u nobody:nogroup -v $PWD/data:/usr/share/logstash/data --rm -ti docker.elastic.co/logstash/logstash:7.17.8 bash
nobody@5255f955baef:/usr/share/logstash$ bin/logstash --version
Using bundled JDK: /usr/share/logstash/jdk
logstash 7.17.8

root@kind:~# docker run -u nobody:nogroup -v $PWD/data:/usr/share/logstash/data --rm -ti docker.elastic.co/logstash/logstash:7.17.7 bash
nobody@4f6e2456591f:/usr/share/logstash$ bin/logstash --version
Using bundled JDK: /usr/share/logstash/jdk
logstash 7.17.7

Plugins installed: (bin/logstash-plugin list --verbose)

AS in from docker image:

root@kind:~# docker run -u nobody:nogroup -v $PWD/data:/usr/share/logstash/data --rm -ti docker.elastic.co/logstash/logstash:7.17.7 bash
nobody@d919a69b950a:/usr/share/logstash$ bin/logstash-plugin list --verbose
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-codec-avro (3.3.1)
logstash-codec-cef (6.2.5)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.1)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.1)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.2.2)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.6)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.1.5)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.11.1)
logstash-filter-fingerprint (3.3.2)
logstash-filter-geoip (7.2.12)
logstash-filter-grok (4.4.2)
logstash-filter-http (1.2.1)
logstash-filter-json (3.2.0)
logstash-filter-kv (4.5.0)
logstash-filter-memcached (1.1.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.6)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.1.1)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.3.1)
logstash-filter-truncate (1.0.5)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.3)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.1.3)
logstash-input-azure_event_hubs (1.4.4)
logstash-input-beats (6.2.6)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (1.1.12)
logstash-input-elasticsearch (4.12.3)
logstash-input-exec (3.4.0)
logstash-input-file (4.4.4)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.4.5)
logstash-input-http_poller (5.1.0)
logstash-input-imap (3.2.0)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-s3 (3.8.3)
logstash-input-snmp (1.3.1)
logstash-input-snmptrap (3.1.0)
logstash-input-sqs (3.1.3)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.6.0)
logstash-input-tcp (6.2.7)
logstash-input-twitter (4.1.0)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.2)
logstash-integration-elastic_enterprise_search (2.1.2)
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.2.6)
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka (10.9.0)
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-rabbitmq (7.3.1)
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-cloudwatch (3.0.10)
logstash-output-csv (3.0.8)
logstash-output-elasticsearch (11.4.1)
logstash-output-email (4.1.1)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.2.5)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-s3 (4.3.7)
logstash-output-sns (4.0.8)
logstash-output-sqs (6.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.0.2)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.0.6)
logstash-patterns-core (4.3.4)

root@kind:~# docker run -u nobody:nogroup -v $PWD/data:/usr/share/logstash/data --rm -ti docker.elastic.co/logstash/logstash:7.17.8 bash
nobody@1db7966df365:/usr/share/logstash$ bin/logstash-plugin list --verbose
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-codec-avro (3.3.1)
logstash-codec-cef (6.2.6)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.1)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.1)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.2.2)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.6)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.1.5)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.11.1)
logstash-filter-fingerprint (3.3.2)
logstash-filter-geoip (7.2.12)
logstash-filter-grok (4.4.3)
logstash-filter-http (1.2.1)
logstash-filter-json (3.2.0)
logstash-filter-kv (4.5.0)
logstash-filter-memcached (1.1.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.6)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.1.1)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.3.1)
logstash-filter-truncate (1.0.5)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.3)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.1.3)
logstash-input-azure_event_hubs (1.4.4)
logstash-input-beats (6.2.6)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (1.1.12)
logstash-input-elasticsearch (4.12.3)
logstash-input-exec (3.4.0)
logstash-input-file (4.4.4)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.4.5)
logstash-input-http_poller (5.1.0)
logstash-input-imap (3.2.0)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-s3 (3.8.3)
logstash-input-snmp (1.3.1)
logstash-input-snmptrap (3.1.0)
logstash-input-sqs (3.1.3)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.6.0)
logstash-input-tcp (6.2.7)
logstash-input-twitter (4.1.0)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.2)
logstash-integration-elastic_enterprise_search (2.1.2)
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.2.6)
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka (10.9.0)
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-rabbitmq (7.3.1)
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-cloudwatch (3.0.10)
logstash-output-csv (3.0.8)
logstash-output-elasticsearch (11.4.1)
logstash-output-email (4.1.1)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.2.5)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-s3 (4.3.7)
logstash-output-sns (4.0.8)
logstash-output-sqs (6.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.0.3)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.0.6)
logstash-patterns-core (4.3.4)

JVM (e.g. java -version):

OS version (uname -a if on a Unix-like system):

Any Linux version - tested on the following

root@kind:~# uname -a
Linux kind 5.15.0-33-generic #34~20.04.1-Ubuntu SMP Thu May 19 15:51:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Success with 7.17.7

  1. create an empty 'data' folder in your filesystem.
  2. chown nobody data
  3. chmod -R ugo+rw data
  4. docker run -u nobody:nogroup -v $PWD/data:/usr/share/logstash/data --rm -ti --entrypoint bash docker.elastic.co/logstash/logstash:7.17.7
  5. logstash -e 'input { stdin { } } output { stdout {} }'

Failure with 7.17.8

  1. create an empty 'data' folder in your filesystem.
  2. chown nobody data
  3. chmod -R ugo+rw data
  4. docker run -u nobody:nogroup -v $PWD/data:/usr/share/logstash/data --rm -ti --entrypoint bash docker.elastic.co/logstash/logstash:7.17.8
  5. logstash -e 'input { stdin { } } output { stdout {} }'
  6. Observe Failure

7.17.8 Failure - using nobody:nogroup

# 7.17.8 Failure - using nobody:nogroup
mkdir data.7.17.8
chown nobody data.7.17.8
chmod -R ugo+rw data.7.17.8
docker run -u nobody:nogroup -v $PWD/data.7.17.8:/usr/share/logstash/data --rm -ti  --entrypoint bash docker.elastic.co/logstash/logstash:7.17.8

nobody@0a35bff2d197:/usr/share/logstash$ logstash -e 'input { stdin { } } output { stdout {} }'
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[FATAL] 2023-01-10 18:30:31.009 [main] Logstash - Logstash stopped processing because of an error: (LoadError) no such file to load -- logstash/build
org.jruby.exceptions.LoadError: (LoadError) no such file to load -- logstash/build
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/api/commands/system/basicinfo_command.rb:20) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/api/command_factory.rb:19) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/api/modules/base.rb:19) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:20) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:18) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:23) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:44) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:92) ~[?:?]
nobody@0a35bff2d197:/usr/share/logstash$ exit
exit
root@kind:~#

Success with 7.17.7 - using nobody:nogroup - CTRL-C to stop

# Success with 7.17.7 - using nobody:nogroup - CTRL-C to stop
mkdir data.7.17.7
chown nobody data.7.17.7
chmod -R ugo+rw data.7.17.7
docker run -u nobody:nogroup -v $PWD/data.7.17.7:/usr/share/logstash/data --rm -ti  --entrypoint bash  docker.elastic.co/logstash/logstash:7.17.7

nobody@d47a3308a495:/usr/share/logstash$ logstash -e 'input { stdin { } } output { stdout {} }'
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2023-01-10T18:50:03,183][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2023-01-10T18:50:03,200][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.7", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.16+8 on 11.0.16+8 +indy +jit [linux-x86_64]"}
[2023-01-10T18:50:03,204][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2023-01-10T18:50:03,254][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2023-01-10T18:50:03,274][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2023-01-10T18:50:03,806][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-01-10T18:50:03,851][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"6f060474-0c58-48ab-80dd-39a196200fc1", :path=>"/usr/share/logstash/data/uuid"}
[2023-01-10T18:50:05,320][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
[2023-01-10T18:50:05,323][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and may be removed in a future release.
Please configure Metricbeat to monitor Logstash. Documentation can be found at: 
https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
[2023-01-10T18:50:05,746][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-01-10T18:50:05,813][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-01-10T18:50:06,128][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2023-01-10T18:50:06,268][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch: Temporary failure in name resolution", :exception=>Manticore::ResolutionFailure, :cause=>java.net.UnknownHostException: elasticsearch: Temporary failure in name resolution}
[2023-01-10T18:50:06,273][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Temporary failure in name resolution"}
[2023-01-10T18:50:06,308][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch", :exception=>Manticore::ResolutionFailure, :cause=>java.net.UnknownHostException: elasticsearch}
[2023-01-10T18:50:06,313][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2023-01-10T18:50:06,322][WARN ][logstash.licensechecker.licensereader] Attempt to validate Elasticsearch license failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch"}
[2023-01-10T18:50:06,348][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2023-01-10T18:50:06,385][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2023-01-10T18:50:06,651][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
^C[2023-01-10T18:50:07,948][WARN ][logstash.runner          ] SIGINT received. Shutting down.
[2023-01-10T18:50:08,748][INFO ][org.reflections.Reflections] Reflections took 158 ms to scan 1 urls, producing 119 keys and 419 values 
[2023-01-10T18:50:09,867][WARN ][deprecation.logstash.codecs.line] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-01-10T18:50:09,902][WARN ][deprecation.logstash.inputs.stdin] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-01-10T18:50:10,486][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x2fd5fd2a run>"}
^C[2023-01-10T18:50:11,231][FATAL][logstash.runner          ] SIGINT received. Terminating immediately..
[2023-01-10T18:50:11,382][FATAL][org.logstash.Logstash    ] 
org.jruby.exceptions.ThreadKill: null
nobody@d47a3308a495:/usr/share/logstash$ exit
exit
root@kind:~#

Running 7.17.8 as root works - no need to show full output (same as with 7.17.7)

# Running 7.17.8 as root works - no need to show full output
mkdir data.7.17.8.root
chmod -R ugo+rw data.7.17.8.root
docker run -v $PWD/data.7.17.8.root:/usr/share/logstash/data --rm -ti  --entrypoint bash docker.elastic.co/logstash/logstash:7.17.8
logstash@d0cc92175a9e:~$ logstash -e 'input { stdin { } } output { stdout {} }'
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
...

Provide logs (if relevant):

lioncubs commented 1 year ago

Found the issue - in backtrace

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[FATAL] 2023-01-10 18:30:31.009 [main] Logstash - Logstash stopped processing because of an error: (LoadError) no such file to load -- logstash/build
org.jruby.exceptions.LoadError: (LoadError) no such file to load -- logstash/build

The file to load that fails logstash/build.rb not logstash/build

The Permissions for the following file had been changed from 7.17.7 -> 7.17.8

# logstash:7.17.7 - File is OK
root@kind:/home/radware/git/waas/waas_backend/docker-images/logstash# docker run -u nobody:nogroup -v $PWD/data.7.17.7:/usr/share/logstash/data --rm -ti  --entrypoint bash docker.elastic.co/logstash/logstash:7.17.7

nobody@4b9d4f571e35:/usr/share/logstash$ ls -al ./logstash-core/lib/logstash/build.rb 
-rw-rw-r-- 1 logstash root 156 Oct 13 13:24 ./logstash-core/lib/logstash/build.rb
nobody@4b9d4f571e35:/usr/share/logstash$ #Verify that this is the ONLY file that is not other+r
nobody@4b9d4f571e35:/usr/share/logstash$ find . ! -perm /o+r
nobody@4b9d4f571e35:/usr/share/logstash$ 

# logstash:7.17.8 - File is NOT OK
root@kind:/home/radware/git/waas/waas_backend/docker-images/logstash# docker run -u nobody:nogroup -v $PWD/data.7.17.8:/usr/share/logstash/data --rm -ti  --entrypoint bash docker.elastic.co/logstash/logstash:7.17.8

nobody@45c2639679b8:/usr/share/logstash$  ls -al ./logstash-core/lib/logstash/build.rb
-rw-rw---- 1 logstash root 156 Nov 30 16:11 ./logstash-core/lib/logstash/build.rb
nobody@45c2639679b8:/usr/share/logstash$ #Verify that this is the ONLY file that is not other+r
nobody@45c2639679b8:/usr/share/logstash$ find . ! -perm /o+r
./logstash-core/lib/logstash/build.rb
nobody@45c2639679b8:/usr/share/logstash$

Currently we have patched our docker image from docker.elastic.co/logstash/logstash:7.17.8 with the following changes, and all works - So there is your Fix :slight_smile:

Dockerfile

# Get Original Docker image from elastic.co
FROM docker.elastic.co/logstash/logstash:7.17.8

# fix permission issue on /usr/share/logstash/logstash-core/lib/logstash/build.rb in 7.17.8 (also exists in 8.6.0)
RUN chmod 0664 /usr/share/logstash/logstash-core/lib/logstash/build.rb

NOTE: I have also verified that this issue ALSO exists with 8.6.0 - So I assume any changes to 8.x since Nov 30 16:11 also has the same issue - Please make the same change in the 8.x Branch

kaisecheng commented 1 year ago

The same issue was found in 8.5.1 but not in 8.5.0 diff of two versions

kaisecheng commented 1 year ago

build.rb is created from temp file from rake artifact:generate_build_metadata. The temp file has permissions 0600. In the docker build process, chmod and chown change the permission of user and group. I do not see any change related to others. It is unclear to me why v8.5.0 has -rw-rw-r--, while v8.5.1 has -rw-rw----.

The fix will be update Dockerfile.j2 to add read permission to others

kaisecheng commented 1 year ago

btw, I have tested locally with v8.5.0 and run the docker image build process, but the build.rb is missing read permission.