Directory under patterns directory can crash logstash at startup

[JalehD]

• Version: 5.3.1
• Operating System: any
• Config File (if you have sensitive info, please remove it):
• Sample Data: no need for data
• Steps to Reproduce:

1- Install logstash 5.3.1, 2- Create a patterns directory and reference it using pattern_directory in conf. below is a conf sample: `input { stdin { } }

filter { grok { patterns_dir => "/opt/logstash/patterns" match => { "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}' } }

date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] locale => en } }

output { elasticsearch { hosts => ["localhost:9200"] } }`

3- Create an empty directory under the patterns directory. 4- start logstash and see it crash with below error

[2017-05-05T11:04:40,864][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x8b7e48c @id=\"c1ec15479fd638b036420c5eb13aa96522533774-2\", @klass=LogStash::Filters::Grok, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x4212ce4 @metric=#<LogStash::Instrument::Metric:0x4e8f512b @collector=#<LogStash::Instrument::Collector:0x5a287b29 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x6d0c7764 @store=#<Concurrent::Map:0x60c7302d @default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x54f52317>, @fast_lookup=#<Concurrent::Map:0x4d9dfb93 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :\"c1ec15479fd638b036420c5eb13aa96522533774-2\", :events]>, @logger=#<LogStash::Logging::Logger:0x1abb0402 @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0xdffbf7a>>, @filter=<LogStash::Filters::Grok patterns_dir=>[\"/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/patterns\"], match=>{\"message\"=>\"%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\\\[%{HTTPDATE:timestamp}\\\\] \\\"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\\\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}\"}, id=>\"c1ec15479fd638b036420c5eb13aa96522533774-2\", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>\"*\", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>[\"_grokparsefailure\"], timeout_millis=>30000, tag_on_timeout=>\"_groktimeout\">>", :error=>"undefined methodclose' for nil:NilClass"} [2017-05-05T11:04:40,872][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<NoMethodError: undefined method close' for nil:NilClass>, :backtrace=>["/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:83:inadd_patterns_from_file'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:405:in add_patterns_from_files'", "org/jruby/RubyArray.java:1613:ineach'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:401:in add_patterns_from_files'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:284:inregister'", "org/jruby/RubyArray.java:1613:in each'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:280:inregister'", "org/jruby/RubyHash.java:1342:in each'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:275:inregister'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/pipeline.rb:257:in register_plugin'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/pipeline.rb:268:inregister_plugins'", "org/jruby/RubyArray.java:1613:in each'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/pipeline.rb:268:inregister_plugins'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/pipeline.rb:278:in start_workers'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/pipeline.rb:207:inrun'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/agent.rb:389:in start_pipeline'"]}

[JalehD]

Additional info from @guyboertje " when using the standard patterns_files_glob of "*" a directory will be globbed in the list of files found but we don't check for directories later on when we open the file. this ruby File.exists?(folder) returns true for a directory <- bug!!! we should check that its a file not a folder - this is in the grok filter line 402 of v3.4.0

"

[original-brownbear]

@guyboertje on it as discussed :)

[suyograo]

Fixed in #111