search

janmg / logstash-input-azure_blob_storage

This is a plugin for Logstash to fetch files from Azure Storage Accounts
Other
29 stars 8 forks source link

undefined method `each' for nil:NilClass #21

Closed pacrutchet closed 2 years ago

pacrutchet commented 2 years ago

Hello,

I'm using the following config to ingest the flowlogs of Azure NSG (version 1):

azure_blob_storage { storageaccount => "xxxxxxxxflowlogs" access_key => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" container => "insights-logs-networksecuritygroupflowevent" registry_create_policy => "resume" logtype => "nsgflowlog" codec => "json" tags => "nsg" id => "nsg_5" }

But I getting the following error and crash: [2021-12-20T17:38:49,688][ERROR][logstash.javapipeline ][main][nsg_3] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::AzureBlobStorage container=>"insights-logs-networksecuritygroupflowevent", codec=><LogStash::Codecs::JSON id=>"json_e3ee24a1-4cef-4acd-88ad-fxxxxxxxxxxxxxxxx", enable_metric=>true, charset=>"UTF-8">, logtype=>"nsgflowlog", storageaccount=>"xxxxxxflowlogs", access_key=>, registry_create_policy=>"resume", id=>"nsg_3", tags=>["nsg"], enable_metric=>true, dns_suffix=>"core.windows.net", registry_path=>"data/registry.dat", interval=>60, addfilename=>false, debug_until=>0, debug_timer=>false, skip_learning=>false, file_head=>"{\"records\":[", file_tail=>"]}", path_filters=>["*/"]>

Error: undefined method `each' for nil:NilClass

Exception: NoMethodError

Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_blob_storage-0.12.0/lib/logstash/inputs/azure_blob_storage.rb:392:in nsgflowlog' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_blob_storage-0.12.0/lib/logstash/inputs/azure_blob_storage.rb:257:inblock in run' org/jruby/RubyHash.java:1415:in each' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_blob_storage-0.12.0/lib/logstash/inputs/azure_blob_storage.rb:226:inrun' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:409:in inputworker' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:400:inblock in start_input'

How can I figure out which record is breaking the plugin ? Is there a way to ignore the bad record and move on ?

Thanks for your great plugin!

janmg commented 2 years ago

You can set debug_until to enable debug logging for the first amount of events, depending on how much events you expect to succeed before it goes wrong, I use 5000.

I can check which for each is in line 392 which unexpectedly doesn't have a value and catch the error, log some crash info and continue unless a parameter is set. I'll do it hopefully tomorrow

pacrutchet commented 2 years ago

that would be great since it is inside : [2021-12-20T21:46:18,873][INFO ][logstash.inputs.azureblobstorage][main][nsg_3] worklist contains 1475 blobs [2021-12-20T21:46:18,873][INFO ][logstash.inputs.azureblobstorage][main][nsg_3] 3: processing data/registry from 0 to 940946

That is a lot of records to check :)

janmg commented 2 years ago

I pushed version 0.12.1 which has catches any exception and prints this message, so you know which name and how many records it has ... now I come to think of it, maybe the records size isn't acutally filled and will cause an execption. I'll take it out and publish 0.12.2

@logger.error("NSG Flowlog problem for #{name}, with #{json["records"].size} records and error message #{e.message}")

janmg commented 2 years ago

I pushed 0.12.2, now you will have a log entry with error that starts with "NSG Floglog problem" and then the filename that caused the issue and the message of the exception. would be great if you can report back the exception message, that could give a clue on why the JSON is empty in your case

pacrutchet commented 2 years ago

Worked great! The problem was the file data/registy created by the logstash plugin made by Azure that I tried in the past.

I've deleted it and it now works without errors