Getting Error while using SinceDB in azure blob plugin.

[Shri-Harsha]

I am using AzureBlob Plugin to pull the logs of NSG Flow Logs and Azure ActivityLogs to the same ELK instance. My Plugin Input Configuration for the following Logs are as follows. NSG Flow Logs: input { azureblob { storage_account_name => "test1" storage_access_key => "xxxxxxxxxxxxxxx" container => "insights-logs-networksecuritygroupflowevent" codec => "json" type => "nsg_flow_logs"

Refer https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-read-nsg-flow-logs

     # Typical numbers could be 21/9 or 12/2 depends on the nsg log file types
     file_head_bytes => 12
     file_tail_bytes => 2
     sincedb => "/usr/share/logstash/data/nsg_flow_logs_sincedb"
 }

}

For Azure Activity Logs: input { azureblob { storage_account_name => "test2" storage_access_key => "yyyyyyy" container => "insights-operational-logs" codec => "json" type => "activitylogs" sincedb => "/usr/share/logstash/data/azure_activity_logs_sincedb" } }

But the Logstash is throwing the following error: [2018-08-27T06:09:28,137][ERROR][logstash.inputs.logstashinputazureblob] Unknown setting 'sincedb' for azureblob [2018-08-27T06:09:28,146][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:89:in config_init'", "/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:62:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/plugin_factory.rb:97:in plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:110:inplugin'", "(eval):8:in <eval>'", "org/jruby/RubyKernel.java:994:ineval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:82:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:inblock in converge_state'"]}

But I did found that sincedb is acceptable setting for the azure blob plugin. So what is wrong in my configuration.

[brahmnes]

Hi @Shri-Harsha,

Can you point me to the documentation where it says sincedb is a parameter? If you look at the source, there is no mentioning of this parameter anywhere: https://github.com/Azure/azure-diagnostics-tools/blob/master/Logstash/logstash-input-azureblob/lib/logstash/inputs/azureblob.rb

[Shri-Harsha]

Ok I got it. I got confused with other fork. Is there any chance for you to introduce it?

[brahmnes]

Hi @Shri-Harsha,

I don't know what sincedb is supposed to do. Can you describe the behavior/functionality you need? If you want, you can submit a PR and we can review.

[Shri-Harsha]

Hi @brahmnes , Here you can find every thing about sincedb: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html

Simply, it can keep track of the input state. If we don't use sincedb giving multiple inputs for the same input plugin or different plugin with in a single instance will corrupt our data and fields by merging some, by missing some. It can happen even though our inputs have different output indexes.

[brahmnes]

Hi @Shri-Harsha,

Thank you for the link. It sure looks like a useful setting to have. We don't have dev resource or plan to add this at the moment. Feel free to submit a PR.

[Shri-Harsha]

Hi @brahmnes,

@Nordes did done this. But I am getting error when installing his plugin.

Validating /home/ubuntu/azure-diagnostics-tools/Logstash/logstash-input-azureblob/logstash-input-azureblob-0.9.5.gem Installing logstash-input-azureblob Plugin version conflict, aborting ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-core": In snapshot (Gemfile.lock): logstash-core (= 6.4.0)

In Gemfile: logstash-core-plugin-api (>= 0) java depends on logstash-core (= 6.4.0) java

logstash-input-syslog (>= 0) java depends on
  logstash-filter-grok (>= 0) java depends on
    logstash-core (>= 5.6.0) java

logstash-input-azureblob (= 0.9.5) java depends on
  logstash-core (~> 2.0) java

logstash-core (>= 0) java

Running bundle update will rebuild your snapshot from scratch, using only the gems in your Gemfile, which may resolve the conflict.