search

Azure / azure-diagnostics-tools

Plugins and tools for collecting, processing, managing, and visualizing diagnostics data and configuration
98 stars 92 forks source link

Logstash-input-azureblob - Error undefined method version for nil:NilClass #219

Closed kevinCodeSavvy closed 4 years ago

kevinCodeSavvy commented 4 years ago

Hello,

Hoping someone within the community has come across this issue we are facing and may have managed to resolve it or have any ideas?

Basically we have installed the azureblob plugin (0.9.13) to ingest logs from azure blob. Our logstash is on version 6.8.6-1

Our input and output config is as follows:

input {
    azureblob {
        storage_account_name => "storage_name"
        storage_access_key => "storage_access_key"
        container => "storage_container_name"
        add_field => { "service" => "blob" }
        tags => [ 'azure' ]
        codec => "json"
    }
}

output {
   if [service] == "blob" and "azure" in [tags] {
     elasticsearch {
        user => "service_account_user"
        password => "password"
        hosts => ["elasticsearch:9200"]
        cacert => '/etc/logstash/certs/ca.crt'
        index => "logstash-blob-v2-%{+xxxx.ww}"
        ssl => true
        ssl_certificate_verification => false
    }
  }
}

The error we are seeing in the logstash logs are:

[ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined methodversion' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:135:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:259:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in block in register_plugins'", "org/jruby/RubyArray.java:1792:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:413:instart_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:311:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:217:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176:in block in start'"], :thread=>"#<Thread:0x86fc3c5 run>"}

Your help would be much appreciated. Many thanks.

Regards, Kevin

pinochioze commented 4 years ago

Hi L1nja, I have faced this issue a few months ago, actually I can not find the root cause, but it seems your logstash server stand behind a firewall or due to network system. I got this issue with logstash server is windows 10 and ussing proxy network. Hope this can help you.

kevinCodeSavvy commented 4 years ago

Hi Pinochioze,

Thank you for the reply. Much appreciated.

We do have proxy setup on our logstash server.

You mean to say you have in your input.conf file something like this?

input {
    azureblob {
        storage_account_name => "storage_name"
        storage_access_key => "storage_access_key"
        container => "storage_container_name"
        proxy_uri => "http://x.x.x.x:3128/"
        add_field => { "service" => "blob" }
        tags => [ 'azure' ]
        codec => "json"
    }
}
pinochioze commented 4 years ago

No L1nja, this configuration "proxy_uri => "http://x.x.x.x:3128/" is not in Azureblob logstash plugin. With my case I let my server connect the internet directly to testing the logstash configuration file. If your server is Linux, you can configure proxy with http_proxy in ENV variable

kevinCodeSavvy commented 4 years ago

Hi Pinochioze,

Thank you.

Yes I can confirm that our linux logstash server does have connection to the internet. As we are able to install azureblob plugin via /usr/share/logstash/bin/logstash-plugin install logstash-input-azureblob

Our env is set for http_proxy as well

The error you were getting was exactly the same we are seeing?

pinochioze commented 4 years ago

Yes, It's the same issue, I have just made the test again. Can you try connect your server without proxy and to the internet directly?

> [2020-04-23T20:12:57,365][ERROR][logstash.javapipeline    ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `version' for nil:NilClass>, :backtrace=>["C:/1/logstash-7.6.0/vendor/local_gems/c8e22907/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:154:in `register'", 
> "C:/1/logstash-7.6.0/logstash-core/lib/logstash/java_pipeline.rb:200:in `block in register_plugins'", 
> "org/jruby/RubyArray.java:1814:in `each'", 
> "C:/1/logstash-7.6.0/logstash-core/lib/logstash/java_pipeline.rb:199:in `register_plugins'", 
> "C:/1/logstash-7.6.0/logstash-core/lib/logstash/java_pipeline.rb:310:in `start_inputs'", 
> "C:/1/logstash-7.6.0/logstash-core/lib/logstash/java_pipeline.rb:270:in `start_workers'", 
> "C:/1/logstash-7.6.0/logstash-core/lib/logstash/java_pipeline.rb:154:in `run'", 
> "C:/1/logstash-7.6.0/logstash-core/lib/logstash/java_pipeline.rb:109:in `block in start'"], "pipeline.sources"=>["C:/1/logstash-7.6.0/appinsight-0.93java-v1.conf"], :thread=>"#<Thread:0x1eaa9326 run>"}
> [2020-04-23T20:12:57,462][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
kevinCodeSavvy commented 4 years ago

Hi Pinochioze,

I am afraid it is a bit complicated than that as our server is a prod instance within AWS. At the moment the only way out to the internet is via our proxy server.

pinochioze commented 4 years ago

If you can not bypass proxy can you try with this $ export LS_JAVA_OPTS="-Dhttp.proxyHost=proxy-srv.mycorp.net -Dhttp.proxyPort=8080"

Or you can add your proxy host and port in jvm.options in /etc/logstash

kevinCodeSavvy commented 4 years ago

in /etc/logstash/jvm.options

what is the valid attribute field to insert proxy host and port?

pinochioze commented 4 years ago

Can you try with this, insert at the end of the file 2 below lines

-Dhttp.proxyHost=proxy-srv.mycorp.net
-Dhttp.proxyPort=8080
kevinCodeSavvy commented 4 years ago

Hi Pinochioze, In this case I have added the following to the jvm.options:

-Dhttp.proxyHost=x.x.x.x -Dhttp.proxyPort=3128

x.x.x.x is just the IP address of the proxy server.

Now I am getting a different error:

[ERROR][logstash.inputs.logstashinputazureblob] Oh My, An error occurred. LeaseNotPresentWithLeaseOperation (409): There is currently no lease on the blob

I can confirm that the storage and container name is correct. I have also checked the storage access key which is also correct.

However, by putting "registry_lease_duration" in the input may have possibly work. As shown below:

input {
    azureblob {
        storage_account_name => "storage_name"
        storage_access_key => "storage_access_key"
        container => "storage_container_name"
        add_field => { "service" => "blob" }
        tags => [ 'azure' ]
        codec => "json"
        registry_lease_duration => -1
    }
}
pinochioze commented 4 years ago

I think your original issue was fixed, can you get your data from Azure blob? This issue "LeaseNotPresentWithLeaseOperation (409)" happens when logstash plugin try to accquire lease of the blob isn't leasesd. can you post full of the error log so I can know which line of code makes this issue registry_lease_duration => -1 : this option will hold the lease your blob unlimited, you should set it 60s. I think you should set the "registry_path", logstash plugin will always write to this blob so it may cause of 409 error

kevinCodeSavvy commented 4 years ago

Pipeline starts successfully, however from time to time, I am getting the logstash error of:

[ERROR][logstash.inputs.logstashinputazureblob] Oh My, An error occurred. LeaseNotPresentWithLeaseOperation (409): There is currently no lease on the blob.
RequestId:ded98ffe-e01e-0051-143d-1dde99000000
Time:2020-04-28T09:13:40.8260975Z: ["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/http/retry_policy.rb:58:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/http/http_request.rb:110:in `block in with_filter'", "org/jruby/RubyMethod.java:116:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/http/signer_filter.rb:28:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/http/http_request.rb:110:in `block in with_filter'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/service.rb:36:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/filtered_service.rb:34:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/signed_service.rb:41:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/service/storage_service.rb:62:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/blob_service.rb:62:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/blob_service.rb:445:in `break_lease'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/blob.rb:627:in `break_blob_lease'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:337:in `acquire_lease'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:360:in `register_for_read'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:165:in `process'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:151:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `block in start_input'"] {:exception=>#<Azure::Core::Http::HTTPError:2014 @status_code: 409, @http_response: #<Azure::Core::Http::HttpResponse:0x7b7a94af @http_response=#<Faraday::Response:0x1e05703f @on_complete_callbacks=[], @env=#<Faraday::Env @method=:put @body="\xEF\xBB\xBF<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>LeaseNotPresentWithLeaseOperation</Code><Message>There is currently no lease on the blob.\nRequestId:ded98ffe-e01e-0051-143d-1dde99000000\nTime:2020-04-28T09:13:40.8260975Z</Message></Error>"

I have changes the input.conf registry_lease_duration from -1 to 60

Still no change

kevinCodeSavvy commented 4 years ago

It seems like if I change the input config to the below, then I see logs in Kibana temporarily. However, I get the error below in logstash logs:

[ERROR][logstash.inputs.logstashinputazureblob] Oh My, An error occurred. Error:undefined method `length' for nil:NilClass: Trace: ["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:210:in `process'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:151:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `block in start_input'"] {:exception=>#<NoMethodError: undefined method `length' for nil:NilClass>}
input {
    azureblob {
        storage_account_name => "storage_name"
        storage_access_key => "storage_access_key"
        container => "storage_container_name"
        add_field => { "service" => "blob" }
        tags => [ 'azure' ]
        codec => line
        registry_lease_duration => -1
        file_head_bytes => 21
        file_tail_bytes => 9
    }
}
kevinCodeSavvy commented 4 years ago

I have now managed to resolve the issue by doing a couple of things:

  1. Changing the input.conf file to:
input {
    azureblob {
        storage_account_name => "storage_name"
        storage_access_key => "storage_access_key"
        container => "storage_container_name"
        add_field => { "service" => "blob" }
        tags => [ 'azure' ]
        codec => "json"
        registry_lease_duration => 60
        file_head_bytes => 0
        file_tail_bytes => 0
    }
}
  1. Added the following to the jvm.options file in /etc/logstash:

-Dhttp.proxyHost=x.x.x.x -Dhttp.proxyPort=3128

x.x.x.x is the IP address of the proxy server we are using that our AWS logstash instance is using.

  1. Setting the http_proxy in /etc/environment

Hope this will help others who may come across this issue in the future. Many thanks to pinochioze and good luck!