LeaseNotPresentWithLeaseOperation (409) error

[JulianCeaser]

Logstash is showing the following error when I am using the logstash-input-azureblob plugin

RequestId:6f270250-001e-000c-3adc-5ced4d000000

Time:2019-08-27T13:35:26.1897321Z: ["/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-core-0.1.15/lib/azure/core/http/retry_policy.rb:58:in `call'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-core-0.1.15/lib/azure/core/http/http_request.rb:110:in `with_filter'", "org/jruby/RubyMethod.java:116:in `call'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-core-0.1.15/lib/azure/core/http/signer_filter.rb:28:in `call'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-core-0.1.15/lib/azure/core/http/http_request.rb:110:in `with_filter'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-core-0.1.15/lib/azure/core/service.rb:36:in `call'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-core-0.1.15/lib/azure/core/filtered_service.rb:34:in `call'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-core-0.1.15/lib/azure/core/signed_service.rb:41:in `call'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-storage-0.15.0.preview/lib/azure/storage/service/storage_service.rb:62:in `call'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/blob_service.rb:62:in `call'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/blob_service.rb:445:in `break_lease'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/blob.rb:627:in `break_blob_lease'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:337:in `acquire_lease'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:326:in `acquire_lease'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:360:in `register_for_read'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:346:in `register_for_read'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:165:in `process'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:151:in `run'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:456:in `inputworker'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:449:in `start_input'"] {:exception=>#<Azure::Core::Http::HTTPError:481593824 @status_code: 409, @http_response: #<Azure::Core::Http::HttpResponse:0x3e93a0f @http_response=#<Faraday::Response:0x6e6a24d8 @on_complete_callbacks=[], @env=#<Faraday::Env @method=:put @body="\xEF\xBB\xBF<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>LeaseNotPresentWithLeaseOperation</Code><Message>There is currently no lease on the blob.\nRequestId:6f270250-001e-000c-3adc-5ced4d000000\nTime:2019-08-27T13:35:26.1897321Z</Message></Error>" @url=#<URI::HTTPS:0x291e9e37 URL:https://aermontproddiag.blob.core.windows.net/insights-logs-networksecuritygroupflowevent/data/registry?comp=lease> @request=#<Faraday::RequestOptions open_timeout=60> @request_headers={"User-Agent"=>"logstash-input-azureblob/0.9.11; Azure-Storage/0.15.0-preview (Ruby 1.9.3-p551; Linux linux)", "x-ms-date"=>"Tue, 27 Aug 2019 13:35:25 GMT", "x-ms-version"=>"2016-05-31", "DataServiceVersion"=>"1.0;NetFx", "MaxDataServiceVersion"=>"3.0;NetFx", "Content-Type"=>"application/atom+xml; charset=utf-8", "x-ms-lease-action"=>"break", "x-ms-lease-break-period"=>"30", "Content-Length"=>"0", "Authorization"=>"SharedKey aermontproddiag:gLsgrwZU+RQT8V+/SODyarm+DAM+m5wOD9gmjKP5nng="} @ssl=#<Faraday::SSLOptions verify=true> @response=#<Faraday::Response:0x6e6a24d8 ...> @response_headers={"content-length"=>"242", "content-type"=>"application/xml", "server"=>"Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0", "x-ms-request-id"=>"6f270250-001e-000c-3adc-5ced4d000000", "x-ms-version"=>"2016-05-31", "date"=>"Tue, 27 Aug 2019 13:35:25 GMT", "connection"=>"close"} @status=409>>, @uri=#<URI::HTTPS:0x3da7c791 URL:https://aermontproddiag.blob.core.windows.net/insights-logs-networksecuritygroupflowevent/data/registry?comp=lease>>, @uri: #<URI::HTTPS:0x3da7c791 URL:https://aermontproddiag.blob.core.windows.net/insights-logs-networksecuritygroupflowevent/data/registry?comp=lease>, @description: "There is currently no lease on the blob.\nRequestId:6f270250-001e-000c-3adc-5ced4d000000\nTime:2019-08-27T13:35:26.1897321Z", @type: "LeaseNotPresentWithLeaseOperation">}

[2019-08-27T13:37:48,529][ERROR][logstash.inputs.logstashinputazureblob] Oh My, An error occurred. LeaseNotPresentWithLeaseOperation (409): There is currently no lease on the blob.

My azureblob input plugin is like this

input {
    azureblob {
        storage_account_name => ""
        storage_access_key => ""
        container => "insights-logs-networksecuritygroupflowevent"
        codec => "json"

        # Refer https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-read-nsg-flow-logs
        # Typical numbers could be 21/9 (NSG v1) or 12/2 (NSG v2) depends on the nsg log file types
        file_head_bytes => 12
        file_tail_bytes => 2

        # Enable / tweak these settings when event is too big for codec to handle.
        # break_json_down_policy => "with_head_tail"
        # break_json_batch_count => 2

        # registry file policies
        registry_lease_duration => -1
        registry_create_policy => "start_over"
    }
}

I have attempted using both registry_lease_duration => -1 and 60 but it does not seem to help in either case. Please help me understand what the problem and solution is.

[arunp-motorq]

Did you find a solution to this ?

[kevinCodeSavvy]

I am having the same issue.

Still trying to find a solution. If I do find something useful then I will post that on here too

[kevinCodeSavvy]

The solution I got is to add these to your jvm.options file in your /etc/logstash/ path

-Dhttp.proxyyHost=x.x.x.x -Dhttp.proxyPort=xxxx

x.x.x.x - IP address of your proxy server xxxx - your proxy server port eg 3128

This works if you are using a proxy server.

[dudubis]

I'm having the same issue. But not sure how the poxy setting are related as I'm not using a proxy. The error talks about a broken or a non exiting lease: @description: "There is currently no lease on the blob.\nRequestId:b2818fb2-e01e-013a-42ac-24e112000000\nTime:2020-05-07T20:19:48.2441783Z", @type: "LeaseNotPresentWithLeaseOperation">}

Here is my configuration:

    `storage_account_name => ""
    storage_access_key => ""
    container => "insights-logs-applicationgatewayaccesslog"
    registry_create_policy => "start_over"
    registry_path => "data/reg12344"
    tags => [ "azureblobreq" ]`

is registry_lease_duration => -1 the answer for this issue?

EDIT: adding registry_lease_duration => -1 didn't help

[pinochioze]

Hi Dudubis, This issue may be due to writing on the registry file 'data/reg12344', Do you make sure there is only this logstash writes on this blob at one time? "registry_lease_duration => -1" this option helps to wait until the writing to registry file is done then will break the lease. So if it doen't help, I think there should be something to break the lease