Failure getting/acquiring lease.

Mugen1991 commented 6 years ago

I am currently running 3 Logstash instances for a couple of eventhubs. Every now and then I get the following errors. Is there some parameter to prevent this ? Does this mean, i will have duplicate events or lost events?

[2018-07-04T14:11:03,401][WARN ][com.microsoft.azure.eventprocessorhost.PartitionManager] host logstash-bf6c4ec2-b326-470a-8244-4a1f10f91a17: Failure getting/acquiring lease, skipping
com.microsoft.azure.storage.StorageException: The client could not finish the operation within specified maximum execution timeout.
        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:243) ~[azure-storage-7.0.0.jar:?]
        at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1341) ~[azure-storage-7.0.0.jar:?]
        at com.microsoft.azure.eventprocessorhost.AzureBlobLease.lambda$isExpired$0(AzureBlobLease.java:87) ~[azure-eventhubs-eph-1.0.0.jar:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.util.concurrent.TimeoutException: The client could not finish the operation within specified maximum execution timeout.
        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:242) ~[azure-storage-7.0.0.jar:?]
        ... 4 more
[2018-07-04T14:11:03,402][ERROR][logstash.inputs.azure.errornotificationhandler] Error with Event Processor Host.  {:exception_received_event_args=>"com.microsoft.azure.eventprocessorhost.ExceptionReceivedEventArgs@58e5fd0b"}
[2018-07-04T14:11:03,404][WARN ][com.microsoft.azure.eventprocessorhost.PartitionManager] host logstash-bf6c4ec2-b326-470a-8244-4a1f10f91a17: Exception stealing lease
com.microsoft.azure.storage.StorageException: The client could not finish the operation within specified maximum execution timeout.
        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:243) ~[azure-storage-7.0.0.jar:?]
        at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1341) ~[azure-storage-7.0.0.jar:?]
        at com.microsoft.azure.eventprocessorhost.AzureBlobLease.lambda$isExpired$0(AzureBlobLease.java:87) ~[azure-eventhubs-eph-1.0.0.jar:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.util.concurrent.TimeoutException: The client could not finish the operation within specified maximum execution timeout.
        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:242) ~[azure-storage-7.0.0.jar:?]
        ... 4 more

jakelandis commented 6 years ago

@Mugen1991 - the WARN messages are very noisy from the underlying library. IMO, many of those WARN should be DEBUG since as far as I can tell are fairly normal and routine for lease acquirement for the distribution of nodes. In testing I have seen these alot, and never had any of the WARN messages warrant much attention. (unless you don't see any data, then a misconfiguration like a typo in the connection string is usually hidden in noisy the WARN logs).

However, that ERROR message is new. In testing, I could never get this error to actually fire. It also appears that the logging is incorrect for the error. I have corrected the logging for the error in 1.0.1 that may shed more light on what that ERROR was logged.

bin/logstash-plugin update  logstash-input-azure_event_hubs

mbrancato commented 5 years ago

I'm hitting this with an ERROR logged as well. When I run a plugin update, I get the following:

Updating logstash-input-azure_event_hubs
No plugin updated

Events do seem to flow for a while before getting this error.

logstash-plugins / logstash-input-azure_event_hubs

Failure getting/acquiring lease. #8