search

Azure / azure-event-hubs-go

Golang client library for Azure Event Hubs https://azure.microsoft.com/services/event-hubs
MIT License
90 stars 69 forks source link

Recovery changes #232

Closed richardpark-msft closed 3 years ago

richardpark-msft commented 3 years ago

(draft to review with @jhendrixMSFT)

Changes to the recovery algorithm based on some issues found when EventHubs was throttling requests:

  1. The recovery algorithm could get into a cycle where two goroutines could continue to recover, even when a new link was in place. This fix depends on the change in https://github.com/Azure/go-amqp/pull/42, which allows us to access the link ID (and thus identify if the link that's current is still the "bad" one from when we failed.
  2. When recovering attempt to recover the link first, and not the entire connection, which is generally more expensive.
richardpark-msft commented 3 years ago

@jhendrixMSFT - Okay, pulled this out of draft.

I made a change into the Recovery code to only consider ErrLinkDetached to be a recoverable close error. I didn't touch the others, however, but I think we'll want to revisit if a distinction like that makes sense for session and connection.

I also added in a stress folder (in internal) that has a little program (throttling.go) that lets you see that the throttling does occur, we do detach and that we do recovery. It's not automated yet but it'll let us have a little testbed for the future when we go through the recovery code. It also proves that the status code that comes back from the AMQP layer now is ErrLinkDetached.

richardpark-msft commented 3 years ago

/azp run Azure.azure-event-hubs-go

azure-pipelines[bot] commented 3 years ago
Azure Pipelines successfully started running 2 pipeline(s).
richardpark-msft commented 3 years ago

/azp run Azure.azure-event-hubs-go

azure-pipelines[bot] commented 3 years ago
Azure Pipelines successfully started running 2 pipeline(s).
richardpark-msft commented 3 years ago

/azp run Azure.azure-event-hubs-go

azure-pipelines[bot] commented 3 years ago
Azure Pipelines successfully started running 2 pipeline(s).
richardpark-msft commented 3 years ago

/azp run Azure.azure-event-hubs-go

azure-pipelines[bot] commented 3 years ago
Azure Pipelines successfully started running 2 pipeline(s).
richardpark-msft commented 3 years ago

All tests except for the one known test (https://github.com/Azure/azure-event-hubs-go/issues/225) are passing. Merging.