Open andsel opened 1 year ago
💯 I strongly support this improvement. We had a case where LS (EPH from Azure point of view) loses the Lease. Slight backpressure may also be a reason for loosing the Lease easily. Additionally, lib has some bugs (eg.: link) which will not be solved it we don't upgrade.
Problem
Logstash Azure EventHub input plugin is implemented using a pretty old version of the Azure client library (
com.microsoft.azure:azure-eventhubs:2.3.2
released on 6 May 2019). The Azure client was rationalized and was createdAzure-sdk-for-java
, creating the libraryazure-messaging-eventhubs
bumping the version to5.x
(current latest is5.13.1
released 20 Sep 2022). This change is a complete rewrite and the API are not compatible, packages were moved and classes renamed or dropped. Using an old library carries problem of lack of compatibility and some subtle errors in the plugin that uses it.Goals
The goal is adapt the plugin to use the latest
azure-messaging-eventhubs
client library.Design
To have a code that could be easily updated to future versions of the Azure client library a rewrite of most of the code in Java is required. The Java code base should be surrounded by Ruby shim to act as a Ruby plugin, with the same spirit used in the Beats input plugin.
Changes at library level
com.azure.<area>.<service>
:com.microsoft.azure.eventhubs
tocom.azure.messaging.eventhubs
com.microsoft.azure:azure-eventhubs-eph
which was an optional library which exposed thecom.microsoft.azure.eventprocessorhost.IEventProcessor
callback interface to do processing has been deprecated. The plugin used this to attach the classLogStash::Inputs::Azure::Processor
, now processing happens to EventProcessorClient which has to be build configuring the factory EventProcessorClientBuilder. The previousbecomes a more declarative definition, which moves from batch of events to single event
private static void onEvent(EventContext eventContext) { EventData event = eventContext.getEventData(); // process the data }