Azure DevEx Architecture Board: Introduction to Event Hubs Redis Checkpointstore

[azure-sdk]

Service Name: Event Hubs Checkpointstore For Redis Review Created By: Srikanta Nagaraja Review Date: 2/7/2023 2:05 PM PT

Hero Scenarios Link: Not Provided Architecture Diagram Link: Not Provided Core Concepts Doc Link: Not Provided APIView Links: Java,

Description:

Context

CheckpointStore is an interface we ship so customers can write their own implementation if they want to store their checkpoints somewhere while running EventProcessorClient.

We ship one implementation of this interface using Azure Storage Blob. The proposal is to ship another implementation that supports Redis. In Java, there are a few popular Redis libraries; we want to ship one using Jedis.

Implementations and naming

Each language has one implementation using Azure Blob Storage. A couple of languages implemented the checkpoint store for Azure Table storage.

We suggest azure-messaging-eventhubs-checkpointstore-jedis because both libraries are popular so it is possible someone may want to ship one using Lettuce in the future, so it would be helpful to differentiate between Lettuce and Jedis when customers are choosing an implementation.

Existing implementations and naming

Azure Blob Storage

Language	Package Name	Usage
Java	azure-messaging-eventhubs-checkpointstore-blob	Create BlobCheckpointStore
JS	@azure/eventhubs-checkpointstore-blob	Create BlobCheckpointStore
.NET	Azure.Messaging.EventHubs.Processor	BlobContainerClient is an arg in EventProcessorClient constructor
Python	azure-eventhub-checkpointstoreblob	Create BlobCheckpointStore
Python	azure-eventhub-checkpointstoreblob-aio	Create BlobCheckpointStore

Azure Table storage

Language	Package Name	Usage	Note
Java	-	-
JS	@azure/eventhubs-checkpointstore-table	Create TableCheckpointStore	Shipped beta
.NET	-	-
Python	azure-eventhub-checkpointstoretable	Create TableCheckpointStore	Not shipped

Redis

Language	Package Name	Usage	Library
Java	azure-messaging-eventhubs-checkpointstore-jedis	Create JedisRedisCheckpointStore	Jedis
JS	-	-
.NET	-	-
Python	-	-

Scenario: Using EventProcessorClient with Jedis

HostAndPort hostAndPort = new HostAndPort("{your-hostname}.redis.cache.windows.net", 6380);
JedisClientConfig clientConfig = DefaultJedisClientConfig.builder()
    .password("{your-access-key}")
    .ssl(true)
    .build();
JedisPool jedisPool = new JedisPool(hostAndPort, clientConfig);

// Instantiate an instance of the checkpoint store with configured JedisPool.
CheckpointStore checkpointStore = new JedisRedisCheckpointStore(jedisPool);

EventProcessorClient eventProcessorClient = new EventProcessorClientBuilder()
    .connectionString("event-hub-namespace-connection-string}", "{event-hub-name}")
    .consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
    .processEvent(eventContext -> onEvent(eventContext))
    .processError(errorContext -> onError(errorContext))
    .checkpointStore(checkpointStore)
    .buildEventProcessorClient();

eventProcessorClient.start();

Detailed meeting information and documents provided can be accessed here

[KrzysztofCwalina]

Why is this just for Java?

[conniey]

Why is this just for Java?

As far as we know, there has yet to be an ask in other languages, but in Java, we have had one customer ask about integration with Redis. Also, it was our intern project this past summer.

[tg-msft]

Recording (MS INTERNAL ONLY)