streamnative / flink

Apache Flink
Apache License 2.0
9 stars 7 forks source link

[BUG] Flink pulsar sink delayed message incompatible with DeliveryGuarantee.EXACTLY_ONCE #218

Open predator4ann opened 2 years ago

predator4ann commented 2 years ago

Describe the bug When use PulsarSink with delivery guarantee DeliveryGuarantee.EXACTLY_ONCE and delaySendingMessage, the deliverAtTime would be invalid and consumer can poll message before deliverAtTime.

To Reproduce Steps to reproduce the behavior:

  1. SinkDemo
    PulsarSink.builder()
            .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
            .setServiceUrl(serviceUrl)
            .setAdminUrl(adminUrl)
            .setSerializationSchema(serializationSchema)
            .setTopics(topic)
            .setProperties(props)
            .delaySendingMessage(MessageDelayer.fixed(Duration.ofMinutes(1)))
            .build();
  2. SourceDemo
    PulsarSource.builder()
            .setServiceUrl(serviceUrl)
            .setAdminUrl(adminUrl)
            .setStartCursor(StartCursor.earliest())
            .setDeserializationSchema(deserializationSchema)
            .setTopics(topic)
            .setSubscriptionName("my-subscription")
            .setSubscriptionType(SubscriptionType.Shared)
            .build();

Expected behavior SourceDemo should receive message after at least 1 minute, actually pulsar source receive message before deliverAtTime in message metadata.

Screenshots N/A

Additional context Java client api : org.apache.flink:flink-connector-pulsar:1.15.1 Deploy mode: standalone on docker(2.10.1) & cluster on k8s(2.9.x)

syhily commented 2 years ago

Can you provided your environment?

syhily commented 2 years ago

It should be fixed by: https://github.com/apache/pulsar/issues/17534

syhily commented 2 years ago

The PR is ready: https://github.com/apache/pulsar/pull/17548

syhily commented 2 years ago

We will get this fixed after bump the client to 2.11.0

predator4ann commented 2 years ago

The PR is merged: https://github.com/apache/pulsar/pull/17548