[RIP-46] Rocketmq producer side supports idempotency

[kaori-seasons]

In the current process of rocketmq producing messages, there is uncertainty in the network call itself, that is, the so-called processing state, so there will be repeated situations. This problem is also encountered by many other MQ products. The usual method is to require the consumer to deduplicate when consuming messages. The idempotent basis should be generated by the message producer. When sending a message, we can pass the key of the message. Set the id, the corresponding API is org.apache.rocketmq.common.message.setKeys(String keys) The logic of this part should be optimized by the producer

FEATURE REQUEST

1. Please describe the feature you are requesting.

2. Provide any additional detail on your proposed use case for this feature.

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

4. If there are some sub-tasks involved, use -[] for each sub-task and create a corresponding issue to map to the sub-task:

• sub-task1-issue-number: sub-task1 description here,
• sub-task2-issue-number: sub-task2 description here,
• ...

[zhouxinyu]

@duhenglucky please assign to me

It's a big feature, do you have any plan or proposal if you want to pick up this issue?

[kaori-seasons]

The details are still being sorted out , and because other issues are being implemented, the plan has not yet been sorted out such as issue-30, issue-4256

[duhenglucky]

@complone It's a great improvement proposal. hope to see your detailed design. we'd better discuss it in the community with RIP first, @ltamber and I are willing to join in and complete this feature together.

[kaori-seasons]

The real existence of this function I think I am referring to kafka, etc. to send the transcationId that implements mq。

[azhsmesos]

@complone It's a great improvement proposal. hope to see your detailed design. we'd better discuss it in the community with RIP first, @ltamber and I are willing to join in and complete this feature together.

May I ask where is the corresponding RIP discussed? Can you give me a link

[kaori-seasons]

阿

@azhsmesos I think I need to investigate the implementation mechanism of rocketmq transactions first. I am currently sorting out the related method call timings of kafka and rocketmq transactions. After sorting out, I will establish a RIP discussion

[kaori-seasons]

Hello everyone, the problems we need to solve at this stage are: In order to improve the accuracy of data and ensure the consistency between the source and the target in the end-to-end transmission process, it is necessary to do idempotent at the producer or consumer However, if consumers cannot connect to the nameserver to obtain partition information temporarily due to network jitter, they need to implement idempotent mechanism in the producer For this reason, our goal is to align the produerId and transactionId of kafka. There are three points to note at this stage by following the code

• First, before sending data, you need to manually create a transaction coordinator to control the one to many relationship between PID and transcationId. To this end, you need to ensure that the consumption progress of offset is notified to the tc in a timely manner and that the messages of submitted but unsettled transactions will not be deleted by mistake during the simultaneous process of the scheduled deletion task
• Whether it is necessary to judge the progress of the current offset and the global PID visible transaction offset in PullLiteProducer in order to enable PID ->transactionid to achieve cross partition transaction consistency
• When downstream consumers consume, they need to ensure that messages marked as transactionId in the upstream market are perceived by downstream consumers

[kaori-seasons]

see RIP-47 : https://docs.google.com/document/d/1nResLevPbeGmKwSQiId_jw0tfBhJtoPZTduOQL3qxNg/edit?usp=sharing

[github-actions[bot]]

This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.

[github-actions[bot]]

This issue was closed because it has been inactive for 3 days since being marked as stale.