shouldRetryToCorrelateMessageSubscriptionAfterOpenedSubscription is flaky

[ana-vinogradova-camunda]

Summary

• How often does the test fail? - Failed once
• Does it block your work? - No
• Do we suspect that it is a real failure? - Don't know

Failures

Example assertion failure

org.awaitility.core.ConditionTimeoutException: 
Condition with alias 'retry correlation' didn't complete within 10 seconds because assertion condition defined as a Lambda expression in io.camunda.zeebe.engine.processing.message.MessageStreamProcessorTest 
interPartitionCommandSender.sendCommand(
    0,
    PROCESS_MESSAGE_SUBSCRIPTION,
    CORRELATE,
    
);
Wanted 2 times:
-> at io.camunda.zeebe.engine.processing.message.MessageStreamProcessorTest.lambda$shouldRetryToCorrelateMessageSubscriptionAfterOpenedSubscription$5(MessageStreamProcessorTest.java:170)
But was 35 times:

Logs

Link to the build: https://github.com/camunda/zeebe/actions/runs/8786935713/job/24110951937?pr=17646

Logs: 1_Unit tests.txt

[mustafadagher]

ZPA triage

• The issue is unclear right now, @nicpuppa and @mustafadagher will take a look on it.

[nicpuppa]

I was able to reproduce the failure locally.

The failure is happening because of this lines. I think the problem is a race condition with the time of the clock and the PendingMessageSubscriptionChecker

[korthout]

Unassigned due to unavailability

[koevskinikola]

ZPA mob programming results:

• We managed to replicate the issue locally
• We couldn't get any additional information as to why the flakiness occurs.
• Our hypothesis is that by using the SimpleProcessingScheduleService#runDelayed method for scheduling the checker class, we introduce a flexible delay that makes the test flaky. By replacing the method above with the SimpleProcessingScheduleService#runAt method, we believe the flakiness will be removed since the latter introduces a fixed timestamp for the task execution. The following is our attempt to confirm our hypothesis.
  • We could replicate the flakiness locally after ~550 runs of the test.
  • After applying the fix (locally), the flakiness didn't appear even after 3200 runs of the test.