Duplicate code executions if consumer channel was unexpectedly closed

[if0s]

Case of infobip https://elasticio.slack.com/archives/C01FLMQ0PB5/p1724140722291269?thread_ts=1723557442.846699&cid=C01FLMQ0PB5

Case of Gomocha

Hi Pavel, Olexandr,

I've created an additional flow for NOU, [AFAS->FMP daily] Stock_Sync->StockItem. This morning, I noticed that my chron expression was incorrect, it was running every minute from 3:00 to 3:59. This is now corrected.

Could you please check whether the page size of 100 is defective? For example, when retrieving a sample, I don't get 100 entities. Additionally, during execution, the number of expected iterations would be about 2.5k, instead of the actual 5.9k. The dataset consists out of about 250k records.

Best Regards,

Bastiaan Pierhagen

Slack conversation

Sailor Bug Report

Description

If the component encounters an unexpectedly closed consumer channel, the sailor will automatically restore the channel and resume processing the previous message by executing the component's code a second time, while the code from the previous execution continues to run.

This can be a significant problem if you're processing a message for an extended period. For example, if you have a trigger that needs to retrieve 50,000 objects from a system, it will do so in batches of 100 objects each - taking 100, emitting 100, and so on. To retrieve all the data, the component may require several minutes or even hours. If, during this process, an error occurs stating Channel unexpectedly closed, but we were listening. Reconnecting and re-listening queue the sailor will restart the main process. However, the previous process will continue to run as well, resulting in the component emitting a significant number of duplicates. If this error occurs several times, it may also lead to an OOM error.

Sailor Version

2.7.2

[if0s]

This error seems to appear 31 minutes after the flow has started.

[if0s]

Looks like it's RabbitMQ problem - Delivery Acknowledgement Timeout