flakey_flow fails with 12 or 24 messages unacknowledged.

[petersilva]

• there are two polls configured to share responsibility for a directory during the test.
• as part of the test the broker is repeatedly stopped and started (to observe robustness to that event.)
• as part of the test the active poll is swapped ever time the broker is restarted.

the flakey_tests on github often fail with 12 or 24 messages unacked, but are otherwise correct. It seems to be a problem with:

• flowcb/gather/message/ gather() is called by poll flow to get messages posted in order to update it's recent files cache.
• the flowcb/scheduled/poll.py has a gather routine that pauses... waiting for next polling interval.

so after reading a batch of messages, we then pause until the next scheduled polling interval... then we continue processing and eventually ack them.

It would seem better to acknolwedge them before we pause.

[petersilva]

[petersilva]

tagging it harmless because there does not seem to be any data loss or actual real-life problem that would result from this. This is an admittedly sub-optimal behaviour, but it results from an obscure condition and torture test that should not cause anything other than display issues in operations.

[reidsunderland]

I think I found the problem.

The messages just go back and forth between ready and unacked because the poll is receiving them and rejecting them. But when the poll is in poll_catching_up state, rejected messages don't get acked.

https://github.com/MetPX/sarracenia/blob/5482e4922c7c78fea5414daa1865e6f5f0f3e6c0/sarracenia/flow/__init__.py#L603-L606

And because those messages never go away, there's always incoming messages, so poll_catching_up just stays True.

[reidsunderland]

I didn't update this discussion. After adding the ack of worklist.rejected in 2f03697, the flakey test was still inconsistent, but since it's been too long since I actually tested it, I forget what was still wrong. :(

After that didn't work 100% of the time, I made the change in 32f3d56.

That change: 1) always does work, even if the poll is in catching up state. 2) work is the normal place where worklist.incoming, ok and rejected get acked, so since it always does work now, the special case (just for polls when catching up) of acking the worklists wasn't needed inside the run loop.

This change eliminated duplicate posts from the poll, and if I remember right also fixed the unacked messages issue. So I think it's at least partially a good change and worth working on some more.

But it's not ready yet, I think it's because the polls would occasionally not post some messages and I'm not quite sure why.

---

Whether or not my change in 32f3d56 is good, we should ack worklist.rejected to eliminate the unacked messages. Maybe 2f03697 + @petersilva's changes to the flakey test in https://github.com/MetPX/sr_insects/pull/63 are enough to get consistent results.