Finished but out-of-order signage points (SPs) are incorrectly reported as skipped

[ems1985]

Describe the bug

Due to increased stress on systems caused by the recent "dust storm" (read more here: https://github.com/Chia-Network/chia-blockchain/discussions/9049) it can happen that signage points (SPs) are cached and then finished out of order. It seems when SPs are out of order chiadog thinks that the SPs were skipped, even though they are finished a few seconds later.

This screenshot shows the skipped SP warning plus the log for the warning time showing the out-of-order SPs:

Here is a log from the chat discussing this possible issue:

Environment:

• OS: Ubuntu Server
• Python version: 3.8.10
• PIP version: pip 21.3.1
• Chia version: 1.2.10
• Chiadog version: 0.7.0
• Harvester: local

[martomi]

We have a test for out-of-order signage points here: https://github.com/martomi/chiadog/blob/main/tests/chia_log/logs/finished_signage_point/scrambled.txt

Seems that doesn't cover current events on the network. I may look into this but since I don't have access to running node at the moment, someone needs to prepare and upload a text file with example sequence that fails the test as the linked above example. This format also works. Then I can look into it.

If someone else wants to look into it, please start by adding a test here that fails and we can start working on a fix: https://github.com/martomi/chiadog/blob/main/tests/chia_log/handlers/test_finished_signage_point_handler.py

[ems1985]

Unfortunately all I have is this screenshot above. Don't have the old log files anymore and I didn't get the skipped SPs message anymore since the dust storm appears to be over...

[martomi]

Closing due to lack of additional actionable information.