initial Post indices included in challenge does not equal to the initial Post indices included in the atx

[pigmej]

We have multiple occurrences of initial Post indices included in challenge does not equal to the initial Post indices included in the atx in the logs from different users.

It needs to be debugged and categorized correctly. User claim that he DID NOT reinitialize or change initialization. Third_PC_with_4070_withoutRewards(1).zip spacemesh-log-7f8f332c.txt.2(1).zip

[pigmej]

Another example spacemesh-log-7f8f332c(2).zip

[pigmej]

spacemesh-log-7f8f332c.txt.2(2).zip

here are the logs from close to the beginning of initialization process.

[poszu]

Thanks for the logs @pigmej. Below my findings:

It seems that the bug was triggered by restarting a node while it was waiting for a poet proof.

Here's what happened:

1. smesher initialized data, created initial POST and submitted a challenge to poet. The NipostChallenge contains initial post indices.
2. an update happened, node was restarted
   • the initial post was not persisted
3. the restarted node regenerated the inital POST. Because K2 PoW is multithreaded it found a different pow nonce. It resulted in a completely different initial POST proof (but also a valid one)
4. An ATX was created:
   • nipost challenge contains POST proof indices from the first proof
   • ATX contains indices from the second proof

Relevant logs:

2023-06-11T14:23:41.268+0200    INFO    7711f.post  post setup completed
# FIRST INITIAL PROOF - THIS ON IS PUT INTO NIPOST CHALLENGE
2023-06-11T14:31:43.231+0200    INFO    7711f.post  Found proof for nonce: 139, k2pow: **1152921504606975728**
2023-06-13T09:00:02.061+0200    DEBUG   7711f.nipostBuilder submitting challenge to poet proving service
... update and restart
2023-06-14T13:44:21.689+0200    INFO    00000.defaultLogger App version: v0.3.8-beta.0. Git: 55911eb - 55911ebf27cbef298dc0ccfaeadddad07327e794
# SECOND INITIAL PROOF - THIS ONE IS PUT INTO ATX
2023-06-14T13:52:14.004+0200    INFO    7711f.post  Found proof for nonce: 134, k2pow: **5764607523034248415**

This issue is already mitigated by https://github.com/spacemeshos/go-spacemesh/pull/4493

[poszu]

The fix confirmed on the testnet-05, users were able to finally publish their ATXs.