Closed shufps closed 1 year ago
We have the logs of all COOs and Tendercoos here (retention time 7 days):
(Please ask me for credentials to access the logging server)
invalid proposal: already applied
also happened here: https://github.com/iotaledger/inx-tendercoo/issues/47
I also have pprof profiles for each COO and inx-tendercoo if needed (except the inx-tendercoo on coo-2 that hung in the node is not synced
loop).
OK, the issue is the following:
!meta.Solid || meta.ReferencedByMilestoneIndex > 0 || meta.ShouldReattach
.One important improvement on the tendercoo side must be to also check !meta.Solid || meta.ReferencedByMilestoneIndex > 0 || meta.ShouldReattach
when proposing so that no invalid parents can get proposed. In that case at least a restart or the re-triggering should fix the state. I will integrate this.
However, this only fixes the symptoms. We would need to understand why !meta.Solid || meta.ReferencedByMilestoneIndex > 0 || meta.ShouldReattach
was true for a selected tip! I was strongly assumed that this can never happen. Maybe @alexsporn or @muXxer can help out here, with what exactly those flags mean and how this could have happened for the Hornets.
seems to be fixed.
Alphanet fell out of sync and is not generating new milestones since today about 1:45AM
The weird thing is that neither the inx-tendercoos nor the hornet nodes crashed.
Logs of hornets look normal - they just didn't get milestones anymore.
Logs of inx-tendercoo look weird.
Example:
Important to notice are the timestamps of the logging. There are 2,5h+ where nothing at all happened.
Then the message
trigger next milestone preemptively
appears and inx-tendercoo is doing a little bit.There were no errors but there is one
DEBUG
message that seems suspicious: