`Server` encounters `MissingDataException`s

[helins]

This looks fairly new. MissingDataExceptions relating to data from other peers are very common but those ones seem to happen when doing some local work:

["2023-07-03T11:40:12.897" :warn [convex.peer.AThreadedComponent 34] ["Unexpected exception in Server component: " {:trace [[convex.core.data.RefSoft getValue "RefSoft.java" 97] [convex.core.data.VectorLeaf copyToArray "VectorLeaf.java" 449] [convex.core.data.VectorTree copyToArray "VectorTree.java" 353] [convex.core.data.VectorLeaf copyToArray "VectorLeaf.java" 445] [convex.core.data.AVector toArray "AVector.java" 155] [java.util.ArrayList addAll "ArrayList.java" 670] [convex.peer.CVMExecutor prevalidateSignatures "CVMExecutor.java" 73] [convex.peer.CVMExecutor loop "CVMExecutor.java" 50] [convex.peer.AThreadedComponent$ComponentTask run "AThreadedComponent.java" 29] [java.lang.Thread run "Thread.java" 833]],:message "Missing hash:0x2e402ff230c16da7657b8951680ff4ae0d5ffc44b14ba1f2a8ae929cbbc2c6d4 in store EtchStore at: /tmp/peer/store.etch",:exception convex.core.exceptions.MissingDataException}]]

[helins]

Also, it is common that the Etch instance I collect after the run has some missing data, rendering block analysis impossible (BPS, TPS, etc). Probably related?

[mikera]

Hmm that's in the new signature parallel validation code. Let me check.

[mikera]

There is in general no guarantee that the store contains all data at any point in time (it could for example have persisted some partial messages before receiving the rest). When it is necessary that data is complete, it's up to the peer to validate this (i.e. is the data fully persisted) before proceeding.

[helins]

E.g. Counts from peer log files (10 min run):

2.cvx:43
3.cvx:100
4.cvx:94
7.cvx:40
5.cvx:91
1.cvx:55
0.cvx:45580
9.cvx:2840
8.cvx:71935
6.cvx:73313

A very important problem, which might be somehow related, is Etch analytics is broken because of missing data exceptions that prevent analysing blocks (TPS etc).

[helins]

Looks resolved (tested against 80589055). No occurence within 7 full runs!

Note: Might explain why #495 looks resolved

[mikera]

Closing this for now as I think this specific issue is resolved.