wwestgarth
commented
11 months ago I don't think there is anything we can do about tendermint needing two available nodes to RPC with, but I do think this is a further indication that we have the order of events the wrong way around and we should be state-syncing in the core node before/at-the-same-time as the datanode is auto-initialising. Doing this will also fix this issue: https://github.com/vegaprotocol/vega/issues/10244
In general I think we have a lot of usability issues with using network-history that we need to resolve that I think we should address in controlled, planned way because the architecture of the network-history code in the data-node is clunky and "spaghetti". If we try to just throw in little fixes it'll just get worse. It needs a bit of a rewrite.
Heres a brief selection of things we need to address: Performance:
- IPFS does not handle big files well. We need to improve the compression of the history segments to make them smaller. This was considered here https://github.com/vegaprotocol/vega/issues/9817 but rejected due to issues with backwards compatibility, but I do not think that is a good enough excuse. We need to work it out because it will clearly improve the speed of getting.
- We should look into the chunking algorithms that can be used with
.Unixfs().Add()best suit out data. Increasing the chunk size can apparently increaseGet()performance, but may reduce in fewer shared blocks between segments. So we need to understand what works for us. - https://github.com/vegaprotocol/vega/issues/10042 mentions an IPFS bug that cuts connections. We should investigate this further and file and IPFS bug, and also try to work around it because my understanding is that Kubo's boostrapper continually tries to reconnect when node connects drop too low. A similar lost-connection issue used to plague the system-tests, but with a little effort we were able to resolve it: https://github.com/vegaprotocol/vega/issues/10222
Usability:
- Reorder so core starts from a snapshot first, then sends the data-node starts auto-initialising. I think having them start in parallel is a trivial change (and probably worth while on its own) but it is worth considering having the data-node start auto-initialising strictly afterwards (which has the benefit that core can tell the data-node its block-height). I think this will be a much bigger change though.
- Add "progress logging" when fetching a segment so that we users can understand the progress better. The kubo library does it so it must be possible: https://github.com/ipfs/kubo/blob/master/core/commands/get.go#L231
- Attach meta-data to each segment ID so that we can attach useful information about a segment (which version created it, the compression format etc etc) so that we can resolve the apparent backward-compatibility problems.
Memory:
- We already have this issue https://github.com/vegaprotocol/vega/issues/9808 where the summary is that the current architecture of auto-initialisation requires all segments to be pulled up front. This is a terrible design and we need to rewrite it.
Problem encountered
I am not sure if it is a bug, but definitely makes starting the node annoying.
The timeline:
10 mlnblocks for my data-nodeAFTER 3 DAYS. However, because I hadonerpc server(api1.vega.community) which was not healthy it moved all syncing time to the trash because starting the node failed and after restart it started syncing network history sync from the beginning.Observed behaviour
I got the following error after 3 days of syncing the network history:
Then vegavisor restarted binaries and downloading 10 mln of blocks from the network history started again.
Expected behaviour
I am not sure what the should be correct behaviour, but we should not lose our syncing progress when something is not right :( We can have much more issues than only a single tendermint peer issue like networking issue, some hiccups on the server and all of the above will end with failure and syncing from the beginning.
Steps to reproduce
Software version
v0.73.9
Failing test
No response
Jenkins run
No response
Configuration used
Data node config:
Tendermint config:
Relevant log output