Closed catShaark closed 1 year ago
still needs profiling on live osmosis node to compare the results
will this be able to be part of the next release or should we wait for the next release (unclear when that will be)
This won't be state breaking so if it's ready we can add this to the next release. But I get errors backporting this to v0.19.x
gentle ping
I figure out the deadlock issue is due to the mem db iterator being channel based, the iterator will lock the db during iteration which inturn cause the deadlock. There're already issue on cosmos-db related to this https://github.com/cosmos/cosmos-db/issues/56, https://github.com/cosmos/cosmos-db/issues/47
The sdk doesn't use memdb anymore. Maybe we have our own memdb here
So I have run benchmark on a live osmosis node. Comparing normal iavl batch vs batch with flusher (flush to state at 100KB), the result is almost the same for the case of normal blocks ( blocks that don't cause heavy commit to state ) but for epoch blocks that cause heavy commit, the commit time is reduced by 25%.
running 10 blocks including epoch block with 100KB flush
running 10 blocks including epoch block with normal iavl batch
the benchmark also proves that iavl with batch is not state breaking. I've run 10000 blocks with no app hash error so far
amazing this is super useful!!
gentle ping on this pr, if you are running this on mainnets then it should be good to merge?
yes, we can have this on mainnets. Do we want the flush thresh hold to be configurable. If so then we have to change iavl.NewMutableTreeWithOpts
which is used in sdk root multi logic, that means we also need to make changes to the sdk
here is the commit time of a osmosis epoch block : | Flush threshold | Commit time |
---|---|---|
25 | 27.17s | |
50 | 26.72s | |
100 | 26.8s | |
250 | 26.19s | |
500 | 27.5s | |
750 | 26.76s | |
1000 | 41.83s | |
no-flush | 49.61s |
Seems like we got the same performance in the range from 25 KB to 750KB. I think we should choose 100KB to be the flush threshold.
gentle ping, need help completing this?
thanks for pinging, I will finish this by tomorow
I've made a new PR for this https://github.com/cosmos/iavl/pull/807
Closes #619
nodeDB
struct andNewMutableTreeWithOpts
to replace uses ofdbm.Batch
toBatchWithFlusher