iavl 0.19.4

[faddat]

I'm currently downloading a snapshot that contains terra's latest state.

I would like it if other team member can explicitly check to ensure that this is a non-breaking change:

git clone https://github.com/classic-terra/classsic
cd classic
git checkout v1.0.x
go install ./...
terrad init test
cd ~/.terrad
aria2c -x5 https://dl2.quicksync.io/columbus-5-pruned.20230102.0410.tar.lz4
lz4 -d columbus-5-pruned.20230102.0410.tar.lz4 | tar xf -
terrad start

The chain should convert the database, then start. There should be no apphash error.

When that's done, I'm good to go with work towards upgrading to sdk v0.45.x

You should expect for the transition to take genuinely a long time.

• [ ] Implement support for IAVL 0.19.4 in Cosmos-SDK release/v0.45x

[vuong177]

I've tried with v1.x version, and get apphash err.

[vuong177]

We should upgrade through an upgrade module. It's more safely.

[faddat]

oh interesting.

I got a lot more than an apphash that's for sure.

[ZaradarBH]

https://dl2.quicksync.io/columbus-5-pruned.20230102.0410.tar.lz4 returns a 404. Do you have another location I can fetch the genesis from you want us to test with? :)

[inon-man]

@ZaradarBH new snapshot can be found at https://quicksync.io/networks/terra.html

[inon-man]

v1.0.4 (latest stable) sync ok

11:13PM INF commit synced commit=436F6D6D697449447B5B373320302038312031342032353420343820313835203938203135392031343820323230203834203130352031373120313539203133203230382033312032312031393420313735203435203130312031313920323230203433203136332031383020323038203235352033203133395D3A4136434434447D
11:13PM INF committed state app_hash=4900510EFE30B9629F94DC5469AB9F0DD01F15C2AF2D6577DC2BA3B4D0FF038B height=10931533 module=state num_txs=3
11:13PM INF indexed block height=10931533 module=txindex
11:13PM INF executed block height=10931534 module=state num_invalid_txs=0 num_valid_txs=1
11:13PM INF commit synced commit=436F6D6D697449447B5B313635203334203733203736203138352034332031363720313334203235352032303020323139203230332031333620313835203233332031373020313739203133382032342030203134302031363620313230203136372031303220313635203136362039392037392034302035203230395D3A4136434434457D
11:13PM INF committed state app_hash=A522494CB92BA786FFC8DBCB88B9E9AAB38A18008CA678A766A5A6634F2805D1 height=10931534 module=state num_txs=1
11:13PM INF indexed block height=10931534 module=txindex
11:13PM INF executed block height=10931535 module=state num_invalid_txs=0 num_valid_txs=120
11:13PM INF commit synced commit=436F6D6D697449447B5B3131302031363720313834203136302037322032353320323620323131203333203232322031323420363520363720323020393020313033203135382031333120313320313531203133362039372031313220383920353220313137203137203139352037312034362038382034385D3A4136434434467D
11:13PM INF committed state app_hash=6EA7B8A048FD1AD321DE7C4143145A679E830D9788617059347511C3472E5830 height=10931535 module=state num_txs=120
11:13PM INF indexed block height=10931535 module=txindex
11:13PM INF executed block height=10931536 module=state num_invalid_txs=0 num_valid_txs=2

[inon-man]

v1.0.x is doing lots of goleveldb compaction on application.db

❯ terrad start --home . 11:14PM INF starting ABCI with Tendermint

17 minutes so far

[ZaradarBH]

I was think this behavior might be due to the massive WASM blobs in the genesis? Most likely the OWNDAO-style casinos and similar dapps could be causing issues at the DB level due to their sheer size? At least that is the theory I am following while looking at the persistency architecture of Cosmos-SDK. But I think we should try to do some more benchmarking and possible find time to do a event storming session on the subject to map out how it works in detail and then we can start analyzing it for performance bottlenecks and discuss possible workaround. :)

[ZaradarBH]

https://www.eventstorming.com/

[inon-man]

I was think this behavior might be due to the massive WASM blobs in the genesis? Most likely the OWNDAO-style casinos and similar dapps could be causing issues at the DB level due to their sheer size? At least that is the theory I am following while looking at the persistency architecture of Cosmos-SDK. But I think we should try to do some more benchmarking and possible find time to do a event storming session on the subject to map out how it works in detail and then we can start analyzing it for performance bottlenecks and discuss possible workaround. :)

Good point but this is nothing to do with the genesis. Let's talk about it in separate issue.

[inon-man]

I am doing CPU profiling of terrad due to the long start up time. (after the compaction)

---

iavl-profile1.pdf

Looks like iavl enableFastStorageAndCommit is trying to load up all the blocks at the beginning. The questions are

1. Is this necessary? Can it be done in the background?
2. Is this one time thing?

   Looks like it is. https://github.com/cosmos/iavl/blob/b97511aee4c37dd3c99c3f81e5ec3d7e12303d3c/mutable_tree.go#L667-L710

[inon-man]

It is syncing!

12:49AM INF committed state app_hash=07BE7794C03DD1C3F13B82013E664A57D0D23B0767AAAC2941C66DAD6D1E1E10 height=10932416 module=state num_txs=5
12:49AM INF indexed block height=10932416 module=txindex
12:49AM INF executed block height=10932417 module=state num_invalid_txs=0 num_valid_txs=2
12:49AM INF commit synced commit=436F6D6D697449447B5B323336203235342031383120353520382037382031313820313135203230362031303320383720323035203137302031363720313134203234322032303820323436203137203934203133312038312032343420313735203235332031373220313530203736203135372031373820323232203139315D3A4136443043317D
12:49AM INF committed state app_hash=ECFEB537084E7673CE6757CDAAA772F2D0F6115E8351F4AFFDAC964C9DB2DEBF height=10932417 module=state num_txs=2
12:49AM INF indexed block height=10932417 module=txindex
12:49AM INF burned tokens from module account amount=488585683uluna from=burn module=x/bank
12:49AM INF executed block height=10932418 module=state num_invalid_txs=0 num_valid_txs=6
12:49AM INF commit synced commit=436F6D6D697449447B5B31392032343620323532203137352031373320353420343920383120313330203133352031303920313335203135362031313420333020313437203839203139312031342032323220313639203131372031383820373720323032203232203235332031353920393720313038203232203133345D3A4136443043327D
12:49AM INF committed state app_hash=13F6FCAFAD36315182876D879C721E9359BF0EDEA975BC4DCA16FD9F616C1686 height=10932418 module=state num_txs=6
12:49AM INF indexed block height=10932418 module=txindex
12:49AM INF burned tokens from module account amount=200000000000uluna from=burn module=x/bank
12:49AM INF executed block height=10932419 module=state num_invalid_txs=0 num_valid_txs=1
12:49AM INF commit synced commit=436F6D6D697449447B5B32302031333320302031393220313138203232342034302032353420313737203138302037352033322031353820373020323035203131312031343220313534203834203636203534203137332031393520373820373420312032313720313233203836203920323438203137325D3A4136443043337D
12:49AM INF committed state app_hash=148500C076E028FEB1B44B209E46CD6F8E9A544236ADC34E4A01D97B5609F8AC height=10932419 module=state num_txs=1
12:49AM INF indexed block height=10932419 module=txindex

I had to replace tm-db to github.com/terra-money/tm-db v0.6.7-performance.3 to boost start up time (had to ulimit max open file)

[ZaradarBH]

zaradar@Rebel01:/mnt/c/Users/tobia/source/repos/classic$ ./build/terrad start
12:31AM INF starting ABCI with Tendermint
12:31AM INF Starting multiAppConn service impl=multiAppConn module=proxy
12:31AM INF Starting localClient service connection=query impl=localClient module=abci-client
12:31AM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
12:31AM INF Starting localClient service connection=mempool impl=localClient module=abci-client
12:31AM INF Starting localClient service connection=consensus impl=localClient module=abci-client
12:31AM INF Starting EventBus service impl=EventBus module=events
12:31AM INF Starting PubSub service impl=PubSub module=pubsub
12:31AM INF Starting IndexerService service impl=IndexerService module=txindex
12:31AM INF ABCI Handshake App Info hash="\x11u\x1f�\r� \x16�����U�N\x18||%�S䇗yt��s5:" height=10931472 module=consensus protocol-version=0 software-version=1.0.4-1-gc8f133c
12:31AM INF ABCI Replay Blocks appHeight=10931472 module=consensus stateHeight=10931472 storeHeight=10931472
12:31AM INF Completed ABCI Handshake - Tendermint and App are synced appHash="\x11u\x1f�\r� \x16�����U�N\x18||%�S䇗yt��s5:" appHeight=10931472 module=consensus
12:31AM INF Version info block=11 p2p=8 tendermint_version=0.34.14
12:31AM INF This node is not a validator addr=7F72CB44415FDBC7F08B2938BCD211077EAFB0AC module=consensus pubKey=DmIxEzdglx0qwaBEily4qSWAwdWZqj19DCbW9CK1Xqk=
12:31AM INF P2P Node ID ID=1bf983127af1b2701d4f40855ed95fddc88a900f file=/home/zaradar/.terra/config/node_key.json module=p2p
12:31AM INF Adding persistent peers addrs=[] module=p2p
12:31AM INF Adding unconditional peer ids ids=[] module=p2p
12:31AM INF Add our address to book addr={"id":"1bf983127af1b2701d4f40855ed95fddc88a900f","ip":"0.0.0.0","port":26656} book=/home/zaradar/.terra/config/addrbook.json module=p2p
12:31AM INF Starting Node service impl=Node
12:31AM INF Starting pprof server laddr=localhost:6060
12:31AM INF Starting RPC HTTP server on 127.0.0.1:26657 module=rpc-server
12:31AM INF Starting P2P Switch service impl="P2P Switch" module=p2p
12:31AM INF Starting Consensus service impl=ConsensusReactor module=consensus
12:31AM INF Reactor  module=consensus waitSync=true
12:31AM INF Starting Evidence service impl=Evidence module=evidence
12:31AM INF Starting StateSync service impl=StateSync module=statesync
12:31AM INF Starting PEX service impl=PEX module=pex
12:31AM INF Starting AddrBook service book=/home/zaradar/.terra/config/addrbook.json impl=AddrBook module=p2p
12:31AM INF Starting Mempool service impl=Mempool module=mempool
12:31AM INF Starting BlockchainReactor service impl=BlockchainReactor module=blockchain
12:31AM INF Starting BlockPool service impl=BlockPool module=blockchain
12:31AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
12:31AM INF No addresses to dial. Falling back to seeds module=pex
12:31AM INF Saving AddrBook to file book=/home/zaradar/.terra/config/addrbook.json module=p2p size=0

It boots up on my machine, but the apphash is looking alittle funky. Might just be a WSL2 issue. :) It took roughly 30 mins to sync on my machine and used around 60 GB memory.

[inon-man]

It boots up on my machine, but the apphash is looking alittle funky. Might just be a WSL2 issue. :) It took roughly 30 mins to sync on my machine and used around 60 GB memory.

It's been a long time since I lost my 👃 because of funky logs. Shit smells nothing when you smell it a lot. My terrad is taking 8GB mem on iMac 2020.. likely the last iMac with Intel processor.

[ZaradarBH]

Sounds like we should invest in getting you some better hardware. I know at least two whales who would instantly buy and ship you a new computer if it improves your work experience :P

[inon-man]

Oh I never complain about the hardware. I am using 2020 imac because I am still in love with x86 architecture lol

[ZaradarBH]

I still miss my old x386 PC and Norton Commander 4.0 :D Anyways to be honest I have been really keen on trying out the GitHub codespaces since that just a containerized IDE. So maybe we should get Ed to fork over some of the compute budget and play around with that :D

[LuncBurner]

Issue -- Jacob is concerned snapshot may not work. @edk208 will speak with Jagmot to determine if we can obtain access to snapshots.

[inon-man]

@LuncBurner what snapshot do you speaking of? please describe details..

[edk208]

@inon-man Jacob is concerned about the quicksync full archive snapshots

[ZaradarBH]

@inon-man @nghuyenthevinh2000 @edk208 https://github.com/classic-terra/cosmos-sdk/pull/6

[ZaradarBH]

The canonical version of cosmos sdk release/v0.45.x already has the IAVL 0.19.5 package, so in a sense this was never needed. Closing this issue.