search

crypto-org-chain / cronos

Cronos is the first Ethereum-compatible blockchain network built on Cosmos SDK technology. Cronos aims to massively scale the DeFi, GameFi, and overall Web3 user community by providing builders with the ability to instantly port apps and crypto assets from other chains while benefiting from low transaction fees, high throughput, and fast finality.
Other
294 stars 237 forks source link

Start with ~10GB genesis.json consume ~150GB of RAM at peaks #1158

Open alpha-omega-labs opened 1 year ago

alpha-omega-labs commented 1 year ago

Hello, Start with ~10GB genesis.json consume ~150GB of RAM at peaks and about 80GB after initialization. Is there a way to improve that, maybe cut some checks? It takes 30min to 10h to finish process. Thank you

yihuang commented 1 year ago

you need to open this to cosmos-sdk, I think there's some streaming json support in newer version.

alpha-omega-labs commented 1 year ago

Yes, they seems to be working on it for some time with genesis.json streaming. https://github.com/cosmos/cosmos-sdk/issues/11601 At the moment start of chain from big (not so big, just ~10GB) genesis.json seems to be not possible because or resource usage - ram and CPU. e.g 10GB genesis.json cause up to 200GB ram spikes during initialization and attempts to sign blocks. Keeping in mind that cronos also have very big state - you might benefit from streaming genesis.json in future too.

yihuang commented 1 year ago

I think there's already some streaming support with the collections module: https://github.com/cosmos/cosmos-sdk/blob/main/collections/genesis.go

alpha-omega-labs commented 1 year ago

Looks like there is no actual way to optimize this, chain, resource wide, cannot be started with .json of big size.

We will rollback on v0.44 and after try to make inplace upgrade to cronos. Few questions: Is v0.47 of cronos available? are you going towards v0.5?

Do you have in-place upgrade code or examples for v0.44 to v0.46.15 jump?

Thank you!

leejw51crypto commented 1 year ago

changing to capnproto and flatbuffer would be helpful (not plain text json) or bson