For the node to test 8mb blocks I had to generate the chain using chainbuilder. It took ~2 weeks to generate 400k of blocks. A few days before chainbuilder finished, I looked at htop and saw the process was running for 12dxxx(I didn't save a screenshot unfortunately and its impossible to know how much the process took after it finishes)
Problem
This is sorta manageable for 8mb blocks, but for further sizes its gonna grow and it feels like tendency is quadratic.
Observing the htop I could see It only saturates like 20% of CPU for a second or two, then silence for sometime and back to 20%. This cycle repeats over and over. Fundamentally, there shouldn't be any issues with saturating all the CPU to progress chain generation, and this may suggest that some operations are sequential, while could be paralellized.
Ideas
If it turns out that the App's storage for blocks is a bottleneck here, we may consider migrating the tool to write direclty to the node storage, essentially making chainbuilder produced artifacts consumable by a BN.
Context
For the node to test 8mb blocks I had to generate the chain using
chainbuilder. It took ~2 weeks to generate 400k of blocks. A few days beforechainbuilderfinished, I looked athtopand saw the process was running for12dxxx(I didn't save a screenshot unfortunately and its impossible to know how much the process took after it finishes)Problem
This is sorta manageable for 8mb blocks, but for further sizes its gonna grow and it feels like tendency is quadratic.
Observing the
htopI could see It only saturates like 20% of CPU for a second or two, then silence for sometime and back to 20%. This cycle repeats over and over. Fundamentally, there shouldn't be any issues with saturating all the CPU to progress chain generation, and this may suggest that some operations are sequential, while could be paralellized.Ideas
chainbuilderproduced artifacts consumable by a BN.