Blockchain storage limit ENV variable

[shawaj]

See here

https://github.com/helium/blockchain-core/pull/808/ Supersedes: https://github.com/NebraLtd/hm-dashboard/issues/759

Acceptance Criteria

• BLOCKCHAIN_ROCKSDB_GC_BYTES: 8GB (bytes!)
• blocks_to_protect_from_gc: 6000

[shawaj]

https://engineering.helium.com/2021/12/30/blockchain-release-snapshot-loading-and-storage-improvements.html

[shawaj]

@SebastianMaj FYI I think the blocks to protect thing needs to go in the blockchain section of sys.config (hm-block-tracker) not as an ENV variable

Ref from above link...

a blocks_to_protect_from_gc variable in the blockchain section of sys.config (current default: 100000)

[shawaj]

Also of note is:

The last 100k of blocks and snapshots are protected from GC.

In https://github.com/helium/blockchain-core/pull/808

I guess this is just the default but I wonder if 6000 blocks is too little. We should enquire with Helium I think

[Kerrryu]

Also of note is:

The last 100k of blocks and snapshots are protected from GC.

In helium/blockchain-core#808

I guess this is just the default but I wonder if 6000 blocks is too little. We should enquire with Helium I think

That would be a good thing to clarify with them just to be safe considering their value is so high. What would be the best way to contact them about this?

[shawaj]

@marvinmarnold has them on the internal/external channel on the Helium Discord.

Alternatively maybe just a comment on the github? (Just asked here https://github.com/helium/blockchain-core/pull/808#issuecomment-1015726045 )

But the Discord probably quicker

[shawaj]

@SebastianMaj @marvinmarnold I think that 100k might be a typo there. The code seems to say 10k as the default, unless I'm mistaken:

https://github.com/helium/blockchain-core/blob/3cd6bca6c5595a1363a9bbd625ef254383a4141b/src/blockchain.erl#L1988

[shawaj]

Last but not least, it seems that BLOCKCHAIN_ROCKSDB_GC_BYTES should only be set when we need to prune the storage, and it prunes it by at least the amount of bytes we have requested.

I'm not sure what happens if you have it set permanently or whether that will cause issues?

But perhaps it could just be set on a schedule if it's not good for it to be set permanently.

In any case, worth investigating further

[marvinmarnold]

The current documentation says 10k, they must have updated. I'll reach out on Discord and get some input too.

[shawaj]

Yeah maybe it's 10k then. Worth checking this and whether we can just set and further the environment variable rather than only setting it when we need to prune it

[shawaj]

Reply here looks like 6k will be fine https://github.com/helium/blockchain-core/pull/808#issuecomment-1015938490

[marvinmarnold]

Via discord, 2k a possible min.

[shawaj]

Via discord, 2k a possible min.

Yeah same on the other link @marvinmarnold - wonder if 5k is best just to be safe.

How big is the average block in bytes I wonder?

[vpetersson]

I've manually set this variable on TestNet. Kevin is monitoring the situation.