foldinto maximum call size exceed

[qiweiii]

keep producing blocks, after one night

Then cannot restart:

[ermalkaleci]

are you trying to resume?

[qiweiii]

Also reported by @StrawberryFlavor, he can restart after remove resume. I guess it's related to have too many key value pair in db?

[ermalkaleci]

when we resume it will load all blocks from db, although it will drop older blocks as it loads and keep a certain amount in memory I am not sure how storageLayer is handled. need to check

[qiweiii]

Found that actually foldinto call stack depth is related to number of blocks in memory, for example, if each block has 10 layers, 1000 blocks * 10 will be 10000, that could exceed maximum call stack size (my chrome's max stack size is 11006).

Need to make foldinto not related to number of layers in memory.

Is it okay to do this?

async foldInto(into: StorageLayer, maxDepth = 1000, curDepth = 0) {
    logger.debug({ curDepth }, 'foldInto depth')
    if (curDepth > maxDepth) {
      logger.warn({ curDepth, maxDepth }, 'foldInto max depth reached')
      return this.#parent
    }
    const newParent = await this.#parent?.foldInto(into, maxDepth, curDepth + 1)
    // ...
}

[xlc]

Throw error before stackoverflow isn't going to actually help. Should we ensure limit number of blocks in memory to cap the max depth and then see if there is a better way to handle the storage layers to reduce number of them. For example, we could merge all the layers for a newly produced block.

[qiweiii]

When when build block, layers in new block already merged in finalblock with storageDiff:

https://github.com/AcalaNetwork/chopsticks/blob/dc6626e41108e36d06abeeed5377205b56c934b6/packages/core/src/blockchain/block-builder.ts#L304-L308

I also thinked abt giving each layer an id, and mark a layer is folded to avoid repeated fold. But I am afraid if previous storage is changed somehow, then the newer block won't fold it again.

Another way I can think of is to use a global store for storage key value pairs. Could I ask what is the reason we used StorageLayer in first place?

[xlc]

StorageLayer mimic the actual blockchain implementation that each change set are stored in layer so that they can be recovered easily. For example, when building a block, executing a transaction, it could do something and later throw an error to discard all the previous changes. StorageLayer allow those revert to be implemented efficiently.

But yeah we don't have to strictly following that if we found out it is causing more issues for our own use case.