Open giskook opened 3 months ago
It seems increase sequencer's executor's dbMTCacheSize
can cover this situation.
Only happend when sequencer set SequentialProcessL2Block
to false
I think from this pr: https://github.com/0xPolygonHermez/zkevm-node/pull/3604
sequencer may halt under loadtest situation,
root cause:
wip batch (tx-by-tx process) will build his state on executor's dbMTCache, but dbMTCache is a LRU, so after StateRootSyncInterval
time running, old state may dropped by executor. Then when sequencer open a new L2block, it may fail because of missing key.
Way to solve:
dbMTCacheSize
StateRootSyncInterval
X Layer testnet may halt following below configuration: sequencer
SequentialProcessL2Block = false
Sequencer.Finalizer.StateRootSyncInterval = "3600s"
executor
"dbMTCacheSize": 1024,
Hi, this is a known issue with the associative cache in executor/prover, to fix it you need to disable it (useAssociativeCache = false) in executor configuration. The prover team is working to fix the issue.
useAssociativeCache
Our executor do not set useAssociativeCache
, after check the prover's code, i think it's default to false.
Executor‘s configure
{
"runExecutorServer": true,
"runExecutorClient": false,
"runExecutorClientMultithread": false,
"runHashDBServer": true,
"runHashDBTest": false,
"runAggregatorServer": false,
"runAggregatorClient": false,
"runFileGenBatchProof": false,
"runFileGenAggregatedProof": false,
"runFileGenFinalProof": false,
"runFileProcessBatch": false,
"runFileProcessBatchMultithread": false,
"runFileExecutor": false,
"runKeccakScriptGenerator": false,
"runKeccakTest": false,
"runStorageSMTest": false,
"runBinarySMTest": false,
"runMemAlignSMTest": false,
"runSHA256Test": false,
"runBlakeTest": false,
"executeInParallel": true,
"useMainExecGenerated": true,
"useProcessBatchCache": false,
"saveRequestToFile": false,
"saveInputToFile": false,
"saveDbReadsToFile": false,
"saveDbReadsToFileOnChange": false,
"saveOutputToFile": false,
"saveProofToFile": false,
"saveResponseToFile": false,
"saveFilesInSubfolders": false,
"loadDBToMemCache": false,
"loadDBToMemCacheInParallel": false,
"loadDBToMemTimeout": 30000000,
"dbMTCacheSize": 1024,
"dbProgramCacheSize": 1024,
"opcodeTracer": false,
"logRemoteDbReads": false,
"logExecutorServerInput": false,
"logExecutorServerInputGasThreshold": 1048576,
"logExecutorServerResponses": false,
"logExecutorServerTxs": false,
"executorServerPort": 50071,
"executorROMLineTraces": false,
"executorTimeStatistics": false,
"executorClientPort": 50071,
"executorClientHost": "127.0.0.1",
"executorClientLoops": 1,
"hashDBServerPort": 50061,
"hashDBURL": "local",
"aggregatorServerPort": 50081,
"aggregatorClientPort": 50081,
"aggregatorClientHost": "127.0.0.1",
"aggregatorClientMockTimeout": 10000000,
"aggregatorClientWatchdogTimeout": 60000000,
"mapConstPolsFile": false,
"mapConstantsTreeFile": false,
"inputFile": "testvectors/performance/121_tx_input.json",
"inputFile2": "testvectors/aggregatedProof/recursive1.zkin.proof_1.json",
"outputPath": "runtime/output",
"configPath": "config",
"zkevmCmPols_disabled": "runtime/zkevm.commit",
"zkevmCmPolsAfterExecutor_disabled": "runtime/zkevm.commit",
"c12aCmPols": "runtime/c12a.commit",
"recursive1CmPols_disabled": "runtime/recursive1.commit",
"recursive2CmPols_disabled": "runtime/recursive2.commit",
"recursivefCmPols_disabled": "runtime/recursivef.commit",
"finalCmPols_disabled": "runtime/final.commit",
"publicsOutput": "public.json",
"proofFile": "proof.json",
"databaseURL": "postgresql://xgon_prover_user:xxxxx@xgon-state-pap.cluster-cpvlnksujjmh.ap-east-1.rds.amazonaws.com:5432/xgon_prover_db",
"dbNodesTableName": "state.nodes",
"dbProgramTableName": "state.program",
"dbMultiWrite": true,
"dbFlushInParallel": false,
"dbConnectionsPool": true,
"dbNumberOfPoolConnections": 506,
"dbMetrics": false,
"dbClearCache": false,
"dbGetTree": true,
"dbReadRetryDelay": 100000,
"cleanerPollingPeriod": 600,
"requestsPersistence": 3600,
"maxExecutorThreads": 100,
"maxProverThreads": 100,
"maxHashDBThreads": 200
}
System information
zkEVM Node version:
v0.6.7
OS & Version:Linux
Commit hash : (-) Network:Mainnet/Testnet
Expected behaviour
Actual behaviour
X Layer(node 0.6.7) under load test will halt sequencer
Steps to reproduce the behaviour
Under load test, when create a new l2block, because the prev lock is not proccessed, so the new l2block may halt the sequencer.
Backtrace
node's log (X Layer's, but the logical is the same)
or
executor's log
When submitting logs: please submit them as text and not screenshots.