Node sync issue error processing trusted batch

System information

zkEVM Node version: v0.6.1 zkEVM Prover version: v5.0.4 OS & Version: Linux container Network: Mainnet

Expected behaviour

Node is syncing fine.

Actual behaviour

Started to receive many errors related to trusted batch, solving it via deleting latest batch from statedb.batch and ~10 blocks from statedb.block, but then receiving error again^

{
   "level":"error",
   "ts":1710367087.9292197,
   "caller":"l2_shared/processor_trusted_batch_sync.go:192",
   "msg":"syncTrustedState: batch[1998727/1998729] mode full: error processing trusted batch. Error: last L2Block 10725593 in the database 0x34e5edaf1210dfea2c404d82b472eb40b83b10e56a49999aae397c476ee33ead and the trusted batch 0xa51b1f18532cc93519558987ceb600241c59ae93871490082a1fbb5d1cbc723a are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)",
   "pid":1,
   "version":"v0.6.1",
   "stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"
}

After syncing a while the node threw panic for some reason

stack trace

``` {"level":"info","ts":1710369914.1445806,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:270","msg":"producer: send a message to consumer to indicate that we are synchronized. highestBlockRequested:19429174","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710369914.1540272,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 1998734, latestSyncedBatch: 1998732, lastVerifiedBatchNumber: 1998724","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710369914.1540658,"caller":"synchronizer/synchronizer.go:367","msg":"Syncing L1 blocks in parallel lastEthBlockSynced=19429146","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710369914.2055974,"caller":"synchronizer/synchronizer.go:465","msg":"Starting L1 sync orchestrator in parallel block: 19429146","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710369914.2056813,"caller":"l1_parallel_sync/l1_sync_orchestration.go:146","msg":"orchestration: starting consumer","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710369914.205692,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:81","msg":"consumer: Starting consumer. HighestBlockProcessed: 19429146","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710369914.2057388,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:183","msg":"consumer: processing rollupInfo #31: range:[19429147, latest] num_blocks [1] highest_block [19429174] statistics:wasted_time_waiting_for_data [0s] last_process_time [7.630933352s] block_per_second [188588588.588589]","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710369914.207066,"caller":"etrog/processor_l1_sequence_batches.go:227","msg":"processSequenceBatches: (not found batch) ProcessAndStoreClosedBatch . BatchNumber: 1998733, BlockNumber: 19429174 GER:0x0000000000000000000000000000000000000000000000000000000000000000","pid":1,"version":"v0.6.1"} {"level":"error","ts":1710369914.2165747,"caller":"state/batchV2.go:299","msg":"error executor ProcessBatchV2: rpc error: code = ResourceExhausted desc = %!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/state/batchV2.go:299 github.com/0xPolygonHermez/zkevm-node/state.(*State).sendBatchRequestToExecutorV2()\n/src/state/batchV2.go:263 github.com/0xPolygonHermez/zkevm-node/state.(*State).processBatchV2()\n/src/state/batchV2.go:386 github.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessAndStoreClosedBatchV2()\n/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:228 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).ProcessSequenceBatches()\n/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:73 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).Process()\n/src/synchronizer/actions/processor_manager/processor_manager.go:66 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process()\n/src/synchronizer/synchronizer.go:606 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()\n/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:252 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processUnsafe()\n/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:195 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processIncommingRollupInfoData()\n/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:109 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).step()\n/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:83 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).Start()\n/src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:147 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer.func1()\n)","pid":1,"version":"v0.6.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*State).sendBatchRequestToExecutorV2\n\t/src/state/batchV2.go:299\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).processBatchV2\n\t/src/state/batchV2.go:263\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessAndStoreClosedBatchV2\n\t/src/state/batchV2.go:386\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).ProcessSequenceBatches\n\t/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:228\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).Process\n\t/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:73\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:606\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processUnsafe\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:252\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processIncommingRollupInfoData\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:195\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).step\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:109\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).Start\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:83\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer.func1\n\t/src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:147"} {"level":"error","ts":1710369914.2166698,"caller":"state/batchV2.go:300","msg":"error executor ProcessBatchV2: rpc error: code = ResourceExhausted desc = ","pid":1,"version":"v0.6.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*State).sendBatchRequestToExecutorV2\n\t/src/state/batchV2.go:300\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).processBatchV2\n\t/src/state/batchV2.go:263\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessAndStoreClosedBatchV2\n\t/src/state/batchV2.go:386\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).ProcessSequenceBatches\n\t/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:228\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).Process\n\t/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:73\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:606\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processUnsafe\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:252\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processIncommingRollupInfoData\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:195\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).step\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:109\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).Start\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:83\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer.func1\n\t/src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:147"} {"level":"error","ts":1710369914.216713,"caller":"state/batchV2.go:301","msg":"error executor ProcessBatchV2 response: ","pid":1,"version":"v0.6.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*State).sendBatchRequestToExecutorV2\n\t/src/state/batchV2.go:301\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).processBatchV2\n\t/src/state/batchV2.go:263\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessAndStoreClosedBatchV2\n\t/src/state/batchV2.go:386\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).ProcessSequenceBatches\n\t/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:228\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).Process\n\t/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:73\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:606\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processUnsafe\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:252\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processIncommingRollupInfoData\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:195\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).step\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:109\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).Start\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:83\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer.func1\n\t/src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:147"} panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x124 pc=0x1264d6e] goroutine 19736 [running]: github.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessAndStoreClosedBatchV2(0xc000b13080, {0x1d5dc38, 0xc000322640}, {0x1e7f8d, {0x14, 0x8e, 0xe7, 0xda, 0xf1, 0x65, ...}, ...}, ...) /src/state/batchV2.go:387 +0x4ae github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).ProcessSequenceBatches(0xc0003a4a80, {0x1d5dc38, 0xc000322640}, {0xc000382600, 0x2, 0x1673100?}, 0x1287736, {0x0, 0xedd8422cf, 0x2ac7da0}, ...) /src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:228 +0x2d46 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).Process(0xc0012aa0f0?, {0x1d5dc38?, 0xc000322640?}, {{0x18d265d?, 0x74e2bc?}, 0xc000a2d580?}, 0x410645?, {0x1d69040?, 0xc001c0cd08?}) /src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:73 +0x8b github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process(0x6?, {0x1d5dc38, 0xc000322640}, 0xc001c0cd08?, {{0x18d265d?, 0x0?}, 0x0?}, 0xc000ce8000, {0x1d69040, 0xc001c0cd08}) /src/synchronizer/actions/processor_manager/processor_manager.go:66 +0x1d4 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange(0xc000f46000, {0xc000ce8000, 0x1, 0x101?}, 0xc000a2dbe8?) /src/synchronizer/synchronizer.go:606 +0x925 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processUnsafe(0xc0012cc000, {{0x128771b, 0xffffffffffffffff}, {0xc000ce8000, 0x1, 0x1}, 0xc000c89b00, 0x0, 0xc00071ed20}) /src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:252 +0x5c2 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processIncommingRollupInfoData(0xc0012cc000, {{0x128771b, 0xffffffffffffffff}, {0xc000ce8000, 0x1, 0x1}, 0xc000c89b00, 0x0, 0xc00071ed20}) /src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:195 +0x4b3 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).step(0xc0012cc000) /src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:109 +0x173 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).Start(0xc0012cc000, {0x1d5dc38?, 0xc000323900?}, 0x0?) /src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:83 +0x168 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer.func1() /src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:147 +0xbe created by github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer in goroutine 391 /src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:144 +0x169 ```

And after the restart here we go again with the wrong trusted batch:

node logs

``` {"level":"error","ts":1710370050.0644684,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[1998735/1998736] error processing trusted batch 1998735: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710370050.0646856,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370051.068992,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 1998734, latestSyncedBatch: 1998734, lastVerifiedBatchNumber: 1998724","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370051.0690598,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370051.069081,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370051.1075046,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:1998734 syncTrustedState:1998736 (max Batch on network: 1998736)","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370051.3870082,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:57","msg":"We have this GlobalExitRoot (0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb) in L1block 19429027, so we are synced from L1 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998734 globalExitRoot: 0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb ","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370051.3872137,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998734/1998736] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 1998734","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370051.409931,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=20966/20935, proverID=7980f636-07d4-43d2-89d6-838987ad1295/7980f636-07d4-43d2-89d6-838987ad1295","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370051.4099772,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 20935, executor have write 20966","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370051.7332203,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:57","msg":"We have this GlobalExitRoot (0x30ed22426569deab4cfe385579c917ae1f07221e3b94c81245ea94aba466758f) in L1block 19429081, so we are synced from L1 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998735 globalExitRoot: 0x30ed22426569deab4cfe385579c917ae1f07221e3b94c81245ea94aba466758f ","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370051.7333446,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998735/1998736] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370053.9832506,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 21037","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370054.2167802,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[1998735/1998736] mode full: Batch 1998735: batchl2data len:36987 processed and stored: l2block[10726461-10726696] txs[27] oldStateRoot: 0x75ae261b78b535df2712bdeb59b51a6f774e340c993e9c3a1789f761380dd06c -> newStateRoot:0x16d6314ad5b23af0951a2110cb73eb035410b6373e20d95228d0c498e1e8134a","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370054.2216856,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0xda9f1d668d349c47e571c0641ceb09f42b86e69d4e16f6e76ee6c0e2660b7f2a 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x16d6314ad5b23af0951a2110cb73eb035410b6373e20d95228d0c498e1e8134a 0xb3f903467201c428548261b49d02ec01c79946e45e4336fcd8886864f0f81b1a 0x579e27323cdf233cf47d410465662ee897f32dd2b5b2e2299597dc69fecaa35d [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc0006ecc70 25443 10726696 1125899906842624 5378829 1710369792 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc001e0fae8 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 [{0x727eae01cd6e40248d249caea3057339265df08b61a24d4cb133768db2257061 }] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0x7fc50a8bcb920dbaab19c023da996885dfd055baa360a325413291b855dd4e9e}","pid":1,"version":"v0.6.1"} {"level":"error","ts":1710370054.2218394,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[1998735/1998736] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710370054.2219543,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[1998735/1998736] mode full: error processing trusted batch. Error: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710370054.222058,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[1998735/1998736] error processing trusted batch 1998735: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710370054.2222428,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370055.2258563,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 1998734, latestSyncedBatch: 1998734, lastVerifiedBatchNumber: 1998724","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370055.2259398,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370055.225963,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370055.2511442,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:1998734 syncTrustedState:1998736 (max Batch on network: 1998736)","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370055.469605,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:57","msg":"We have this GlobalExitRoot (0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb) in L1block 19429027, so we are synced from L1 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998734 globalExitRoot: 0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb ","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370055.469759,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998734/1998736] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 1998734","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370055.483649,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=21040/21037, proverID=7980f636-07d4-43d2-89d6-838987ad1295/7980f636-07d4-43d2-89d6-838987ad1295","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370055.4836767,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 21037, executor have write 21040","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370055.7722223,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:57","msg":"We have this GlobalExitRoot (0x30ed22426569deab4cfe385579c917ae1f07221e3b94c81245ea94aba466758f) in L1block 19429081, so we are synced from L1 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998735 globalExitRoot: 0x30ed22426569deab4cfe385579c917ae1f07221e3b94c81245ea94aba466758f ","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370055.7723475,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998735/1998736] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370057.6473503,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 21052","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370057.8437352,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[1998735/1998736] mode full: Batch 1998735: batchl2data len:36987 processed and stored: l2block[10726461-10726696] txs[27] oldStateRoot: 0x75ae261b78b535df2712bdeb59b51a6f774e340c993e9c3a1789f761380dd06c -> newStateRoot:0x16d6314ad5b23af0951a2110cb73eb035410b6373e20d95228d0c498e1e8134a","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370057.8646288,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0xda9f1d668d349c47e571c0641ceb09f42b86e69d4e16f6e76ee6c0e2660b7f2a 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x16d6314ad5b23af0951a2110cb73eb035410b6373e20d95228d0c498e1e8134a 0xb3f903467201c428548261b49d02ec01c79946e45e4336fcd8886864f0f81b1a 0x579e27323cdf233cf47d410465662ee897f32dd2b5b2e2299597dc69fecaa35d [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc001f19d30 25443 10726696 1125899906842624 5378829 1710369792 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc001d2cdb0 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 [{0x727eae01cd6e40248d249caea3057339265df08b61a24d4cb133768db2257061 }] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0x7fc50a8bcb920dbaab19c023da996885dfd055baa360a325413291b855dd4e9e}","pid":1,"version":"v0.6.1"} {"level":"error","ts":1710370057.8648028,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[1998735/1998736] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710370057.8649192,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[1998735/1998736] mode full: error processing trusted batch. Error: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710370057.8650284,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[1998735/1998736] error processing trusted batch 1998735: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710370057.8652534,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.6086614,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:19429198 -> new block:19429199","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.6087053,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:483","msg":"producer: New last block on L1: 19429199 -> fullRange: [19429174, 19429199] extendedRange: [19429199, 19429199]","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.6087456,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 1 of 10 ) result: [inFilter:0 + inChannel:20 > maximum:25? ==> allow new req] segment [19429175, latest]/UNSAFE -> [LAUNCHED] [NoNextRange] ","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.6087627,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [synchronized] to [working]","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.67224,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [63.392411ms] err: [] ] result:[ blockRange: [19429175, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true]]","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.6723108,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:597","msg":"producer: sendind data to consumer: [0/1] -> range:[[19429175, latest]] Sending results [data] to consumer:data: blockRange: [19429175, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true] NO_CTRL ","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.6723344,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [working] to [synchronized]","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.6723433,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:270","msg":"producer: send a message to consumer to indicate that we are synchronized. highestBlockRequested:19429174","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.869238,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 1998734, latestSyncedBatch: 1998734, lastVerifiedBatchNumber: 1998724","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.8692884,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.8693042,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370058.8946204,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:1998734 syncTrustedState:1998736 (max Batch on network: 1998736)","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370059.1203156,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:57","msg":"We have this GlobalExitRoot (0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb) in L1block 19429027, so we are synced from L1 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998734 globalExitRoot: 0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb ","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370059.1204906,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998734/1998736] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 1998734","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370059.1267412,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=21069/21052, proverID=7980f636-07d4-43d2-89d6-838987ad1295/7980f636-07d4-43d2-89d6-838987ad1295","pid":1,"version":"v0.6.1"} {"level":"info","ts":1710370059.1267684,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 21052, executor have write 21069","pid":1,"version":"v0.6.1"} ```

Again, dropping a few blocks helped just for a while:

zkevm-node Logs

``` {"level":"info","ts":1710370801.8056965,"caller":"synchronizer/synchronizer.go:115","msg":"Permissionless: creating and Initializing L2 synchronization components","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370801.8057504,"caller":"synchronizer/synchronizer.go:123","msg":"Adding check of L2Block hash on close batch when sync from trusted node","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370801.805828,"caller":"synchronizer/synchronizer.go:136","msg":"L1SynchronizationMode is parallel","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370801.8058608,"caller":"synchronizer/synchronizer.go:208","msg":"Sync started","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370801.8912823,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 1998734, latestSyncedBatch: 1998734, lastVerifiedBatchNumber: 1998734","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370801.89133,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370801.8915045,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370801.99509,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:1998734 syncTrustedState:1998737 (max Batch on network: 1998737)","pid":1,"version":"v0.6.2"} {"level":"warn","ts":1710370802.3035028,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:73","msg":"We are behind this block number. GER=0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb sync in block 19429027 and we are in block 19428925 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998734 globalExitRoot: 0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb ","pid":1,"version":"v0.6.2"} {"level":"error","ts":1710370802.3036757,"caller":"l2_shared/processor_trusted_batch_sync.go:179","msg":"syncTrustedState: batch[1998734/1998737] error checking GlobalExitRoot from TrustedBatch. Error: %!(EXTRA *errors.errorString=must sync from L1, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:179 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:179\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710370802.3038075,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[1998734/1998737] error processing trusted batch 1998734: must sync from L1%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710370802.30402,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: must sync from L1","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3040676,"caller":"synchronizer/synchronizer.go:351","msg":"Syncing from trusted node need data from L1","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.30409,"caller":"synchronizer/synchronizer.go:367","msg":"Syncing L1 blocks in parallel lastEthBlockSynced=19428925","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3554158,"caller":"synchronizer/synchronizer.go:465","msg":"Starting L1 sync orchestrator in parallel block: 19428925","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3554914,"caller":"l1_parallel_sync/l1_sync_orchestration.go:111","msg":"orchestration: producer is not running. Resetting the state to start from block 19428925 (last on DB)","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3555174,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:225","msg":"producer: Reset(19428925) queue cmd and discarding all info in channel","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3555305,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [no_running] to [reseting]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3555624,"caller":"l1_parallel_sync/l1_sync_orchestration.go:146","msg":"orchestration: starting consumer","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3555748,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:78","msg":"consumer: Starting consumer. setting HighestBlockProcessed: 19428925 (lastEthBlockSynced)","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3555846,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:81","msg":"consumer: Starting consumer. HighestBlockProcessed: 19428925","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3555803,"caller":"l1_parallel_sync/l1_sync_orchestration.go:120","msg":"orchestration: starting producer","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3556583,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:334","msg":"producer: starting L1 sync from: lastBlockStoreOnStateDB: invalid, highestBlockRequested:invalid, lastBlockOnL1: invalid, amountOfBlocksInEachRange: 1000, processingRanges: len(ranges): 0, errorRanges: len(ranges): 0","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3556774,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [reseting] to [idle]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3557417,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:445","msg":"producer: received a reset(19428925)","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3557563,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [idle] to [reseting]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3557844,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [reseting] to [idle]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3557897,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:246","msg":"producer: Reset(19428925): reset producer done!","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3581688,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:0 -> new block:19429260","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3581884,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:483","msg":"producer: New last block on L1: 19429260 -> fullRange: [19428925, 19429260] extendedRange: [19428925, 19429260]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3581975,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:377","msg":"producer: producerIdle: have all the data to work, moving to working status. status: lastBlockStoreOnStateDB: 19428925, highestBlockRequested:19428925, lastBlockOnL1: 19429260, amountOfBlocksInEachRange: 1000, processingRanges: len(ranges): 0, errorRanges: len(ranges): 0","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3582065,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [idle] to [working]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3582523,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 1 of 10 ) result: [inFilter:0 + inChannel:0 > maximum:25? ==> allow new req] segment [19428926, latest] -> [LAUNCHED] [NoNextRange] ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370802.3582673,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:404","msg":"producer: producerWorking: still not synchronized with the new block range launch workers again","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.22976,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [871.399057ms] err: [] ] result:[ blockRange: [19428926, latest] len_blocks: [8] len_order:[8] lastBlockOfRangeSet [false] previousBlockSet [false]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.229945,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:597","msg":"producer: sendind data to consumer: [0/1] -> range:[[19428926, latest]] Sending results [data] to consumer:data: blockRange: [19428926, latest] len_blocks: [8] len_order:[8] lastBlockOfRangeSet [false] previousBlockSet [false] NO_CTRL ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2299926,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [working] to [synchronized]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.230011,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:270","msg":"producer: send a message to consumer to indicate that we are synchronized. highestBlockRequested:19429221","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2301342,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:183","msg":"consumer: processing rollupInfo #1: range:[19428926, latest] num_blocks [8] highest_block [19429221] statistics:wasted_time_waiting_for_data [1s] last_process_time [0s] block_per_second [0.000000]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2329812,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19428942, BatchNumber: 1998718, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2340052,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19428942, BatchNumber: 1998719, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2343683,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19428942, BatchNumber: 1998720, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2347214,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19428942, BatchNumber: 1998721, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2351413,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19428942, BatchNumber: 1998722, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2354395,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19428942, BatchNumber: 1998723, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.235753,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19428942, BatchNumber: 1998724, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2379856,"caller":"etrog/processor_l1_info_tree_update.go:52","msg":"L1InfoTree(etrog) stored. BlockNumber: 19428942,GER:0x4388372228e68b5432eeaa2b0857bc3fbd008a468fd61d52bd92b2d7e8306a9c L1InfoTreeIndex: 3586 L1InfoRoot:0x61eb75e636d500bb0e843d618642de6f74f8fa69e32333e311923f6979997b60","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2392244,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=0/0, proverID=e64bff2a-4ca8-49c8-bf70-82c8cb39aec9/","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.239262,"caller":"synchronizer/synchronizer.go:790","msg":"Current proverID is e64bff2a-4ca8-49c8-bf70-82c8cb39aec9","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2392752,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 0, executor have write 0","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370803.2638988,"caller":"etrog/processor_l1_info_tree_update.go:52","msg":"L1InfoTree(etrog) stored. BlockNumber: 19429027,GER:0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb L1InfoTreeIndex: 3587 L1InfoRoot:0xdc41612229a004e21121fd6b97c91c49ffb3e7caf527c2b556baf4f5051c6cb8","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370808.122449,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1998727, BlockNumber: 19429044","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370813.7033331,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1998728, BlockNumber: 19429044","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370814.1063929,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:19429260 -> new block:19429261","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370814.1064343,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:483","msg":"producer: New last block on L1: 19429261 -> fullRange: [19429221, 19429261] extendedRange: [19429261, 19429261]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370814.1064825,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 1 of 10 ) result: [inFilter:0 + inChannel:1 > maximum:25? ==> allow new req] segment [19429222, latest]/UNSAFE -> [LAUNCHED] [NoNextRange] ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370814.1065106,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [synchronized] to [working]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370814.2247932,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [118.167675ms] err: [] ] result:[ blockRange: [19429222, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370814.2249126,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:597","msg":"producer: sendind data to consumer: [0/1] -> range:[[19429222, latest]] Sending results [data] to consumer:data: blockRange: [19429222, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true] NO_CTRL ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370814.224939,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [working] to [synchronized]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370814.2249484,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:270","msg":"producer: send a message to consumer to indicate that we are synchronized. highestBlockRequested:19429221","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370818.5827625,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1998729, BlockNumber: 19429044","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370822.2066095,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1998730, BlockNumber: 19429044","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370824.3466592,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:19429261 -> new block:19429261","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370827.1376967,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1998731, BlockNumber: 19429044","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370831.3602183,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1998732, BlockNumber: 19429044","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370831.362748,"caller":"etrog/processor_l1_info_tree_update.go:52","msg":"L1InfoTree(etrog) stored. BlockNumber: 19429072,GER:0xfadbd1bd4070d3965773aab57f3b5ca5c43e85a543cae8668d57bf52b36029f0 L1InfoTreeIndex: 3588 L1InfoRoot:0x1c8f33b5c96358b3b9b6d079161d71c6607e0e3738a7b0ce2d0570655e5268b9","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370831.3636503,"caller":"etrog/processor_l1_info_tree_update.go:52","msg":"L1InfoTree(etrog) stored. BlockNumber: 19429081,GER:0x30ed22426569deab4cfe385579c917ae1f07221e3b94c81245ea94aba466758f L1InfoTreeIndex: 3589 L1InfoRoot:0x004e9f4e70863f005ec26739371b7c6b0cbd7d97a7a96ccbf17845258409a34a","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370831.364497,"caller":"etrog/processor_l1_info_tree_update.go:52","msg":"L1InfoTree(etrog) stored. BlockNumber: 19429146,GER:0x9d2dadeed1252295aef75443815e10c28e8ab9b86a3a3a1d290d44e55000898b L1InfoTreeIndex: 3590 L1InfoRoot:0xc5f9050f545e08389dcb11ddb12cf4585c2525cddb7c83e6b2a0702d174e3547","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370834.3502533,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:19429261 -> new block:19429262","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370834.3503118,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:483","msg":"producer: New last block on L1: 19429262 -> fullRange: [19429221, 19429262] extendedRange: [19429262, 19429262]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370834.3503625,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 1 of 10 ) result: [inFilter:0 + inChannel:3 > maximum:25? ==> allow new req] segment [19429222, latest]/UNSAFE -> [LAUNCHED] [NoNextRange] ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370834.3503876,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [synchronized] to [working]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370834.472336,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [121.919593ms] err: [] ] result:[ blockRange: [19429222, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370834.472432,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:597","msg":"producer: sendind data to consumer: [0/1] -> range:[[19429222, latest]] Sending results [data] to consumer:data: blockRange: [19429222, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true] NO_CTRL ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370834.4724512,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [working] to [synchronized]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370834.4724586,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:270","msg":"producer: send a message to consumer to indicate that we are synchronized. highestBlockRequested:19429221","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370835.0071297,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1998733, BlockNumber: 19429174","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7953496,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1998734, BlockNumber: 19429174","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7967906,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19429221, BatchNumber: 1998725, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7971063,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19429221, BatchNumber: 1998726, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7973514,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19429221, BatchNumber: 1998727, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7976432,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19429221, BatchNumber: 1998728, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7978613,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19429221, BatchNumber: 1998729, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7980697,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19429221, BatchNumber: 1998730, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7982461,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19429221, BatchNumber: 1998731, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.798429,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19429221, BatchNumber: 1998732, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7986395,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19429221, BatchNumber: 1998733, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7988598,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19429221, BatchNumber: 1998734, isTrusted: true","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.7997274,"caller":"etrog/processor_l1_info_tree_update.go:52","msg":"L1InfoTree(etrog) stored. BlockNumber: 19429221,GER:0x31ec84e8e386139e8feaefd9563deb6808a27bf781dc87d1b9d7d9e53c2ebb4f L1InfoTreeIndex: 3591 L1InfoRoot:0x3e5f087ca61e288e0be8ea8ce84e1f1042703416081065268dc8166fe359646a","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.800029,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:143","msg":"consumer: received a fullSync but still have 4 items in channel to process, so not stopping consumer","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8000524,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:119","msg":"consumer: processed ControlData[action:eventIsFullySynced param:19429221]. Result: %!s()","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8001118,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:183","msg":"consumer: processing rollupInfo #2: range:[19429222, latest] num_blocks [0] highest_block [19429221] statistics:wasted_time_waiting_for_data [0s] last_process_time [35.56983501s] block_per_second [0.439024]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8001788,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:165","msg":"consumer: Verified previous block 19429221 not the same: OK","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8001945,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:236","msg":"consumer: Empty block range: [19429222, latest]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8002086,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:143","msg":"consumer: received a fullSync but still have 2 items in channel to process, so not stopping consumer","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8002377,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:119","msg":"consumer: processed ControlData[action:eventIsFullySynced param:19429221]. Result: %!s()","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8002517,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:183","msg":"consumer: processing rollupInfo #3: range:[19429222, latest] num_blocks [0] highest_block [19429221] statistics:wasted_time_waiting_for_data [0s] last_process_time [65.75µs] block_per_second [0.439022]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8002634,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:236","msg":"consumer: Empty block range: [19429222, latest]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8002732,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:136","msg":"consumer: received a fullSync and nothing pending in channel to process, so stopping consumer. lastBlock: 19429221","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8002841,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:119","msg":"consumer: processed ControlData[action:eventIsFullySynced param:19429221]. Result: consumer:stopped because is synchronized","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.800294,"caller":"l1_parallel_sync/l1_sync_orchestration.go:154","msg":"orchestration: consumer finished","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8003137,"caller":"l1_parallel_sync/l1_sync_orchestration.go:176","msg":"orchestration: consumer has finished. No error","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.8003259,"caller":"l1_parallel_sync/l1_sync_orchestration.go:186","msg":"orchestration: finished L1 sync orchestration With LastBlock. Last block synced: 19429221 err:nil","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370838.800336,"caller":"synchronizer/synchronizer.go:396","msg":"L1 state fully synchronized","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370839.8039932,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 1998734, latestSyncedBatch: 1998734, lastVerifiedBatchNumber: 1998734","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370839.8040478,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370839.804069,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370839.8342283,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:1998734 syncTrustedState:1998737 (max Batch on network: 1998737)","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370840.101766,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:57","msg":"We have this GlobalExitRoot (0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb) in L1block 19429027, so we are synced from L1 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998734 globalExitRoot: 0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370840.1019888,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998734/1998737] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 1998734","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370840.4512377,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:57","msg":"We have this GlobalExitRoot (0x30ed22426569deab4cfe385579c917ae1f07221e3b94c81245ea94aba466758f) in L1block 19429081, so we are synced from L1 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998735 globalExitRoot: 0x30ed22426569deab4cfe385579c917ae1f07221e3b94c81245ea94aba466758f ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370840.451302,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998735/1998737] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370842.4652205,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 9","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370842.6702108,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[1998735/1998737] mode full: Batch 1998735: batchl2data len:36987 processed and stored: l2block[10726461-10726696] txs[27] oldStateRoot: 0x75ae261b78b535df2712bdeb59b51a6f774e340c993e9c3a1789f761380dd06c -> newStateRoot:0x16d6314ad5b23af0951a2110cb73eb035410b6373e20d95228d0c498e1e8134a","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370842.678033,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0xda9f1d668d349c47e571c0641ceb09f42b86e69d4e16f6e76ee6c0e2660b7f2a 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x16d6314ad5b23af0951a2110cb73eb035410b6373e20d95228d0c498e1e8134a 0xb3f903467201c428548261b49d02ec01c79946e45e4336fcd8886864f0f81b1a 0x579e27323cdf233cf47d410465662ee897f32dd2b5b2e2299597dc69fecaa35d [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc000df8840 25443 10726696 1125899906842624 5378829 1710369792 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc001303878 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 [{0x727eae01cd6e40248d249caea3057339265df08b61a24d4cb133768db2257061 }] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0x7fc50a8bcb920dbaab19c023da996885dfd055baa360a325413291b855dd4e9e}","pid":1,"version":"v0.6.2"} {"level":"error","ts":1710370842.6782248,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[1998735/1998737] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710370842.6783326,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[1998735/1998737] mode full: error processing trusted batch. Error: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710370842.6784105,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[1998735/1998737] error processing trusted batch 1998735: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710370842.6788354,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10726696 in the database 0xef0c162297e62458bcbf523abc55fe6a1d3f9d3e6f9382dcfbea1b8c43b1aa29 and the trusted batch 0xab8a6a75abf5066e8a4035c18b6ad9d679ca8f121ffde2a925889bf936b79188 are different","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.5978289,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:19429262 -> new block:19429263","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.5978844,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:483","msg":"producer: New last block on L1: 19429263 -> fullRange: [19429221, 19429263] extendedRange: [19429263, 19429263]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.5979228,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 1 of 10 ) result: [inFilter:0 + inChannel:0 > maximum:25? ==> allow new req] segment [19429222, latest]/UNSAFE -> [LAUNCHED] [NoNextRange] ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.5979412,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [synchronized] to [working]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.6833844,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 1998734, latestSyncedBatch: 1998734, lastVerifiedBatchNumber: 1998734","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.6834476,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.6834617,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.6839552,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [85.972991ms] err: [] ] result:[ blockRange: [19429222, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.6840248,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:597","msg":"producer: sendind data to consumer: [0/1] -> range:[[19429222, latest]] Sending results [data] to consumer:data: blockRange: [19429222, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true] NO_CTRL ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.6840475,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [working] to [synchronized]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.684058,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:270","msg":"producer: send a message to consumer to indicate that we are synchronized. highestBlockRequested:19429221","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.7112415,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:1998734 syncTrustedState:1998737 (max Batch on network: 1998737)","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.9304354,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:57","msg":"We have this GlobalExitRoot (0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb) in L1block 19429027, so we are synced from L1 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998734 globalExitRoot: 0xff5f8519491797f1381db961694c52abde00557e0a931026d7330902e977b0cb ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.9306302,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998734/1998737] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 1998734","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.933526,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=9/9, proverID=e64bff2a-4ca8-49c8-bf70-82c8cb39aec9/e64bff2a-4ca8-49c8-bf70-82c8cb39aec9","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370844.933563,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 9, executor have write 9","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370845.264286,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:57","msg":"We have this GlobalExitRoot (0x30ed22426569deab4cfe385579c917ae1f07221e3b94c81245ea94aba466758f) in L1block 19429081, so we are synced from L1 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998735 globalExitRoot: 0x30ed22426569deab4cfe385579c917ae1f07221e3b94c81245ea94aba466758f ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710370845.2644272,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998735/1998737] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":1,"version":"v0.6.2"} ```

Nah, it syncs up to head batch and then fails:

{"level":"info","ts":1710371744.7810743,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:236","msg":"consumer: Empty block range: [19429306, latest]","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.7810829,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:143","msg":"consumer: received a fullSync but still have 1 items in channel to process, so not stopping consumer","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.7810907,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:119","msg":"consumer: processed ControlData[action:eventIsFullySynced param:19429305]. Result: %!s(<nil>)","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.781105,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:183","msg":"consumer: processing rollupInfo #15: range:[19429306, latest] num_blocks [0] highest_block [19429305] statistics:wasted_time_waiting_for_data [0s] last_process_time [7.96µs] block_per_second [0.381153]","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.7811155,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:236","msg":"consumer: Empty block range: [19429306, latest]","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.783303,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:19429335 -> new block:19429336","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.7833438,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:483","msg":"producer: New last block on L1: 19429336 -> fullRange: [19429305, 19429336] extendedRange: [19429336, 19429336]","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.7833576,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:404","msg":"producer: producerWorking: still not synchronized with the new block range launch workers again","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.7834048,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1  ( working: 1 of 10 ) result: [inFilter:0 + inChannel:0 > maximum:25? ==> allow new req]  segment [19429306, latest]/UNSAFE -> [LAUNCHED] [NoNextRange]  ","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.821745,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange:  generic:[typeOfRequest: [rollup] duration: [38.256886ms] err: [<nil>]  ]  result:[ blockRange: [19429306, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true]]","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.821773,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:597","msg":"producer: sendind data to consumer: [0/1] -> range:[[19429306, latest]] Sending results [data] to consumer:data: blockRange: [19429306, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true]  NO_CTRL  ","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8217943,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [working] to [synchronized]","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8218043,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:270","msg":"producer: send a message to consumer to indicate that we are synchronized. highestBlockRequested:19429305","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8218393,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:183","msg":"consumer: processing rollupInfo #16: range:[19429306, latest] num_blocks [0] highest_block [19429305] statistics:wasted_time_waiting_for_data [0s] last_process_time [10.84µs] block_per_second [0.381060]","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8218496,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:236","msg":"consumer: Empty block range: [19429306, latest]","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8218572,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:136","msg":"consumer: received a fullSync and nothing pending in channel to process, so stopping consumer. lastBlock: 19429305","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8218632,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:119","msg":"consumer: processed ControlData[action:eventIsFullySynced param:19429305]. Result: consumer:stopped because is synchronized","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8218682,"caller":"l1_parallel_sync/l1_sync_orchestration.go:154","msg":"orchestration: consumer finished","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8218782,"caller":"l1_parallel_sync/l1_sync_orchestration.go:176","msg":"orchestration: consumer has finished. No error","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8218882,"caller":"l1_parallel_sync/l1_sync_orchestration.go:186","msg":"orchestration: finished L1 sync orchestration With LastBlock. Last block synced: 19429305 err:nil","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8218968,"caller":"synchronizer/synchronizer.go:396","msg":"L1 state fully synchronized","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.824792,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 1998737, latestSyncedBatch: 1998737, lastVerifiedBatchNumber: 1998734","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.824811,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8248255,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371744.8995245,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:1998737 syncTrustedState:1998740 (max Batch on network: 1998740)","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371745.1578214,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998737/1998740] mode nothing:  Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 1998737","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371745.471844,"caller":"l2_sync_etrog/check_sync_status_to_process_batch.go:57","msg":"We have this GlobalExitRoot (0x31ec84e8e386139e8feaefd9563deb6808a27bf781dc87d1b9d7d9e53c2ebb4f) in L1block 19429221, so we are synced from L1 CheckL1SyncStatusEnoughToProcessBatch batchNumber:1998738 globalExitRoot: 0x31ec84e8e386139e8feaefd9563deb6808a27bf781dc87d1b9d7d9e53c2ebb4f ","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371745.4719331,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998738/1998740] mode full:  Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371749.7502549,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 31","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371749.9151192,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[1998738/1998740] mode full: Batch 1998738: batchl2data len:12837 processed and stored:  l2block[10727081-10727264] txs[36] oldStateRoot: 0xed624abc493144dc4ecda38ab468b0d95b86d821cb56b3029d8f4163b33abf76 -> newStateRoot:0x35bc939338961fc87dd04f8f18ff1f5d404016ec6b02b4b4440f82475439acb2","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371749.918751,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0xf4e9878c0850223e4ada96aed734d43a301e4ab30509d915ccfb43d8c7b1db61 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x35bc939338961fc87dd04f8f18ff1f5d404016ec6b02b4b4440f82475439acb2 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc001b92c30 515 10727264 1125899906842624 0 1710371496 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc0019d1e00 0x39b76d1b4f0942f92bdb482cf36050f8d1b5ed0b6639c674ad19b71eef9a25cb [] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xff6dea9b319acd937feb35ae0f6c9e4deb2d0b0c4c4c442a47319e3fe1d49e31}","pid":1,"version":"v0.6.2"}
{"level":"error","ts":1710371749.9189122,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[1998738/1998740] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10727264 in the database 0x78945459a4aeee50c3e7e6ab4f250223304b9cbb7f0059e4410e3e1b70bc62fc and the trusted batch 0x39b76d1b4f0942f92bdb482cf36050f8d1b5ed0b6639c674ad19b71eef9a25cb are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"}
{"level":"error","ts":1710371749.919067,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[1998738/1998740] mode full: error processing trusted batch. Error: last L2Block 10727264 in the database 0x78945459a4aeee50c3e7e6ab4f250223304b9cbb7f0059e4410e3e1b70bc62fc and the trusted batch 0x39b76d1b4f0942f92bdb482cf36050f8d1b5ed0b6639c674ad19b71eef9a25cb are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"}
{"level":"error","ts":1710371749.9191742,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[1998738/1998740] error processing trusted batch 1998738: last L2Block 10727264 in the database 0x78945459a4aeee50c3e7e6ab4f250223304b9cbb7f0059e4410e3e1b70bc62fc and the trusted batch 0x39b76d1b4f0942f92bdb482cf36050f8d1b5ed0b6639c674ad19b71eef9a25cb are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"}
{"level":"warn","ts":1710371749.9193957,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10727264 in the database 0x78945459a4aeee50c3e7e6ab4f250223304b9cbb7f0059e4410e3e1b70bc62fc and the trusted batch 0x39b76d1b4f0942f92bdb482cf36050f8d1b5ed0b6639c674ad19b71eef9a25cb are different","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371750.9271817,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 1998737, latestSyncedBatch: 1998737, lastVerifiedBatchNumber: 1998734","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371750.9272194,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371750.927233,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371750.9589975,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:1998737 syncTrustedState:1998740 (max Batch on network: 1998740)","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371751.245062,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[1998737/1998740] mode nothing:  Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 1998737","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710371751.2560651,"caller":"synchronizer/synchronizer.go:838","msg":"Synchronized BLOCKED!: Wating for the flushID to be stored. FlushID to be stored: 31. Latest flushID stored: 30","pid":1,"version":"v0.6.2"}

Seeing the same across multiple nodes that have been upgraded

System information zkEVM Node version: v0.6.1 zkEVM Prover version: v5.0.4 OS & Version: Linux container Network: Mainnet

2024-03-14T06:59:06.009Z    ERROR   l2_shared/trusted_batches_retrieve.go:139   syncTrustedState: batch[1998816/1998819] error processing trusted batch 1998816: last L2Block 10736259 in the database 0x000339ec8eac4e078dc88dda49f6fc5de896374e5b9f250f1731330cefea29fd and the trusted batch 0x0543160e3081dd8c42f5b8dbe94568c06c14209ca13301d06c572c8de547e268 are different%!(EXTRA string=
/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()
/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()
/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()
/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()
/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()
/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()
/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()
/src/cmd/run.go:319 main.runSynchronizer()
)   {"pid": 1, "version": "v0.6.2"}
github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom
    /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139
github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState
    /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102
github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState
    /src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState
    /src/synchronizer/synchronizer.go:648
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync
    /src/synchronizer/synchronizer.go:337
main.runSynchronizer
    /src/cmd/run.go:319

This issue has been seen with node 0.6.2 and prover 5.0.7, the current required versions for the hardfork

Please follow these instructions, you need to use node v0.6.2 and executor v5.0.7, versions 0.6.1 and 0.5.4 have issues that affect to the blockhash calculation:

Connect to the state DB and run the following query: select block_num from state.l2block where block_hash = '0x5757afaf506076c3550fe769692c6d3f67a47ab6e2e4337c0350864dc9896a85'
If the query returns the value 10742146 then continue in the next step. In case the query returns no rows, report it in one of our support channels and we will provide more info about next steps
Stop the RPC, Synchronizer and Executor components (keep the state DB running)
Be sure node (RPC and Synchronizer) is updated to v0.6.2 (keep RPC and Synchronizer stopped)
Be sure Executor is updated to v5.0.7 (keep Executor stopped)
Connect to the state DB and run the following queries (force resync from Elderberry fork): delete from state.batch where batch_num >= 1998875; delete from state.block where block_num >= 19433234;
Start the components in this order: 6.1 Executor 6.2 Synchronizer 6.3 RPC

We did this, and after some catchup still see issues. Here are some log snippets

We did the rollback on one node.
zkevm-sync-1  | 2024-03-14T17:47:27.420Z    INFO    synchronizer/synchronizer.go:396    L1 state fully synchronized {"pid": 1, "version": "v0.6.2"}
zkevm-sync-1  | 2024-03-14T17:47:29.423Z    INFO    synchronizer/synchronizer.go:329    latestSequencedBatchNumber: 1998936, latestSyncedBatch: 1998938, lastVerifiedBatchNumber: 1998931   {"pid": 1, "version": "v0.6.2"}
zkevm-sync-1  | 2024-03-14T17:47:29.423Z    INFO    synchronizer/synchronizer.go:336    Syncing trusted state (permissionless)  {"pid": 1, "version": "v0.6.2"}
zkevm-sync-1  | 2024-03-14T17:47:29.423Z    INFO    l2_shared/trusted_batches_retrieve.go:84    syncTrustedState: Getting trusted state info    {"pid": 1, "version": "v0.6.2"}
zkevm-sync-1  | 2024-03-14T17:47:29.521Z    INFO    l2_shared/trusted_batches_retrieve.go:96    syncTrustedState: latestSyncedBatch:1998938 syncTrustedState:1998938 (max Batch on network: 1998938)    {"pid": 1, "version": "v0.6.2"}
zkevm-sync-1  | 2024-03-14T17:47:59.620Z    WARN    l2_shared/trusted_batches_retrieve.go:120   syncTrustedState: batch[1998938/1998938] failed to get batch 1998938 from trusted state. Error: 502 - error code: 502   {"pid": 1, "version": "v0.6.2"}
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom
zkevm-sync-1  |     /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:120
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState
zkevm-sync-1  |     /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState
zkevm-sync-1  |     /src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState
zkevm-sync-1  |     /src/synchronizer/synchronizer.go:648
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync
zkevm-sync-1  |     /src/synchronizer/synchronizer.go:337
zkevm-sync-1  | main.runSynchronizer
zkevm-sync-1  |     /src/cmd/run.go:319
zkevm-sync-1  | 2024-03-14T17:47:59.621Z    WARN    synchronizer/synchronizer.go:340    error syncing trusted state. Error: 502 - error code: 502   {"pid": 1, "version": "v0.6.2"}
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync
zkevm-sync-1  |     /src/synchronizer/synchronizer.go:340
zkevm-sync-1  | main.runSynchronizer
zkevm-sync-1  |     /src/cmd/run.go:319
zkevm-sync-1  | 2024-03-14T17:48:02.643Z    INFO    synchronizer/synchronizer.go:329    latestSequencedBatchNumber: 1998936, latestSyncedBatch: 1998938, lastVerifiedBatchNumber: 1998931   {"pid": 1, "version": "v0.6.2"}

zkevm-sync-1  | 2024-03-14T18:01:07.604Z    ERROR   l2_shared/processor_trusted_batch_sync.go:192   syncTrustedState: batch[1998939/1998940] mode nothing: error processing trusted batch. Error: when closing the batch, the batch is already close, but  the data on state doesnt match the expected%!(EXTRA string=
zkevm-sync-1  | /src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()
zkevm-sync-1  | /src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()
zkevm-sync-1  | /src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()
zkevm-sync-1  | /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()
zkevm-sync-1  | /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()
zkevm-sync-1  | /src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()
zkevm-sync-1  | /src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()
zkevm-sync-1  | /src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()
zkevm-sync-1  | /src/cmd/run.go:319 main.runSynchronizer()
zkevm-sync-1  | )   {"pid": 1, "version": "v0.6.2"}
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch
zkevm-sync-1  |     /src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom
zkevm-sync-1  |     /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState
zkevm-sync-1  |     /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState
zkevm-sync-1  |     /src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState
zkevm-sync-1  |     /src/synchronizer/synchronizer.go:648
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync
zkevm-sync-1  |     /src/synchronizer/synchronizer.go:337
zkevm-sync-1  | main.runSynchronizer
zkevm-sync-1  |     /src/cmd/run.go:319
zkevm-sync-1  | 2024-03-14T18:01:07.604Z    ERROR   l2_shared/trusted_batches_retrieve.go:139   syncTrustedState: batch[1998939/1998940] error processing trusted batch 1998939: when closing the batch, the batch is already close, but  the data on state doesnt match the expected%!(EXTRA string=
zkevm-sync-1  | /src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()
zkevm-sync-1  | /src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()
zkevm-sync-1  | /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()
zkevm-sync-1  | /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()
zkevm-sync-1  | /src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()
zkevm-sync-1  | /src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()
zkevm-sync-1  | /src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()
zkevm-sync-1  | /src/cmd/run.go:319 main.runSynchronizer()
zkevm-sync-1  | )   {"pid": 1, "version": "v0.6.2"}
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom
zkevm-sync-1  |     /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState
zkevm-sync-1  |     /src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState
zkevm-sync-1  |     /src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState
zkevm-sync-1  |     /src/synchronizer/synchronizer.go:648
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync
zkevm-sync-1  |     /src/synchronizer/synchronizer.go:337
zkevm-sync-1  | main.runSynchronizer
zkevm-sync-1  |     /src/cmd/run.go:319
zkevm-sync-1  | 2024-03-14T18:01:07.605Z    WARN    synchronizer/synchronizer.go:340    error syncing trusted state. Error: when closing the batch, the batch is already close, but  the data on state doesnt match the expected    {"pid": 1, "version": "v0.6.2"}
zkevm-sync-1  | github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync
zkevm-sync-1  |     /src/synchronizer/synchronizer.go:340
zkevm-sync-1  | main.runSynchronizer
zkevm-sync-1  |     /src/cmd/run.go:319

My another node started to display another logs:

zkevm-node logs

``` "level":"info","ts":1710618367.4838488,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:404","msg":"producer: producerWorking: still not synchronized with the new block range launch workers again","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618367.988665,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [6.707626008s] err: [] ] result:[ blockRange: [19441519, 19442519] len_blocks: [26] len_order:[26] lastBlockOfRangeSet [false] previousBlockSet [false]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618367.9888175,"caller":"l1_parallel_sync/l1_syncstatus.go:202","msg":"Unexpected finished block_range [19441519, 19442519], ignoring it: block Range not found","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618367.988843,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:589","msg":"producer: Ignoring result because the range is not longer valid: generic:[typeOfRequest: [rollup] duration: [6.707626008s] err: [] ] result:[ blockRange: [19441519, 19442519] len_blocks: [26] len_order:[26] lastBlockOfRangeSet [false] previousBlockSet [false]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618367.9889216,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 10 of 10 ) result: [inFilter:0 + inChannel:0 > maximum:25? ==> allow new req] segment [19437515, 19438515] -> [LAUNCHED] ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618367.9890106,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:404","msg":"producer: producerWorking: still not synchronized with the new block range launch workers again","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618368.5871267,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [10.602643585s] err: [] ] result:[ blockRange: [19438516, 19439516] len_blocks: [35] len_order:[35] lastBlockOfRangeSet [false] previousBlockSet [false]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618368.5872474,"caller":"l1_parallel_sync/l1_syncstatus.go:202","msg":"Unexpected finished block_range [19438516, 19439516], ignoring it: block Range not found","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618368.5872707,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:589","msg":"producer: Ignoring result because the range is not longer valid: generic:[typeOfRequest: [rollup] duration: [10.602643585s] err: [] ] result:[ blockRange: [19438516, 19439516] len_blocks: [35] len_order:[35] lastBlockOfRangeSet [false] previousBlockSet [false]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618368.5873303,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 10 of 10 ) result: [inFilter:0 + inChannel:0 > maximum:25? ==> allow new req] segment [19438516, 19439516] -> [LAUNCHED] ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618368.5873938,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:404","msg":"producer: producerWorking: still not synchronized with the new block range launch workers again","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618371.893608,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [10.112033657s] err: [] ] result:[ blockRange: [19444522, 19445522] len_blocks: [32] len_order:[32] lastBlockOfRangeSet [false] previousBlockSet [false]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618371.8937378,"caller":"l1_parallel_sync/l1_syncstatus.go:202","msg":"Unexpected finished block_range [19444522, 19445522], ignoring it: block Range not found","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618371.8937593,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:589","msg":"producer: Ignoring result because the range is not longer valid: generic:[typeOfRequest: [rollup] duration: [10.112033657s] err: [] ] result:[ blockRange: [19444522, 19445522] len_blocks: [32] len_order:[32] lastBlockOfRangeSet [false] previousBlockSet [false]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618371.8938305,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 10 of 10 ) result: [inFilter:0 + inChannel:0 > maximum:25? ==> allow new req] segment [19439517, 19440517] -> [LAUNCHED] ","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618371.8938987,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:404","msg":"producer: producerWorking: still not synchronized with the new block range launch workers again","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618373.5943856,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [6.497658809s] err: [] ] result:[ blockRange: [19435513, 19436513] len_blocks: [21] len_order:[21] lastBlockOfRangeSet [false] previousBlockSet [false]]","pid":1,"version":"v0.6.2"} {"level":"info","ts":1710618373.594587,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 10 of 10 ) result: [inFilter:1 + inChannel:0 > maximum:25? ==> allow new req] segment [19440518, 19441518] -> [LAUNCHED] ","pid":1,"version":"v0.6.2"} ```

Guys can you pay attention and shed a light on what's happening please? @ToniRamirezM @tclemos @joanestebanr

Restart hepled for a while but then the node threw panic:

zkevm-node panics

``` {"level":"info","ts":1710619137.5390828,"caller":"etrog/processor_l1_sequence_batches.go:227","msg":"processSequenceBatches: (not found batch) ProcessAndStoreClosedBatch . BatchNumber: 1999216, BlockNumber: 19441905 GER:0x009e5854946b1771ffa30c8fa493a41ff471f35b5986ca53b04241b6c43679fc","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.156994,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:19449676 -> new block:19449677","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.1570277,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:483","msg":"producer: New last block on L1: 19449677 -> fullRange: [19449575, 19449677] extendedRange: [19449677, 19449677]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.1570597,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 1 of 10 ) result: [inFilter:0 + inChannel:8 > maximum:25? ==> allow new req] segment [19449576, latest] -> [LAUNCHED] [NoNextRange] ","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.1570745,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [synchronized] to [working]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.172719,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [15.634492ms] err: [] ] result:[ blockRange: [19449576, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [false]]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.1727588,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:597","msg":"producer: sendind data to consumer: [0/1] -> range:[[19449576, latest]] Sending results [data] to consumer:data: blockRange: [19449576, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [false] NO_CTRL ","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.1727796,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [working] to [synchronized]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.1727893,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:270","msg":"producer: send a message to consumer to indicate that we are synchronized. highestBlockRequested:19449575","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.3864443,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 12364","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.3865318,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1999216, BlockNumber: 19441905","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619140.387097,"caller":"etrog/processor_l1_sequence_batches.go:227","msg":"processSequenceBatches: (not found batch) ProcessAndStoreClosedBatch . BatchNumber: 1999217, BlockNumber: 19441905 GER:0x0000000000000000000000000000000000000000000000000000000000000000","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619143.237181,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 12390","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619143.2372806,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1999217, BlockNumber: 19441905","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619143.2377675,"caller":"etrog/processor_l1_sequence_batches.go:227","msg":"processSequenceBatches: (not found batch) ProcessAndStoreClosedBatch . BatchNumber: 1999218, BlockNumber: 19441905 GER:0x9e9da88edb9bcfc2e691903888a3e3ee9ddf40b8bdfcd5698dde935d35fb689c","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619147.3143504,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 12471","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619147.314468,"caller":"etrog/processor_l1_sequence_batches.go:323","msg":"processSequenceBatches: Storing virtualBatch. BatchNumber: 1999218, BlockNumber: 19441905","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710619147.314925,"caller":"etrog/processor_l1_sequence_batches.go:227","msg":"processSequenceBatches: (not found batch) ProcessAndStoreClosedBatch . BatchNumber: 1999219, BlockNumber: 19441905 GER:0x0000000000000000000000000000000000000000000000000000000000000000","pid":7,"version":"v0.6.2"} {"level":"error","ts":1710619147.316064,"caller":"state/batchV2.go:299","msg":"error executor ProcessBatchV2: rpc error: code = ResourceExhausted desc = %!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/state/batchV2.go:299 github.com/0xPolygonHermez/zkevm-node/state.(*State).sendBatchRequestToExecutorV2()\n/src/state/batchV2.go:263 github.com/0xPolygonHermez/zkevm-node/state.(*State).processBatchV2()\n/src/state/batchV2.go:386 github.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessAndStoreClosedBatchV2()\n/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:228 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).ProcessSequenceBatches()\n/src/synchronizer/actions/elderberry/processor_l1_sequence_batches.go:73 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/elderberry.(*ProcessorL1SequenceBatchesElderberry).Process()\n/src/synchronizer/actions/processor_manager/processor_manager.go:66 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process()\n/src/synchronizer/synchronizer.go:606 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()\n/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:252 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processUnsafe()\n/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:195 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processIncommingRollupInfoData()\n/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:109 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).step()\n/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:83 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).Start()\n/src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:147 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer.func1()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*State).sendBatchRequestToExecutorV2\n\t/src/state/batchV2.go:299\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).processBatchV2\n\t/src/state/batchV2.go:263\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessAndStoreClosedBatchV2\n\t/src/state/batchV2.go:386\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).ProcessSequenceBatches\n\t/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:228\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/elderberry.(*ProcessorL1SequenceBatchesElderberry).Process\n\t/src/synchronizer/actions/elderberry/processor_l1_sequence_batches.go:73\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:606\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processUnsafe\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:252\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processIncommingRollupInfoData\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:195\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).step\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:109\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).Start\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:83\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer.func1\n\t/src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:147"} {"level":"error","ts":1710619147.3161047,"caller":"state/batchV2.go:300","msg":"error executor ProcessBatchV2: rpc error: code = ResourceExhausted desc = ","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*State).sendBatchRequestToExecutorV2\n\t/src/state/batchV2.go:300\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).processBatchV2\n\t/src/state/batchV2.go:263\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessAndStoreClosedBatchV2\n\t/src/state/batchV2.go:386\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).ProcessSequenceBatches\n\t/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:228\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/elderberry.(*ProcessorL1SequenceBatchesElderberry).Process\n\t/src/synchronizer/actions/elderberry/processor_l1_sequence_batches.go:73\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:606\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processUnsafe\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:252\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processIncommingRollupInfoData\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:195\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).step\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:109\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).Start\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:83\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer.func1\n\t/src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:147"} {"level":"error","ts":1710619147.3161333,"caller":"state/batchV2.go:301","msg":"error executor ProcessBatchV2 response: ","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*State).sendBatchRequestToExecutorV2\n\t/src/state/batchV2.go:301\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).processBatchV2\n\t/src/state/batchV2.go:263\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessAndStoreClosedBatchV2\n\t/src/state/batchV2.go:386\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).ProcessSequenceBatches\n\t/src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:228\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/elderberry.(*ProcessorL1SequenceBatchesElderberry).Process\n\t/src/synchronizer/actions/elderberry/processor_l1_sequence_batches.go:73\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:606\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processUnsafe\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:252\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processIncommingRollupInfoData\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:195\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).step\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:109\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).Start\n\t/src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:83\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer.func1\n\t/src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:147"} panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x124 pc=0x1264f8e] goroutine 508 [running]: github.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessAndStoreClosedBatchV2(0xc0008c6000, {0x1d5e018, 0xc000ff24b0}, {0x1e8173, {0x14, 0x8e, 0xe7, 0xda, 0xf1, 0x65, ...}, ...}, ...) /src/state/batchV2.go:387 +0x4ae github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1SequenceBatchesEtrog).ProcessSequenceBatches(0xc0008ad260, {0x1d5e018, 0xc000ff24b0}, {0xc00046a200, 0x4, 0xdd86?}, 0x128a8f1, {0x0, 0xedd867de2, 0x2ac7e20}, ...) /src/synchronizer/actions/etrog/processor_l1_sequence_batches.go:228 +0x2d46 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/elderberry.(*ProcessorL1SequenceBatchesElderberry).Process(0xc0001ff9c0, {0x1d5e018, 0xc000ff24b0}, {{0x18d26ee?, 0x74e2dc?}, 0xc000bc5580?}, 0xc0004b5810, {0x1d69480, 0xc0015f0c48}) /src/synchronizer/actions/elderberry/processor_l1_sequence_batches.go:73 +0x1b8 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process(0x6?, {0x1d5e018, 0xc000ff24b0}, 0xc0015f0c48?, {{0x18d26ee?, 0x0?}, 0x0?}, 0xc0004b5810, {0x1d69480, 0xc0015f0c48}) /src/synchronizer/actions/processor_manager/processor_manager.go:66 +0x1d4 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange(0xc0001a6000, {0xc0004b5000, 0x19, 0x1?}, 0xc0017f1be8?) /src/synchronizer/synchronizer.go:606 +0x925 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processUnsafe(0xc0007f9130, {{0x128a83b, 0x128ac23}, {0xc0004b5000, 0x19, 0x23}, 0xc0004ef4a0, 0x0, 0x0}) /src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:252 +0x5c2 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).processIncommingRollupInfoData(0xc0007f9130, {{0x128a83b, 0x128ac23}, {0xc0004b5000, 0x19, 0x23}, 0xc0004ef4a0, 0x0, 0x0}) /src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:195 +0x4b3 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).step(0xc0007f9130) /src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:109 +0x173 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*l1RollupInfoConsumer).Start(0xc0007f9130, {0x1d5e018?, 0xc000ff28c0?}, 0x0?) /src/synchronizer/l1_parallel_sync/l1_rollup_info_consumer.go:83 +0x168 github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer.func1() /src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:147 +0xbe created by github.com/0xPolygonHermez/zkevm-node/synchronizer/l1_parallel_sync.(*L1SyncOrchestration).launchConsumer in goroutine 262 /src/synchronizer/l1_parallel_sync/l1_sync_orchestration.go:144 +0x169 ```

Restarted again, synced for 5 minutes, then paniced again.

Something is happening during 5 days in the chain. Both of my nodes work sync a few batches and then face with error on trusted batch

"level":"error","ts":1710792658.8007498,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[2000187/2000190] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10866454 in the database 0x7088893fea67af8d094b8af2f16171c124f59c6cd000f92c5663bac8130051d3 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"}
{"level":"error","ts":1710792658.8009229,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[2000187/2000190] mode full: error processing trusted batch. Error: last L2Block 10866454 in the database 0x7088893fea67af8d094b8af2f16171c124f59c6cd000f92c5663bac8130051d3 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"}
{"level":"error","ts":1710792658.801048,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[2000187/2000190] error processing trusted batch 2000187: last L2Block 10866454 in the database 0x7088893fea67af8d094b8af2f16171c124f59c6cd000f92c5663bac8130051d3 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":1,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"}
{"level":"warn","ts":1710792658.8012958,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10866454 in the database 0x7088893fea67af8d094b8af2f16171c124f59c6cd000f92c5663bac8130051d3 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710792659.8054812,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 2000186, latestSyncedBatch: 2000186, lastVerifiedBatchNumber: 2000186","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710792659.805522,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710792659.8055477,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710792659.8377905,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:2000186 syncTrustedState:2000190 (max Batch on network: 2000190)","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710792660.0883129,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000186/2000190] mode nothing:  Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 2000186","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710792660.0893412,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=1876/1848, proverID=448e493f-ce37-4493-bee4-933d92c0c6bd/448e493f-ce37-4493-bee4-933d92c0c6bd","pid":1,"version":"v0.6.2"}
{"level":"info","ts":1710792660.0893958,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 1848, executor have write 1876","pid":1,"version":"v0.6.2"}

node logs

```sh {"level":"info","ts":1710792617.1012294,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19463938, BatchNumber: 2000179, isTrusted: true","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1013656,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19463938, BatchNumber: 2000180, isTrusted: true","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1014936,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19463938, BatchNumber: 2000181, isTrusted: true","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.101585,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19463938, BatchNumber: 2000182, isTrusted: true","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1016762,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19463938, BatchNumber: 2000183, isTrusted: true","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1017666,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19463938, BatchNumber: 2000184, isTrusted: true","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1018527,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19463938, BatchNumber: 2000185, isTrusted: true","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1019368,"caller":"incaberry/processor_l1_verify_batch.go:94","msg":"processVerifyBatches: Storing verifiedB. BlockNumber: 19463938, BatchNumber: 2000186, isTrusted: true","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1029084,"caller":"etrog/processor_l1_info_tree_update.go:52","msg":"L1InfoTree(etrog) stored. BlockNumber: 19463938,GER:0xc507e13d476057c0972ce656b9fb9601c8d1a824d9b2a67dea3c2e707f568cc2 L1InfoTreeIndex: 4334 L1InfoRoot:0x70a0fe5cbbdcd295dec5ef2914271612839c0a68ee950a86cee56424218d5969","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1030514,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:143","msg":"consumer: received a fullSync but still have 6 items in channel to process, so not stopping consumer","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.103064,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:119","msg":"consumer: processed ControlData[action:eventIsFullySynced param:19463938]. Result: %!s()","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1030786,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:183","msg":"consumer: processing rollupInfo #2: range:[19463939, latest] num_blocks [0] highest_block [19463938] statistics:wasted_time_waiting_for_data [0s] last_process_time [50.812318652s] block_per_second [0.465222]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1031046,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:165","msg":"consumer: Verified previous block 19463938 not the same: OK","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1031127,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:236","msg":"consumer: Empty block range: [19463939, latest]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1031215,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:143","msg":"consumer: received a fullSync but still have 4 items in channel to process, so not stopping consumer","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.103127,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:119","msg":"consumer: processed ControlData[action:eventIsFullySynced param:19463938]. Result: %!s()","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1031373,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:183","msg":"consumer: processing rollupInfo #3: range:[19463939, latest] num_blocks [0] highest_block [19463938] statistics:wasted_time_waiting_for_data [0s] last_process_time [33.63µs] block_per_second [0.465222]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.103145,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:236","msg":"consumer: Empty block range: [19463939, latest]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1031504,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:143","msg":"consumer: received a fullSync but still have 2 items in channel to process, so not stopping consumer","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1031556,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:119","msg":"consumer: processed ControlData[action:eventIsFullySynced param:19463938]. Result: %!s()","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1031618,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:183","msg":"consumer: processing rollupInfo #4: range:[19463939, latest] num_blocks [0] highest_block [19463938] statistics:wasted_time_waiting_for_data [0s] last_process_time [5µs] block_per_second [0.465222]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.103169,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:236","msg":"consumer: Empty block range: [19463939, latest]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1031759,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:136","msg":"consumer: received a fullSync and nothing pending in channel to process, so stopping consumer. lastBlock: 19463938","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1031828,"caller":"l1_parallel_sync/l1_rollup_info_consumer.go:119","msg":"consumer: processed ControlData[action:eventIsFullySynced param:19463938]. Result: consumer:stopped because is synchronized","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1031888,"caller":"l1_parallel_sync/l1_sync_orchestration.go:154","msg":"orchestration: consumer finished","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.103204,"caller":"l1_parallel_sync/l1_sync_orchestration.go:176","msg":"orchestration: consumer has finished. No error","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1032135,"caller":"l1_parallel_sync/l1_sync_orchestration.go:186","msg":"orchestration: finished L1 sync orchestration With LastBlock. Last block synced: 19463938 err:nil","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1032212,"caller":"synchronizer/synchronizer.go:396","msg":"L1 state fully synchronized","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1055918,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 2000186, latestSyncedBatch: 2000186, lastVerifiedBatchNumber: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1056294,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1056426,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.1807275,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:2000186 syncTrustedState:2000190 (max Batch on network: 2000190)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.2779238,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:19463962 -> new block:19463963","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.277993,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:483","msg":"producer: New last block on L1: 19463963 -> fullRange: [19463938, 19463963] extendedRange: [19463963, 19463963]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.2780402,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 1 of 10 ) result: [inFilter:0 + inChannel:0 > maximum:25? ==> allow new req] segment [19463939, latest]/UNSAFE -> [LAUNCHED] [NoNextRange] ","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.2780619,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [synchronized] to [working]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.2964413,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [18.322742ms] err: [] ] result:[ blockRange: [19463939, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true]]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.2964861,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:597","msg":"producer: sendind data to consumer: [0/1] -> range:[[19463939, latest]] Sending results [data] to consumer:data: blockRange: [19463939, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true] NO_CTRL ","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.2965057,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [working] to [synchronized]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.2965117,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:270","msg":"producer: send a message to consumer to indicate that we are synchronized. highestBlockRequested:19463938","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.6102264,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000186/2000190] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792617.8627837,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000187/2000190] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792620.4758184,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 17","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792620.5578904,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[2000187/2000190] mode full: Batch 2000187: batchl2data len:13165 processed and stored: l2block[10866300-10866454] txs[33] oldStateRoot: 0xf2cef2c9154100b34f6d6fefd8755b0381de0cd270eecc4a43534460d8e17f72 -> newStateRoot:0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792620.5600266,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0x9bc8b5669bd61330de29db596f59ccb5f6cfdaedaa720d18a4189081db839086 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b 0x7b91061444e7cf855f44f10d254a026d81e9a1c168f5d103b13ec4aa7a93977d 0x5df2455d1dc639d26be74c6a34359d2573d64d32070648e3a855812c9ec1c166 [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 32 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 2 0 0 0 0 0 64 4 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 128 0 0 64 0 0 0 0 4 0 0 0 0 0 0 0 0 16 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 68 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc001026720 1562 10866454 1125899906842624 150540 1710791500 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc000f19020 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 [{0x686908d8dfda073f238203ee48b9a168e13c914a38ed0e94d7913a3539de170d }] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0x3657ff1e9ecbe741ec2a5f4d0f919abe5e68d42f50bcef9828b34a2ba3b64177}","pid":7,"version":"v0.6.2"} {"level":"error","ts":1710792620.5601177,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[2000187/2000190] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792620.5601795,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[2000187/2000190] mode full: error processing trusted batch. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792620.5602303,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[2000187/2000190] error processing trusted batch 2000187: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710792620.560363,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792621.5633528,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 2000186, latestSyncedBatch: 2000186, lastVerifiedBatchNumber: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792621.5633912,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792621.5634017,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792621.6026497,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:2000186 syncTrustedState:2000190 (max Batch on network: 2000190)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792622.0424335,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000186/2000190] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792622.0483282,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=17/17, proverID=d7a42064-433c-4c4a-90dc-7d393e56f54e/d7a42064-433c-4c4a-90dc-7d393e56f54e","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792622.048366,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 17, executor have write 17","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792622.2757773,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000187/2000190] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792624.0692945,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 18","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792624.1457298,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[2000187/2000190] mode full: Batch 2000187: batchl2data len:13165 processed and stored: l2block[10866300-10866454] txs[33] oldStateRoot: 0xf2cef2c9154100b34f6d6fefd8755b0381de0cd270eecc4a43534460d8e17f72 -> newStateRoot:0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792624.1477618,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0x9bc8b5669bd61330de29db596f59ccb5f6cfdaedaa720d18a4189081db839086 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b 0x7b91061444e7cf855f44f10d254a026d81e9a1c168f5d103b13ec4aa7a93977d 0x5df2455d1dc639d26be74c6a34359d2573d64d32070648e3a855812c9ec1c166 [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 32 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 2 0 0 0 0 0 64 4 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 128 0 0 64 0 0 0 0 4 0 0 0 0 0 0 0 0 16 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 68 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc0007f72c8 1562 10866454 1125899906842624 150540 1710791500 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc000634ca8 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 [{0x686908d8dfda073f238203ee48b9a168e13c914a38ed0e94d7913a3539de170d }] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0x3657ff1e9ecbe741ec2a5f4d0f919abe5e68d42f50bcef9828b34a2ba3b64177}","pid":7,"version":"v0.6.2"} {"level":"error","ts":1710792624.1478457,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[2000187/2000190] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792624.1479015,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[2000187/2000190] mode full: error processing trusted batch. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792624.1479516,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[2000187/2000190] error processing trusted batch 2000187: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710792624.1480827,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792625.1510174,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 2000186, latestSyncedBatch: 2000186, lastVerifiedBatchNumber: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792625.151048,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792625.1510582,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792625.181533,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:2000186 syncTrustedState:2000190 (max Batch on network: 2000190)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792625.5566998,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000186/2000190] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792625.5677297,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=18/18, proverID=d7a42064-433c-4c4a-90dc-7d393e56f54e/d7a42064-433c-4c4a-90dc-7d393e56f54e","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792625.567748,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 18, executor have write 18","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792625.7607324,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000187/2000190] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.3170264,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:19463963 -> new block:19463964","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.3170724,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:483","msg":"producer: New last block on L1: 19463964 -> fullRange: [19463938, 19463964] extendedRange: [19463964, 19463964]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.3171,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:549","msg":"producer: launch_worker: num of launched workers: 1 ( working: 1 of 10 ) result: [inFilter:0 + inChannel:2 > maximum:25? ==> allow new req] segment [19463939, latest]/UNSAFE -> [LAUNCHED] [NoNextRange] ","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.317114,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [synchronized] to [working]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.334308,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:577","msg":"producer: Received responseRollupInfoByBlockRange: generic:[typeOfRequest: [rollup] duration: [17.148872ms] err: [] ] result:[ blockRange: [19463939, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true]]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.3343863,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:597","msg":"producer: sendind data to consumer: [0/1] -> range:[[19463939, latest]] Sending results [data] to consumer:data: blockRange: [19463939, latest] len_blocks: [0] len_order:[0] lastBlockOfRangeSet [false] previousBlockSet [true] NO_CTRL ","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.3344135,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:267","msg":"producer: Status changed from [working] to [synchronized]","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.334427,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:270","msg":"producer: send a message to consumer to indicate that we are synchronized. highestBlockRequested:19463938","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.5949547,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 19","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.6674488,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[2000187/2000190] mode full: Batch 2000187: batchl2data len:13165 processed and stored: l2block[10866300-10866454] txs[33] oldStateRoot: 0xf2cef2c9154100b34f6d6fefd8755b0381de0cd270eecc4a43534460d8e17f72 -> newStateRoot:0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792627.6696718,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0x9bc8b5669bd61330de29db596f59ccb5f6cfdaedaa720d18a4189081db839086 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b 0x7b91061444e7cf855f44f10d254a026d81e9a1c168f5d103b13ec4aa7a93977d 0x5df2455d1dc639d26be74c6a34359d2573d64d32070648e3a855812c9ec1c166 [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 32 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 2 0 0 0 0 0 64 4 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 128 0 0 64 0 0 0 0 4 0 0 0 0 0 0 0 0 16 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 68 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc0009cd190 1562 10866454 1125899906842624 150540 1710791500 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc00102f038 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 [{0x686908d8dfda073f238203ee48b9a168e13c914a38ed0e94d7913a3539de170d }] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0x3657ff1e9ecbe741ec2a5f4d0f919abe5e68d42f50bcef9828b34a2ba3b64177}","pid":7,"version":"v0.6.2"} {"level":"error","ts":1710792627.6697848,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[2000187/2000190] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792627.669852,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[2000187/2000190] mode full: error processing trusted batch. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792627.6699245,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[2000187/2000190] error processing trusted batch 2000187: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710792627.6700857,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792628.774233,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 2000186, latestSyncedBatch: 2000186, lastVerifiedBatchNumber: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792628.774294,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792628.77431,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792628.809311,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:2000186 syncTrustedState:2000190 (max Batch on network: 2000190)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792629.179697,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000186/2000190] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792629.1843405,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=19/19, proverID=d7a42064-433c-4c4a-90dc-7d393e56f54e/d7a42064-433c-4c4a-90dc-7d393e56f54e","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792629.1844015,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 19, executor have write 19","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792629.3814235,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000187/2000190] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792631.3722281,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 20","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792631.446632,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[2000187/2000190] mode full: Batch 2000187: batchl2data len:13165 processed and stored: l2block[10866300-10866454] txs[33] oldStateRoot: 0xf2cef2c9154100b34f6d6fefd8755b0381de0cd270eecc4a43534460d8e17f72 -> newStateRoot:0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792631.4501367,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0x9bc8b5669bd61330de29db596f59ccb5f6cfdaedaa720d18a4189081db839086 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b 0x7b91061444e7cf855f44f10d254a026d81e9a1c168f5d103b13ec4aa7a93977d 0x5df2455d1dc639d26be74c6a34359d2573d64d32070648e3a855812c9ec1c166 [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 32 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 2 0 0 0 0 0 64 4 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 128 0 0 64 0 0 0 0 4 0 0 0 0 0 0 0 0 16 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 68 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc0007e2498 1562 10866454 1125899906842624 150540 1710791500 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc0006350b0 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 [{0x686908d8dfda073f238203ee48b9a168e13c914a38ed0e94d7913a3539de170d }] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0x3657ff1e9ecbe741ec2a5f4d0f919abe5e68d42f50bcef9828b34a2ba3b64177}","pid":7,"version":"v0.6.2"} {"level":"error","ts":1710792631.4503455,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[2000187/2000190] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792631.450418,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[2000187/2000190] mode full: error processing trusted batch. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792631.4505112,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[2000187/2000190] error processing trusted batch 2000187: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710792631.4507096,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792632.5066087,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 2000186, latestSyncedBatch: 2000186, lastVerifiedBatchNumber: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792632.5066614,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792632.5066738,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792632.5349352,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:2000186 syncTrustedState:2000190 (max Batch on network: 2000190)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792632.958036,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000186/2000190] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792632.9640834,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=20/20, proverID=d7a42064-433c-4c4a-90dc-7d393e56f54e/d7a42064-433c-4c4a-90dc-7d393e56f54e","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792632.964119,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 20, executor have write 20","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792633.1954782,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000187/2000190] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792635.114236,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 21","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792635.184111,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[2000187/2000190] mode full: Batch 2000187: batchl2data len:13165 processed and stored: l2block[10866300-10866454] txs[33] oldStateRoot: 0xf2cef2c9154100b34f6d6fefd8755b0381de0cd270eecc4a43534460d8e17f72 -> newStateRoot:0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792635.1857245,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0x9bc8b5669bd61330de29db596f59ccb5f6cfdaedaa720d18a4189081db839086 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b 0x7b91061444e7cf855f44f10d254a026d81e9a1c168f5d103b13ec4aa7a93977d 0x5df2455d1dc639d26be74c6a34359d2573d64d32070648e3a855812c9ec1c166 [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 32 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 2 0 0 0 0 0 64 4 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 128 0 0 64 0 0 0 0 4 0 0 0 0 0 0 0 0 16 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 68 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc00139c548 1562 10866454 1125899906842624 150540 1710791500 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc0004fd020 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 [{0x686908d8dfda073f238203ee48b9a168e13c914a38ed0e94d7913a3539de170d }] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0x3657ff1e9ecbe741ec2a5f4d0f919abe5e68d42f50bcef9828b34a2ba3b64177}","pid":7,"version":"v0.6.2"} {"level":"error","ts":1710792635.18583,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[2000187/2000190] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792635.1858912,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[2000187/2000190] mode full: error processing trusted batch. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792635.1859462,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[2000187/2000190] error processing trusted batch 2000187: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710792635.1860652,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792636.1890166,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 2000186, latestSyncedBatch: 2000186, lastVerifiedBatchNumber: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792636.1890528,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792636.1890633,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792636.2346725,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:2000186 syncTrustedState:2000190 (max Batch on network: 2000190)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792636.429985,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000186/2000190] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792636.4356375,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=21/21, proverID=d7a42064-433c-4c4a-90dc-7d393e56f54e/d7a42064-433c-4c4a-90dc-7d393e56f54e","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792636.4356525,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 21, executor have write 21","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792636.8105602,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000187/2000190] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792637.3536446,"caller":"l1_parallel_sync/l1_rollup_info_producer.go:570","msg":"producer: Need a new value for Last Block On L1, doing the request old_block:19463964 -> new block:19463964","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792638.7713845,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 22","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792638.8515096,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[2000187/2000190] mode full: Batch 2000187: batchl2data len:13165 processed and stored: l2block[10866300-10866454] txs[33] oldStateRoot: 0xf2cef2c9154100b34f6d6fefd8755b0381de0cd270eecc4a43534460d8e17f72 -> newStateRoot:0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792638.853652,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0x9bc8b5669bd61330de29db596f59ccb5f6cfdaedaa720d18a4189081db839086 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b 0x7b91061444e7cf855f44f10d254a026d81e9a1c168f5d103b13ec4aa7a93977d 0x5df2455d1dc639d26be74c6a34359d2573d64d32070648e3a855812c9ec1c166 [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 32 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 2 0 0 0 0 0 64 4 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 128 0 0 64 0 0 0 0 4 0 0 0 0 0 0 0 0 16 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 68 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc001705910 1562 10866454 1125899906842624 150540 1710791500 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc00097d188 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 [{0x686908d8dfda073f238203ee48b9a168e13c914a38ed0e94d7913a3539de170d }] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0x3657ff1e9ecbe741ec2a5f4d0f919abe5e68d42f50bcef9828b34a2ba3b64177}","pid":7,"version":"v0.6.2"} {"level":"error","ts":1710792638.853794,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[2000187/2000190] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792638.8538618,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[2000187/2000190] mode full: error processing trusted batch. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792638.8539162,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[2000187/2000190] error processing trusted batch 2000187: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710792638.8540792,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792639.856708,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 2000186, latestSyncedBatch: 2000186, lastVerifiedBatchNumber: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792639.8567607,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792639.8567727,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792639.8861713,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:2000186 syncTrustedState:2000190 (max Batch on network: 2000190)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792640.1727967,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000186/2000190] mode nothing: Processing trusted batch: mode=nothing desc=exactly batches: Equal batch: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792640.1829374,"caller":"synchronizer/synchronizer.go:829","msg":"executor vs local: flushid=22/22, proverID=d7a42064-433c-4c4a-90dc-7d393e56f54e/d7a42064-433c-4c4a-90dc-7d393e56f54e","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792640.1829834,"caller":"synchronizer/synchronizer.go:854","msg":"Pending Flushid fullfiled: 22, executor have write 22","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792640.3884957,"caller":"l2_shared/processor_trusted_batch_sync.go:235","msg":"syncTrustedState: batch[2000187/2000190] mode full: Processing trusted batch: mode=full desc=Batch is not on database, so is the first time we process it","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792642.125472,"caller":"synchronizer/synchronizer.go:772","msg":"pending flushID: 23","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792642.2011008,"caller":"l2_sync_etrog/executor_trusted_batch_sync.go:402","msg":"syncTrustedState: batch[2000187/2000190] mode full: Batch 2000187: batchl2data len:13165 processed and stored: l2block[10866300-10866454] txs[33] oldStateRoot: 0xf2cef2c9154100b34f6d6fefd8755b0381de0cd270eecc4a43534460d8e17f72 -> newStateRoot:0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792642.2028885,"caller":"l2_shared/post_closed_batch_check_l2block.go:51","msg":"&{0x9bc8b5669bd61330de29db596f59ccb5f6cfdaedaa720d18a4189081db839086 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 0x148Ee7dAF16574cD020aFa34CC658f8F3fbd2800 0x0ff3682d51b327bd81bc0abe4da867a84f03795e5a79bbff8383912a5d7b9c5b 0x7b91061444e7cf855f44f10d254a026d81e9a1c168f5d103b13ec4aa7a93977d 0x5df2455d1dc639d26be74c6a34359d2573d64d32070648e3a855812c9ec1c166 [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 32 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 2 0 0 0 0 0 64 4 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 128 0 0 64 0 0 0 0 4 0 0 0 0 0 0 0 0 16 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 68 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0xc000540868 1562 10866454 1125899906842624 150540 1710791500 [] 0x0000000000000000000000000000000000000000000000000000000000000000 0xc00097cbd0 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 [{0x686908d8dfda073f238203ee48b9a168e13c914a38ed0e94d7913a3539de170d }] [] 0x0000000000000000000000000000000000000000000000000000000000000000 0x3657ff1e9ecbe741ec2a5f4d0f919abe5e68d42f50bcef9828b34a2ba3b64177}","pid":7,"version":"v0.6.2"} {"level":"error","ts":1710792642.203,"caller":"l2_shared/processor_trusted_batch_sync.go:262","msg":"syncTrustedState: batch[2000187/2000190] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different, string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ExecuteProcessBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:262\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:190\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792642.203061,"caller":"l2_shared/processor_trusted_batch_sync.go:192","msg":"syncTrustedState: batch[2000187/2000190] mode full: error processing trusted batch. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*ProcessorTrustedBatchSync).ProcessTrustedBatch\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_sync.go:192\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:136\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"error","ts":1710792642.2031136,"caller":"l2_shared/trusted_batches_retrieve.go:139","msg":"syncTrustedState: batch[2000187/2000190] error processing trusted batch 2000187: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom()\n/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState()\n/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67 github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState()\n/src/synchronizer/synchronizer.go:648 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState()\n/src/synchronizer/synchronizer.go:337 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:319 main.runSynchronizer()\n)","pid":7,"version":"v0.6.2","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).syncTrustedBatchesToFrom\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:139\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*TrustedBatchesRetrieve).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/trusted_batches_retrieve.go:102\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/l2_sync/l2_shared.(*SyncTrustedStateExecutorSelector).SyncTrustedState\n\t/src/synchronizer/l2_sync/l2_shared/processor_trusted_batch_selector.go:67\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncTrustedState\n\t/src/synchronizer/synchronizer.go:648\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:337\nmain.runSynchronizer\n\t/src/cmd/run.go:319"} {"level":"warn","ts":1710792642.203254,"caller":"synchronizer/synchronizer.go:340","msg":"error syncing trusted state. Error: last L2Block 10866454 in the database 0xb6def1b9986a1c4d855d25cbd709b926f7b0ed6ab3a7f0f4946759d0bf61abf0 and the trusted batch 0xff483ae382c7c22adcddded34b9b50f57f7a7213871b4a66a84f2946584d79f8 are different","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792643.2498512,"caller":"synchronizer/synchronizer.go:329","msg":"latestSequencedBatchNumber: 2000186, latestSyncedBatch: 2000186, lastVerifiedBatchNumber: 2000186","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792643.2499163,"caller":"synchronizer/synchronizer.go:336","msg":"Syncing trusted state (permissionless)","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792643.24993,"caller":"l2_shared/trusted_batches_retrieve.go:84","msg":"syncTrustedState: Getting trusted state info","pid":7,"version":"v0.6.2"} {"level":"info","ts":1710792643.2766635,"caller":"l2_shared/trusted_batches_retrieve.go:96","msg":"syncTrustedState: latestSyncedBatch:2000186 syncTrustedState:2000190 (max Batch on network: 2000190)","pid":7,"version":"v0.6.2"} ```

Both my nodes stop on the same batch:

syncTrustedState: batch[2000191/2000192] mode full: error checking post closed batch. Error: %!(EXTRA *errors.errorString=last L2Block 10867022 in the database 0x9da6ad206d8b866c686d16c3c13b925f8829ed65a69aa063147c063a797709f8 and the trusted batch 0x3bfe8030c1def56cd408e1457e4ba7dce0b1d6b6393b4f019d61c9e3fd408069 are different

Seems like a bug in the software

This still happening with 2 nodes we have

This needs to be checked as quickly as possible, as nodes sync normally if i rewind to a 1 week old db snapsot but once they reach the tip this starts happening

@alexqrid You running an obsolete node & prover version

These are the versions you need to be running: Instructions Node version: v0.6.2 Prover version: v5.0.7 Bridge version: v0.4.2

After the upgrade do the following

Connect to the state DB and run the following query: select block_num from state.l2block where block_hash = '0x5757afaf506076c3550fe769692c6d3f67a47ab6e2e4337c0350864dc9896a85'
If the query returns the value 10742146 then continue in the next step. In case the query returns no rows, report back
Stop the RPC, Synchronizer and Executor components (keep the state DB running)
Be sure node (RPC and Synchronizer) is updated to v0.6.2 (keep RPC and Synchronizer stopped)
Be sure Executor is updated to v5.0.7 (keep Executor stopped)
Connect to the state DB and run the following queries (force resync from Elderberry fork): delete from state.batch where batch_num >= 1998875; delete from state.block where block_num >= 19433234;
Start the components in this order: 6.1 Executor 6.2 Synchronizer 6.3 RPC

I've been running always the new version and also have this problem, i did procedure and it just keep node syncing for a while and then back to the same. Please acknowledge the problem and start sorting it.

Stop giving people procedures that dont solve the issue. Please solve the issue

@alexqrid You running an obsolete node & prover version

@obynonwane thanks, but unfortunately this is not true. I'm running the latest versions:

image: hermeznetwork/zkevm-node:v0.6.2
image: hermeznetwork/zkevm-prover:v5.0.7

Hi @alexqrid, I check your logs. We have introduced features to check L2Block hashes to be sure that are the same to the trustedNode. I understand that this is a bit annoying but this garantee that you have the same L2Block as Trusted node. Here you have more information about that:

V0.6.1: It checks the hash of last L2BLock after synchronize from TrustedNode (#3375)
v0.6.2: It checks the hash of L2Block after syncrhonize sequenced batches from L1: (#3442)

So, the problem is that you have in database some L2block with the wrong hash and current version of node is able to detected that. So with the right version of the node is not going to fix that because the problem was produced by an old one and is in your node database. The good new is that right now we known the problem and we can fix that:

I suggest to run this tool that will find out which is the discrepancy L2block. A example of executing it against mainnet:

./check_l2block_hashes_backwards_binary.sh https://zkevm-rpc.com/ http://localhost:8545/

This show the exactly point where the hashes from L2block start to differ. Based on this l2block you can find out the delete point for state.batch and state.block:

select virtual_batch.batch_num, virtual_batch.block_num from state.virtual_batch where batch_num=(select batch_num from state.l2block lb where block_num=$1);

@joanestebanr Appreciate your help! Can I just delete this bad blocks or it's better to resync from scratch?

Node 1

``` Comparing blocks from https://zkevm-rpc.com/ and http://zkevm-rpc:8545/ Last blocks: 10893311 10865212 distance: 28099 RANGE from 1 to 10865212 it: 1 OK block: 5432606 Range: [ 1 - 10865212] it: 2 OK block: 8148909 Range: [5432607 - 10865212] it: 3 OK block: 9507061 Range: [8148910 - 10865212] it: 4 BAD block: 10186137 Range: [9507062 - 10865212] it: 5 OK block: 9846599 Range: [9507062 - 10186136] it: 6 BAD block: 10016368 Range: [9846600 - 10186136] it: 7 BAD block: 9931483 Range: [9846600 - 10016367] it: 8 OK block: 9889041 Range: [9846600 - 9931482] it: 9 BAD block: 9910262 Range: [9889042 - 9931482] it: 10 BAD block: 9899651 Range: [9889042 - 9910261] it: 11 BAD block: 9894346 Range: [9889042 - 9899650] it: 12 OK block: 9891693 Range: [9889042 - 9894345] it: 13 OK block: 9893019 Range: [9891694 - 9894345] it: 14 BAD block: 9893682 Range: [9893020 - 9894345] it: 15 BAD block: 9893350 Range: [9893020 - 9893681] it: 16 BAD block: 9893184 Range: [9893020 - 9893349] it: 17 BAD block: 9893101 Range: [9893020 - 9893183] it: 18 OK block: 9893060 Range: [9893020 - 9893100] it: 19 BAD block: 9893080 Range: [9893061 - 9893100] it: 20 BAD block: 9893070 Range: [9893061 - 9893079] it: 21 BAD block: 9893065 Range: [9893061 - 9893069] it: 22 BAD block: 9893062 Range: [9893061 - 9893064] it: 23 BAD block: 9893061 Range: [9893061 - 9893061] last interation: 24 _ last range: 9893060 9893061 L2BLOCK 9893066 BAD "0xcf31ec7b8f1eba6f109905bdc04814fe7e7e36d615edd680c0a258f363628be5" "0x4d4421e00450027c215e9d32d799f6764132cfcd129dfc3a0b003f314f2d54c8" L2BLOCK 9893065 BAD "0x79981e6574bab16a72646687415ff2a40157d6e569f738a3764957d7f66926db" "0xcd39bc64f04010435fbf892180d1be536293e7ae1e1342eaa10b3c6939c8800a" L2BLOCK 9893064 BAD "0xaa2d10748e27dce9b27b6be1d6daffbef4474d3bad4e0c05ea170f602848e81c" "0x6c97d5d98cfd28b9ab9e4f0e732a09474460da0e8b96b31f645504987cdb574b" L2BLOCK 9893063 BAD "0xb915a183bb7dac4d6b8125d97883b2c308097922314a3fcb2c7e5029620ff063" "0xa98f9a01752a05f989fb6d80cb12faee93056292bac837a0797407b05b3896d2" L2BLOCK 9893062 BAD "0xe9f2c534c7b3350127d1467a32b7030b08ab6dd094aa86bd76b544c858ee76fb" "0xe92191d1fdf38e1e3d51506f65ddbb981a782754d9c28a0bdf3d9c36f064cbed" L2BLOCK 9893061 BAD "0x30c163c9b2cee7970fda2841aeb069a667f5dba3d08ba78d73b93dce370ce603" "0x21fe60427c0fba9a1a15899c73f01bd2e1f392bb540971692e5d8d7cb3013808" L2BLOCK 9893060 OK "0x73c8551458013f90cadd3d52aa3e9a9dacea31d9e2d91a9513b13a9d1e9bf6c7" "0x73c8551458013f90cadd3d52aa3e9a9dacea31d9e2d91a9513b13a9d1e9bf6c7" L2BLOCK 9893059 OK "0xaca88cef799d53fa0b118178bca5ae6461c3cf825ca4cae9e93eaecf176d133f" "0xaca88cef799d53fa0b118178bca5ae6461c3cf825ca4cae9e93eaecf176d133f" L2BLOCK 9893058 OK "0xf8650aa2eec1e698f46f4a17c0b07485c88e306db4d24c0f0cf669d7c5f9e6c2" "0xf8650aa2eec1e698f46f4a17c0b07485c88e306db4d24c0f0cf669d7c5f9e6c2" L2BLOCK 9893057 OK "0xbec37870a905841a571559301b75af78dfc9759355fffb8729a2745ffaf47fb4" "0xbec37870a905841a571559301b75af78dfc9759355fffb8729a2745ffaf47fb4" L2BLOCK 9893056 OK "0x6dcb68c0b7b8a5f71338f339a00ce73ba46950b7550964aa45e8e59f6d13d4ce" "0x6dcb68c0b7b8a5f71338f339a00ce73ba46950b7550964aa45e8e59f6d13d4ce" L2BLOCK 9893055 OK "0x42a46a3fc622d4ec2030966278374e2b7cd968cdd625eb542af4af9ad77cd6c6" "0x42a46a3fc622d4ec2030966278374e2b7cd968cdd625eb542af4af9ad77cd6c6" L2BLOCK 9893054 OK "0x619a7608d6ed91fcf82b15ecb5b9d35b53c0187859f06e0cd1c44051f44f1366" "0x619a7608d6ed91fcf82b15ecb5b9d35b53c0187859f06e0cd1c44051f44f1366" L2BLOCK 9893053 OK "0x43dae5bb10425b4ad6e268478343f7128c0e37e9846bba05de1eeb53f1f90f04" "0x43dae5bb10425b4ad6e268478343f7128c0e37e9846bba05de1eeb53f1f90f04" L2BLOCK 9893052 OK "0xe735fd3eea676dbf3cd4f0b83f0f682621b8e6832f3949ad127a83e2b747eea3" "0xe735fd3eea676dbf3cd4f0b83f0f682621b8e6832f3949ad127a83e2b747eea3" ```

Node 2

``` Comparing blocks from https://zkevm-rpc.com/ and http://zkevm-rpc:8545/ Last blocks: 10893396 10893099 distance: 297 RANGE from 1 to 10893099 it: 1 OK block: 5446550 Range: [ 1 - 10893099] it: 2 OK block: 8169825 Range: [5446551 - 10893099] it: 3 OK block: 9531462 Range: [8169826 - 10893099] it: 4 OK block: 10212281 Range: [9531463 - 10893099] it: 5 OK block: 10552690 Range: [10212282 - 10893099] it: 6 OK block: 10722895 Range: [10552691 - 10893099] it: 7 BAD block: 10807997 Range: [10722896 - 10893099] it: 8 BAD block: 10765446 Range: [10722896 - 10807996] it: 9 BAD block: 10744170 Range: [10722896 - 10765445] it: 10 OK block: 10733532 Range: [10722896 - 10744169] it: 11 OK block: 10738851 Range: [10733533 - 10744169] it: 12 OK block: 10741510 Range: [10738852 - 10744169] it: 13 BAD block: 10742840 Range: [10741511 - 10744169] it: 14 BAD block: 10742175 Range: [10741511 - 10742839] it: 15 OK block: 10741842 Range: [10741511 - 10742174] it: 16 OK block: 10742008 Range: [10741843 - 10742174] it: 17 OK block: 10742091 Range: [10742009 - 10742174] it: 18 OK block: 10742133 Range: [10742092 - 10742174] it: 19 BAD block: 10742154 Range: [10742134 - 10742174] it: 20 OK block: 10742143 Range: [10742134 - 10742153] it: 21 BAD block: 10742148 Range: [10742144 - 10742153] it: 22 OK block: 10742145 Range: [10742144 - 10742147] it: 23 OK block: 10742146 Range: [10742146 - 10742147] it: 24 BAD block: 10742147 Range: [10742147 - 10742147] last interation: 25 _ last range: 10742146 10742147 L2BLOCK 10742152 BAD "0x3215bb872763da05afa6c2c40c0b0103ae6d04711dd4435dee04161ed00e15b9" "0xb31a29a61cc32c47a59400813f264c53fcd5e19d2164a800947de712547c6314" L2BLOCK 10742151 BAD "0xe89634086e84a9f7bdcf910bdb0f1f1d0b0ee5aa935510549a3fc6d6a6f18665" "0xd30655b3b0674eb7a342fa4e28b52422476990281a694b22e358916e4b38eece" L2BLOCK 10742150 BAD "0x79bf418e2c68de9b88593aff65b8528772e330d83f914a37ac18b699585f8dfa" "0x276cacfea36b49831d69ec4dafc825b953f40a36fe0875d65cb1f271d9961a4d" L2BLOCK 10742149 BAD "0x928c418e761af8951e31c4ca2d12f236ec8bd878c749236fd94ac0d6cd1eb39f" "0xdfe8ab238800028a2295297e55fd8b9eba8e391b3625196a19b5a2227146a0f0" L2BLOCK 10742148 BAD "0xdb087475aa3f5d4ed72b332d98136cf5374693adf350e4f185c4bc59daa5fc48" "0xdfdf30653e02701458f68eec3557c147fd0cd4b8d14063c88c12aabf28eb602e" L2BLOCK 10742147 BAD "0x1ae1442278bbad7a8f3f8f30fd25c3527d646d954b674031cffad70e5cbbdd15" "0x90147379d21c3a956a4256b6c2a4955f28f2ecd60348e00ba279f2b610bd81a2" L2BLOCK 10742146 OK "0x5757afaf506076c3550fe769692c6d3f67a47ab6e2e4337c0350864dc9896a85" "0x5757afaf506076c3550fe769692c6d3f67a47ab6e2e4337c0350864dc9896a85" L2BLOCK 10742145 OK "0x66e24be8aea10d82ef2dbd0595c5370b4daf6fbae9260901d8d9dfec6b10db22" "0x66e24be8aea10d82ef2dbd0595c5370b4daf6fbae9260901d8d9dfec6b10db22" L2BLOCK 10742144 OK "0xd116d03055d3d9d327ff76fb58c6ab0f0a9b280ac0e4075f7b3f459788dc85a4" "0xd116d03055d3d9d327ff76fb58c6ab0f0a9b280ac0e4075f7b3f459788dc85a4" L2BLOCK 10742143 OK "0xd802afd5f31f4d5aaeb743cd1313364789accd52465fe0b44b7bb716bdc1db06" "0xd802afd5f31f4d5aaeb743cd1313364789accd52465fe0b44b7bb716bdc1db06" L2BLOCK 10742142 OK "0x4eba51dde4354041e385be08a54a0d678391a79fd6e2cc8fb1e2f683f1d7605a" "0x4eba51dde4354041e385be08a54a0d678391a79fd6e2cc8fb1e2f683f1d7605a" L2BLOCK 10742141 OK "0x19c688e57e336b22fe66c25753e83beeb5b71e1ea448c4b82fafc2509da18265" "0x19c688e57e336b22fe66c25753e83beeb5b71e1ea448c4b82fafc2509da18265" L2BLOCK 10742140 OK "0xf43ba21fefee171cd9c6e8ec28a15c3ba8af240ae84944eac02b82ef38e23a7f" "0xf43ba21fefee171cd9c6e8ec28a15c3ba8af240ae84944eac02b82ef38e23a7f" L2BLOCK 10742139 OK "0xdb9bb3ce7fab10db8f37fb6f7488c4e7508c9fb2192e75785076afa975848ca8" "0xdb9bb3ce7fab10db8f37fb6f7488c4e7508c9fb2192e75785076afa975848ca8" L2BLOCK 10742138 OK "0x1ce0f8f224b839e8cfe1705563ad946dc69001fef83de4fd07b22b0360457487" "0x1ce0f8f224b839e8cfe1705563ad946dc69001fef83de4fd07b22b0360457487" ```

We have introduced features to check L2Block hashes to be sure that are the same to the trustedNode.

Does it mean that there was a possibility of double spending, as hashes wasn't validated and checked? And the main reason why it didn't happened is the centralization among single seqouencer from the foundation?

@alexqrid In both cases I suggest to delete from where it starts to be wrong because is going to be faster :

node1 (first bad l2block 9893061) :

delete from state.batch where batch_num >= 1984821;
delete from state.block where block_num >= 19220466;

node2 (first bad l2block 10742147) :

delete from state.batch where batch_num >= 1998875;
delete from state.block where block_num >= 19433281;

Hi @yorickdowne, I check your logs and I see two problems:

First snippet: is a fail from the RPC service error syncing trusted state. Error: 502 - error code: 502 usually retrying this solve the situation
Second snippet: This is not an expected situation, for a better diagnosis I'll need the rest of the logs (info and warning also) . May be, would help the fix for the issue #3441 solved on release v.0.6.3. This fix will not change the situation synchronizing from L2 (it will keep failing) but would allow to synchronize it from L1.

@alexqrid

Does it mean that there was a possibility of double spending, as hashes wasn't validated and checked? And the main reason why it didn't happened is the centralization among single seqouencer from the foundation?

not at all!!

On the one hand there are the L2Hashes which are a representation of what happened and which obviously should match, but we have found inconsistencies when comparing with erigon (logs, comulative gas,...), but nothing in relation to the txs or state.

On the other hand there is the stateroot and this has not changed or varied, therefore there is no difference in the state only at the time of computing these L2Hash.

thanks @joanestebanr , your suggestions helped and I was able to sync my nodes. Closing the issue, as the trigger and root cause were found

Hurried a little bit) Actually node is still syncing but I tried to run suggested tool again and noticed that it displays the same blocks mismatching:

Node 2 logs

``` Comparing blocks from https://zkevm-rpc.com/ and http://zkevm-rpc:8545/ Last blocks: 11007795 10744042 distance: 263753 RANGE from 1 to 10744042 it: 1 OK block: 5372021 Range: [ 1 - 10744042] it: 2 OK block: 8058032 Range: [5372022 - 10744042] it: 3 OK block: 9401037 Range: [8058033 - 10744042] it: 4 OK block: 10072540 Range: [9401038 - 10744042] it: 5 OK block: 10408291 Range: [10072541 - 10744042] it: 6 OK block: 10576167 Range: [10408292 - 10744042] it: 7 OK block: 10660105 Range: [10576168 - 10744042] it: 8 OK block: 10702074 Range: [10660106 - 10744042] it: 9 OK block: 10723058 Range: [10702075 - 10744042] it: 10 OK block: 10733550 Range: [10723059 - 10744042] it: 11 OK block: 10738796 Range: [10733551 - 10744042] it: 12 OK block: 10741419 Range: [10738797 - 10744042] it: 13 BAD block: 10742731 Range: [10741420 - 10744042] it: 14 OK block: 10742075 Range: [10741420 - 10742730] it: 15 BAD block: 10742403 Range: [10742076 - 10742730] it: 16 BAD block: 10742239 Range: [10742076 - 10742402] it: 17 BAD block: 10742157 Range: [10742076 - 10742238] it: 18 OK block: 10742116 Range: [10742076 - 10742156] it: 19 OK block: 10742136 Range: [10742117 - 10742156] it: 20 OK block: 10742146 Range: [10742137 - 10742156] it: 21 BAD block: 10742151 Range: [10742147 - 10742156] it: 22 BAD block: 10742148 Range: [10742147 - 10742150] it: 23 BAD block: 10742147 Range: [10742147 - 10742147] last interation: 24 _ last range: 10742146 10742147 L2BLOCK 10742152 BAD "0x3215bb872763da05afa6c2c40c0b0103ae6d04711dd4435dee04161ed00e15b9" "0x596dfddf595fcc82b88c200c5a1e999e943baa25665d407386527c0d2f39f8a6" L2BLOCK 10742151 BAD "0xe89634086e84a9f7bdcf910bdb0f1f1d0b0ee5aa935510549a3fc6d6a6f18665" "0xb282ec9a6595fac208dcfd32b3fe399a5a2f957bffc881afd10bad6459d683c7" L2BLOCK 10742150 BAD "0x79bf418e2c68de9b88593aff65b8528772e330d83f914a37ac18b699585f8dfa" "0x77b81ce639318428b81d310256fa14da74768c4ed6a7e309e5f84026e5e365f4" L2BLOCK 10742149 BAD "0x928c418e761af8951e31c4ca2d12f236ec8bd878c749236fd94ac0d6cd1eb39f" "0x3c84d5b5b18fda6cadeb4286a2217fe51734ac0fb411e63a09e3b7b36f659ef1" L2BLOCK 10742148 BAD "0xdb087475aa3f5d4ed72b332d98136cf5374693adf350e4f185c4bc59daa5fc48" "0x9a96459ce3507b01ffd7dd5f5afe1955075d745de05c41eafed2b748cc8a71c6" L2BLOCK 10742147 BAD "0x1ae1442278bbad7a8f3f8f30fd25c3527d646d954b674031cffad70e5cbbdd15" "0x49eaa5ed78c19f6cdf2d7174466acfe2b091e9695ad78b329299adeff728ca7a" L2BLOCK 10742146 OK "0x5757afaf506076c3550fe769692c6d3f67a47ab6e2e4337c0350864dc9896a85" "0x5757afaf506076c3550fe769692c6d3f67a47ab6e2e4337c0350864dc9896a85" L2BLOCK 10742145 OK "0x66e24be8aea10d82ef2dbd0595c5370b4daf6fbae9260901d8d9dfec6b10db22" "0x66e24be8aea10d82ef2dbd0595c5370b4daf6fbae9260901d8d9dfec6b10db22" L2BLOCK 10742144 OK "0xd116d03055d3d9d327ff76fb58c6ab0f0a9b280ac0e4075f7b3f459788dc85a4" "0xd116d03055d3d9d327ff76fb58c6ab0f0a9b280ac0e4075f7b3f459788dc85a4" L2BLOCK 10742143 OK "0xd802afd5f31f4d5aaeb743cd1313364789accd52465fe0b44b7bb716bdc1db06" "0xd802afd5f31f4d5aaeb743cd1313364789accd52465fe0b44b7bb716bdc1db06" L2BLOCK 10742142 OK "0x4eba51dde4354041e385be08a54a0d678391a79fd6e2cc8fb1e2f683f1d7605a" "0x4eba51dde4354041e385be08a54a0d678391a79fd6e2cc8fb1e2f683f1d7605a" L2BLOCK 10742141 OK "0x19c688e57e336b22fe66c25753e83beeb5b71e1ea448c4b82fafc2509da18265" "0x19c688e57e336b22fe66c25753e83beeb5b71e1ea448c4b82fafc2509da18265" L2BLOCK 10742140 OK "0xf43ba21fefee171cd9c6e8ec28a15c3ba8af240ae84944eac02b82ef38e23a7f" "0xf43ba21fefee171cd9c6e8ec28a15c3ba8af240ae84944eac02b82ef38e23a7f" L2BLOCK 10742139 OK "0xdb9bb3ce7fab10db8f37fb6f7488c4e7508c9fb2192e75785076afa975848ca8" "0xdb9bb3ce7fab10db8f37fb6f7488c4e7508c9fb2192e75785076afa975848ca8" ```

executed deletion again:

(first bad l2block 10742147) : delete from state.batch where batch_num >= 1998875; delete from state.block where block_num >= 19433281;

Synced up a few batches, checked, and the same result again.

Hello. I also want to add that i have same issue on two of my nodes. I deleted batches and blocks as suggested, for the first time it helped, but after some time nodes stopped to sync again, and after second deletion nodes cant reach head with this error. ERROR synchronizer/synchronizer.go:626 error: checkL2block: L2BlockNumber: 10980670 localL2Block.Hash 0xf23f648827533c8df4dcd73cffcbf147839ba95e87a0afdea4e4a45896d0fd07 and trustedL2Block.Hash 0xd9f776eddee56423d0b91cdfbf500e90e80b24e8fa50899121cadbb087daac81 are different

zkEVM Node version: v0.6.4 zkEVM Prover version: v6.0.0 OS & Version: Linux container Network: Mainnet

I upgraded the version to the latest and solved the out-of-sync problem as described above, but currently the node height is stuck at 10985472 again with the following logs

{"level":"info","ts":1711357977.4481509,"caller":"synchronizer/synchronizer.go:415","msg":"L1 state fully synchronized","pid":58,"version":"v0.6.4"}
{"level":"info","ts":1711357977.4506264,"caller":"synchronizer/synchronizer.go:348","msg":"latestSequencedBatchNumber: 2002026, latestSyncedBatch: 2001448, lastVerifiedBatchNumber: 2002013","pid":58,"version":"v0.6.4"}
{"level":"info","ts":1711357977.4506507,"caller":"synchronizer/synchronizer.go:394","msg":"Syncing L1 blocks sequentially lastEthBlockSynced=19494172","pid":58,"version":"v0.6.4"}
{"level":"info","ts":1711357977.4976807,"caller":"synchronizer/synchronizer.go:519","msg":"Syncing block 19494173 of 19510534","pid":58,"version":"v0.6.4"}
{"level":"info","ts":1711357977.4977312,"caller":"synchronizer/synchronizer.go:520","msg":"Getting rollup info from block 19494173 to block 19494273","pid":58,"version":"v0.6.4"}
{"level":"error","ts":1711357977.6226282,"caller":"state/l1infotree.go:74","msg":"error add new leaf to the L1InfoTree. Error: mismatched leaf count: 4928, expected: 5341\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:217 github.com/0xPolygonHermez/zkevm-node/log.Error()\n/src/state/l1infotree.go:74 github.com/0xPolygonHermez/zkevm-node/state.(*State).AddL1InfoTreeLeaf()\n/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:47 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process()\n/src/synchronizer/actions/processor_manager/processor_manager.go:66 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process()\n/src/synchronizer/synchronizer.go:625 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()\n/src/synchronizer/synchronizer.go:533 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential()\n/src/synchronizer/synchronizer.go:395 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:318 main.runSynchronizer()\n","pid":58,"version":"v0.6.4","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*State).AddL1InfoTreeLeaf\n\t/src/state/l1infotree.go:74\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process\n\t/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:47\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:625\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential\n\t/src/synchronizer/synchronizer.go:533\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:395\nmain.runSynchronizer\n\t/src/cmd/run.go:318"}
{"level":"error","ts":1711357977.622718,"caller":"etrog/processor_l1_info_tree_update.go:49","msg":"error storing the l1InfoTree(etrog). BlockNumber: 19494192, error: mismatched leaf count: 4928, expected: 5341%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:49 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process()\n/src/synchronizer/actions/processor_manager/processor_manager.go:66 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process()\n/src/synchronizer/synchronizer.go:625 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()\n/src/synchronizer/synchronizer.go:533 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential()\n/src/synchronizer/synchronizer.go:395 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:318 main.runSynchronizer()\n)","pid":58,"version":"v0.6.4","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process\n\t/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:49\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:625\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential\n\t/src/synchronizer/synchronizer.go:533\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:395\nmain.runSynchronizer\n\t/src/cmd/run.go:318"}

zkEVM Node version: v0.6.4 zkEVM Prover version: v6.0.0 OS & Version: Linux container Network: Mainnet

I upgraded the version to the latest and solved the out-of-sync problem as described above, but currently the node height is stuck at 10985472 again with the following logs

{"level":"info","ts":1711357977.4481509,"caller":"synchronizer/synchronizer.go:415","msg":"L1 state fully synchronized","pid":58,"version":"v0.6.4"}
{"level":"info","ts":1711357977.4506264,"caller":"synchronizer/synchronizer.go:348","msg":"latestSequencedBatchNumber: 2002026, latestSyncedBatch: 2001448, lastVerifiedBatchNumber: 2002013","pid":58,"version":"v0.6.4"}
{"level":"info","ts":1711357977.4506507,"caller":"synchronizer/synchronizer.go:394","msg":"Syncing L1 blocks sequentially lastEthBlockSynced=19494172","pid":58,"version":"v0.6.4"}
{"level":"info","ts":1711357977.4976807,"caller":"synchronizer/synchronizer.go:519","msg":"Syncing block 19494173 of 19510534","pid":58,"version":"v0.6.4"}
{"level":"info","ts":1711357977.4977312,"caller":"synchronizer/synchronizer.go:520","msg":"Getting rollup info from block 19494173 to block 19494273","pid":58,"version":"v0.6.4"}
{"level":"error","ts":1711357977.6226282,"caller":"state/l1infotree.go:74","msg":"error add new leaf to the L1InfoTree. Error: mismatched leaf count: 4928, expected: 5341\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:217 github.com/0xPolygonHermez/zkevm-node/log.Error()\n/src/state/l1infotree.go:74 github.com/0xPolygonHermez/zkevm-node/state.(*State).AddL1InfoTreeLeaf()\n/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:47 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process()\n/src/synchronizer/actions/processor_manager/processor_manager.go:66 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process()\n/src/synchronizer/synchronizer.go:625 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()\n/src/synchronizer/synchronizer.go:533 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential()\n/src/synchronizer/synchronizer.go:395 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:318 main.runSynchronizer()\n","pid":58,"version":"v0.6.4","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*State).AddL1InfoTreeLeaf\n\t/src/state/l1infotree.go:74\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process\n\t/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:47\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:625\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential\n\t/src/synchronizer/synchronizer.go:533\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:395\nmain.runSynchronizer\n\t/src/cmd/run.go:318"}
{"level":"error","ts":1711357977.622718,"caller":"etrog/processor_l1_info_tree_update.go:49","msg":"error storing the l1InfoTree(etrog). BlockNumber: 19494192, error: mismatched leaf count: 4928, expected: 5341%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:49 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process()\n/src/synchronizer/actions/processor_manager/processor_manager.go:66 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process()\n/src/synchronizer/synchronizer.go:625 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()\n/src/synchronizer/synchronizer.go:533 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential()\n/src/synchronizer/synchronizer.go:395 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:318 main.runSynchronizer()\n)","pid":58,"version":"v0.6.4","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process\n\t/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:49\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:625\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential\n\t/src/synchronizer/synchronizer.go:533\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:395\nmain.runSynchronizer\n\t/src/cmd/run.go:318"}

zkEVM Node version: v0.6.4 zkEVM Prover version: v6.0.0 OS & Version: Linux container Network: Mainnet

{"level":"info","ts":1711375090.556431,"caller":"synchronizer/synchronizer.go:520","msg":"Getting rollup info from block 19494173 to block 19494273","pid":1,"version":"v0.6.4"}
{"level":"error","ts":1711375090.812974,"caller":"state/l1infotree.go:74","msg":"error add new leaf to the L1InfoTree. Error: mismatched leaf count: 4928, expected: 5341\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:217 github.com/0xPolygonHermez/zkevm-node/log.Error()\n/src/state/l1infotree.go:74 github.com/0xPolygonHermez/zkevm-node/state.(*State).AddL1InfoTreeLeaf()\n/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:47 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process()\n/src/synchronizer/actions/processor_manager/processor_manager.go:66 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process()\n/src/synchronizer/synchronizer.go:625 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()\n/src/synchronizer/synchronizer.go:533 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential()\n/src/synchronizer/synchronizer.go:395 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:318 main.runSynchronizer()\n","pid":1,"version":"v0.6.4","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*State).AddL1InfoTreeLeaf\n\t/src/state/l1infotree.go:74\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process\n\t/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:47\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:625\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential\n\t/src/synchronizer/synchronizer.go:533\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:395\nmain.runSynchronizer\n\t/src/cmd/run.go:318"}
{"level":"error","ts":1711375090.813083,"caller":"etrog/processor_l1_info_tree_update.go:49","msg":"error storing the l1InfoTree(etrog). BlockNumber: 19494192, error: mismatched leaf count: 4928, expected: 5341%!(EXTRA string=\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()\n/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:49 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process()\n/src/synchronizer/actions/processor_manager/processor_manager.go:66 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process()\n/src/synchronizer/synchronizer.go:625 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()\n/src/synchronizer/synchronizer.go:533 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential()\n/src/synchronizer/synchronizer.go:395 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:318 main.runSynchronizer()\n)","pid":1,"version":"v0.6.4","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process\n\t/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:49\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process\n\t/src/synchronizer/actions/processor_manager/processor_manager.go:66\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:625\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential\n\t/src/synchronizer/synchronizer.go:533\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:395\nmain.runSynchronizer\n\t/src/cmd/run.go:318"}
{"level":"error","ts":1711375090.813152,"caller":"synchronizer/synchronizer.go:627","msg":"error: mismatched leaf count: 4928, expected: 5341\n/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()\n/src/log/log.go:217 github.com/0xPolygonHermez/zkevm-node/log.Error()\n/src/synchronizer/synchronizer.go:627 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()\n/src/synchronizer/synchronizer.go:533 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential()\n/src/synchronizer/synchronizer.go:395 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()\n/src/cmd/run.go:318 main.runSynchronizer()\n","pid":1,"version":"v0.6.4","stacktrace":"github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange\n\t/src/synchronizer/synchronizer.go:627\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential\n\t/src/synchronizer/synchronizer.go:533\ngithub.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync\n\t/src/synchronizer/synchronizer.go:395\nmain.runSynchronizer\n\t/src/cmd/run.go:318"}
{"level":"warn","ts":1711375090.8133898,"caller":"synchronizer/synchronizer.go:399","msg":"error syncing blocks: mismatched leaf count: 4928, expected: 5341","pid":1,"version":"v0.6.4"}

I have the same problem even after doing the instructions node was running fine before this weekend issues, can you guys help @joanestebanr @obynonwane ?

Thanks a lot,

Stop the RPC, Synchronizer and Executor components (keep the state DB running)
Update node (RPC and Synchronizer) to v0.6.4 (keep RPC and Synchronizer stopped)
Update Executor to v6.0.0 (keep Executor stopped)
Connect to the state DB and run the following queries:
 delete from state.batch where batch_num >= 2001300;
 delete from state.block where block_num >= 19490457;
Start the components in this order:5.1 Executor5.2 Synchronizer5.3 RPC

Why not add a healing functionality into the binary? When a back block is detected, unwind, and proceed.

0xPolygonHermez / zkevm-node