Some problem when testing the forceBatch

[Federico2014]

System information

zkEVM Node version: v0.6.0 OS & Version: OSX Network: private testnet

I find some timestamp and proof verficatoin problems when testing the forceBatch. Here I will describe the process as explicitly as possible. Here, batches 4-5 are ground into a sequence batch, where only batch 6 is forced batch.

After successful sequencing by the on-chain contract, the problems emerged when the synchronizer executed the batch 5:

2024-06-13T18:07:39.398+0800    WARN    etrog/processor_l1_sequence_batches.go:373      Batch: 5. Different field StateRoot. 
       Virtual: 0x43d607833c18fe91a85ad6b0cbdd782148f18cbbdcef481010d6c17ef471134b, 
       Trusted: 0x6acef6cc2f3820f308578296c7afc16e2225066386590f404696fc4a6cdb5155

Then I debugged the problem, I found the executor causes it when the synchronizer executes batch 5:

processBatchResponse, err := s.executorClient.ProcessBatchV2(ctx, processBatchRequest)

The processBatchResponse has the Rom error as follows, which seems to be caused by the timestamp.

Then I analyzed the timestamp for batches 4-6 in the last three rows.

And the forced batch timestamp is :

I find the timestamp used when the synchronizer processes batch 4 and 5, is always the forced batch timestamp: 2024-06-13 09:59:36+00, which is caused by the sequenceSender: https://github.com/0xPolygonHermez/zkevm-node/blob/a39eddac16fc9623860eddce8b1f457342f3a3ba/sequencesender/sequencesender.go#L244

lastSequence.LastL2BLockTimestamp is the forceBatch timestamp because: https://github.com/0xPolygonHermez/zkevm-node/blob/a39eddac16fc9623860eddce8b1f457342f3a3ba/sequencesender/sequencesender.go#L331

The batch 4 timestamp 2024-06-13 09:41:27 is smaller than the forced batch timestamp 2024-06-13 09:59:36, so it can be synchronized successfully. But for batch 5, the timestamp is 2024-06-13 10:04:56, which is greater than the forced batch timestamp 2024-06-13 09:59:36, causing the executor romError when synchronizing.
Then I adjusted the timestamp by modifying the following code from https://github.com/0xPolygonHermez/zkevm-node/blob/a39eddac16fc9623860eddce8b1f457342f3a3ba/synchronizer/actions/elderberry/processor_l1_sequence_batches.go#L73 to

err = g.previousProcessor.ProcessSequenceBatches(ctx, l1Block.SequencedBatches[order.Pos], l1Block.BlockNumber, time.Unix(l1Block.ReceivedAt.Unix(), 0), dbTx)

then batch 5 is synchronized successfully. But for the forced batch 6, the synchronizer shows an error again, it seems the executor mistakes the force batch as normal batch, I find there is a bug here: https://github.com/0xPolygonHermez/zkevm-node/blob/a39eddac16fc9623860eddce8b1f457342f3a3ba/synchronizer/actions/etrog/processor_l1_sequence_batches.go#L157 The forcedBlockHashL1 value is not assigned yet. So I added the following code in line 151:

            var fBHL1 common.Hash = sbatch.PolygonRollupBaseEtrogBatchData.ForcedBlockHashL1
            forcedBlockHashL1 = &fBHL1

Then batch 6 is successfully synchronized.

After batches 4-6 are successfully synchronized as virtual batches, the aggregator can generate the proof for batches 4-6. But when submitting the proof to the on-chain verifier contract, it failed the verification as invalid proof.
https://github.com/0xPolygonHermez/zkevm-contracts/blob/1ad7089d04910c319a257ff4f3674ffd6fc6e64e/contracts/v2/PolygonRollupManager.sol#L1050

I don't know the reasons why the proof failed. I guess we'd better not group the normal batch and force batch together, which will avoid timestamp and proof verification failure proof. so I add the code snippet in sequenceSender module: I redeployed the new testnet and tested the forced batch, made the forced batch sequenced and proved alone, and then the whole process worked successfully.

Furthermore, after the sequencer processes the forced batch, the sequencer will show the following error and halt when processing the next batch. I don't know the reasons. After I restarted the sequencer, it will work normally. But after processing the next force batch, the error shows again.