[Sequencer] KEY_NOT_FOUND on the Executor after deploying Uniswap to integration3 network.

[mandrigin]

When running integration3 network with the legacy upstream executor, we get KEY_NOT_FOUND error trying to deploy Uniswap. That DOES NOT happen if the full witness is used. That DOES NOT happen on the empty network.

The suspicion here that it's one of the edge cases of SMT.

Info

Network: integraton3 Datadir (IPFS): QmSVyp98FQEjPifkLL6Y31BWbiUcWNpHxXiTxW86m6Dcxg

Steps to reproduce

(1) download the datadir

(2) launch the sequencer

CDK_ERIGON_SEQUENCER=1 go1.19 run ./cmd/cdk-erigon --config=/Users/mandrigin/Desktop/polygon-integration3/erigon.yaml

(3) Setup and run the test from https://github.com/0xPolygonHermez/zkevm-bench/tree/master with Uniswap deployment.

Result

Executor eventually goes to the error loop with KEY_NOT_FOUND error.

INFO[05-08|18:15:56.452] executor result                          grpcUrl=51.210.116.237:50071 batch=7978 counters="map[A:0 B:0 D:0 K:0 M:0 P:0 S:0 SHA:0]" exec-root=0x0000000000000000000000000000000000000000000000000000000000000000 our-root=0x9be807101066331d1c02e73274ac48d8e6cbe0de9736ac94e2c66d7b07cf39bf exec-old-root=0x31ca70056bfcfa3c15577f9a86cc497f9aa13357892970a61402ad712810d28d our-old-root=0x31ca70056bfcfa3c15577f9a86cc497f9aa13357892970a61402ad712810d28d blocks-count=0
EROR[05-08|18:15:56.452] executor error                           detail="MainExecutor::logError() Failed calling pHashDB->get() result=ZKR_DB_KEY_NOT_FOUND key=d34997c51ebe7d1a39bd364f7ad3b17b064679257d141248a2ff1548e4a2e672 proverRequest.result=ZKR_DB_KEY_NOT_FOUND step=0 eval=2799372 zkPC=1959 rom.line={ inFREE=1 inFREE0=0 freeInTag={} setE=1 sRD=1 fileName=storage-memory.zkasm line=213 lineStr=    $ => E          :SLOAD} uuid=0e9b1976-1f4f-41cf-a684-702c0dd8ce1c Registers: SR=31732376:f71f3b38:a455f9fa:87f8aa21:68071d91:d046a17d:a3acdef0:cec673c1 CONST=0:0:0:ffffffff:ffffffff:ffffffff:ffffffff:3 FREE=0:0:0:0:0:0:0:0 A=0:0:0:9fb2db26:51e29b57:15d0a9fc:d71e3b45:be6ffb7d B=0:0:0:0:0:0:0:3 C=0:0:0:0:0:0:0:c D=0:0:0:0:0:0:0:1 E=0:0:0:0:0:0:0:1 CTX=3 SP=0 PC=a GAS=cd405 zkPC=7a7 STEP=0 RR=55 HASHPOS=26 RCX=0 cntArith=f234 cntBinary=2d4df cntMemAlign=1 cntKeccakF=e8 cntSha256F=0 cntPoseidonG=417 cntPaddingPG=22c"
EROR[05-08|18:15:56.452] [Verifier] error handling request        batch=7978 err="error in response: MainExecutor::logError() Failed calling pHashDB->get() result=ZKR_DB_KEY_NOT_FOUND key=d34997c51ebe7d1a39bd364f7ad3b17b064679257d141248a2ff1548e4a2e672 proverRequest.result=ZKR_DB_KEY_NOT_FOUND step=0 eval=2799372 zkPC=1959 rom.line={ inFREE=1 inFREE0=0 freeInTag={} setE=1 sRD=1 fileName=storage-memory.zkasm line=213 lineStr=    $ => E          :SLOAD} uuid=0e9b1976-1f4f-41cf-a684-702c0dd8ce1c Registers: SR=31732376:f71f3b38:a455f9fa:87f8aa21:68071d91:d046a17d:a3acdef0:cec673c1 CONST=0:0:0:ffffffff:ffffffff:ffffffff:ffffffff:3 FREE=0:0:0:0:0:0:0:0 A=0:0:0:9fb2db26:51e29b57:15d0a9fc:d71e3b45:be6ffb7d B=0:0:0:0:0:0:0:3 C=0:0:0:0:0:0:0:c D=0:0:0:0:0:0:0:1 E=0:0:0:0:0:0:0:1 CTX=3 SP=0 PC=a GAS=cd405 zkPC=7a7 STEP=0 RR=55 HASHPOS=26 RCX=0 cntArith=f234 cntBinary=2d4df cntMemAlign=1 cntKeccakF=e8 cntSha256F=0 cntPoseidonG=417 cntPaddingPG=22c"

[mandrigin]

Attaching Erigon configs erigon-configs.zip

[mandrigin]

using erigon-noexec.yaml to recover the network (updates the datastream w/o checking with the executor)

[mandrigin]

logs from the Executor

20240508_151546_411587 d679221 a7ea640 witness2db() calculated stateRoot=31ca70056bfcfa3c15577f9a86cc497f9aa13357892970a61402ad712810d28d from size=52879
20240508_151546_411664 d679221 a7ea640 dataStreamBatch2batchL2Data() generated data of size=30997
20240508_151546_411696 d679221 a7ea640 ExecutorServiceImpl::ProcessStatelessBatchV2() got sequencerAddr=86cff31018e31819b4c101b8fd4208bcdccd0126 batchL2DataLength=30997 batchL2Data=0x0b0000000000000000f9...c89b28b7d358c6e91cff oldStateRoot=31ca70056bfcfa3c15577f9a86cc497f9
aa13357892970a61402ad712810d28d oldAccInputHash=0 oldBatchNum=7977 chainId=345 forkId=9 native globalExitRoot=0 timestampLimit=1715181327 from= bUpdateMerkleTree=1 bNoCounters=0 bGetKeys=0 bSkipVerifyL1InfoRoot=1 bSkipFirstChangeL2Block=0 bSkipWriteBlockInfoRoot=0 traceCon
fig=bEnabled=0,bDisableStorage=0,bDisableStack=0,bEnableMemory=0,bEnableReturnData=0,txHashToGenerateFullTrace= UUID=dc4eef32-7f8d-4bf5-b3be-f6f5c34867df gasLimit=0 l1InfoTreeData.size=1=0, stateOverride.size=0
20240508_151546_436058 d679221 a7ea640 --> MAIN_EXECUTOR_EXECUTE starting...
20240508_151551_362291 d679221 a7ea640 zkError: Database::read() requested a key that does not exist (ZKR_DB_KEY_NOT_FOUND): 8bd949723ddb0462fae859e701f245cadf24fdba9c4c4b3fc1156bb63b11d265

[mandrigin]

so the trimmed version of 7978 batch does not contain 8bd949723ddb0462fae859e701f245cadf24fdba9c4c4b3fc1156bb63b11d265

[mandrigin]

could be related to #380

[mandrigin]

Will re-upload a datadir and it should be reproducible with

{
    "jsonrpc": "2.0",
    "method": "zkevm_getBatchWitness",
    "params": [
        7978, "trimmed"
    ],
    "id": "1"
}

[mandrigin]

datadir uploaded: QmV68HjyRBFToJkU3eJGLjd3L6ba5GvescHwL6WiDnyChi

[mandrigin]

trimmed witness: QmYurQEkT4uNoM9iuy4gMoQyiK5atj7GpWR4Q7nbNdSeYK full witness: QmdEpvGnTnUAssX5MV2YxJZeouTYMaq38bzoVZscBKAcHM

[mandrigin]

A=0:0:0:9fb2db26:51e29b57:15d0a9fc:d71e3b45:be6ffb7d B=0:0:0:0:0:0:0:3 C=0:0:0:0:0:0:0:c D=0:0:0:0:0:0:0:1 E=0:0:0:0:0:0:0:1 CTX=3 SP=0 PC=a GAS=cd405 zkPC=7a

9fb2db2651e29b5715d0a9fcd71e3b45be6ffb7d

[mandrigin]

missing storage slot 0x0C at 0x9fb2db2651e29b5715d0a9fcd71e3b45be6ffb7d

[mandrigin]

seems like this account is legit in the EVM exec

GetOrNewStateObject 9fb2db2651e29b5715d0a9fcd71e3b45be6ffb7d

[mandrigin]

okay, but that is a result of SC creation