l2-geth crashes

[maurelian]

In E2E testing I'm currently getting a panic in refl2geth. This happens with everything freshly built.

rollupnode is being built from the input-deriv-test-cleanup of the specs repo. refl2geth is being built from the optimism-prototype branch.

These are the errors logged by the rollupnode:

DEBUG[01-12|16:19:27.008] fetched L1 data for driver               engine=0 input_l1=2fc9c1..cfb2cc:14 input_l2_parent=cd443a..d5e80c:13 finalized_l2=000000..000000
DEBUG[01-12|16:19:27.008] derived L2 block inputs                  engine=0 input_l1=2fc9c1..cfb2cc:14 input_l2_parent=cd443a..d5e80c:13 finalized_l2=000000..000000
ERROR[01-12|16:19:27.041] Failed to extend L2 chain with new L1 block engine=0 l1=2fc9c1..cfb2cc:14 l2=cd443a..d5e80c:13 err="failed to derive execution payload: engine failed to process forkchoice update for block derivation: websocket: close 1006 (abnormal closure): unexpected EOF"
DEBUG[01-12|16:19:27.076] finding next sync step, engine syncing   engine=0 l2=cd443a..d5e80c:13 l1=b1201c..4a5bd4:13
ERROR[01-12|16:19:27.077] Failed to find sync starting point       engine=0 err="failed to fetch L2 head: failed to retrieve head L2 block: dial tcp 127.0.0.1:9001: connect: connection refused"

This is the full logs and trace from refl2geth:

INFO [01-12|16:18:16.589] Set global gas cap                       cap=50,000,000
INFO [01-12|16:18:16.590] Allocated trie memory caches             clean=154.00MiB dirty=256.00MiB
INFO [01-12|16:18:16.590] Allocated cache and file handles         database=/Users/maurelian/temp/rollup-node-experiments-1/data_l2/geth/chaindata cache=512.00MiB handles=5120
INFO [01-12|16:18:16.843] Opened ancient database                  database=/Users/maurelian/temp/rollup-node-experiments-1/data_l2/geth/chaindata/ancient readonly=false
INFO [01-12|16:18:16.845] Initialised chain configuration          config="{ChainID: 901 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: <nil>, Berlin: 0, London: 0, Arrow Glacier: <nil>, MergeFork: 0, Engine: unknown}"
INFO [01-12|16:18:16.846] Disk storage enabled for ethash caches   dir=/Users/maurelian/temp/rollup-node-experiments-1/data_l2/geth/ethash count=3
INFO [01-12|16:18:16.846] Disk storage enabled for ethash DAGs     dir=/Users/maurelian/Library/Ethash count=2
INFO [01-12|16:18:16.846] Initialising Ethereum protocol           network=901 dbversion=<nil>
INFO [01-12|16:18:16.847] Loaded most recent local header          number=0 hash=3732a9..469fe1 td=1 age=1h9m33s
INFO [01-12|16:18:16.847] Loaded most recent local full block      number=0 hash=3732a9..469fe1 td=1 age=1h9m33s
INFO [01-12|16:18:16.847] Loaded most recent local fast block      number=0 hash=3732a9..469fe1 td=1 age=1h9m33s
WARN [01-12|16:18:16.848] Failed to load snapshot, regenerating    err="missing or corrupted snapshot"
INFO [01-12|16:18:16.848] Rebuilding state snapshot
INFO [01-12|16:18:16.849] Resuming state snapshot generation       root=3d12fa..85f8b3 accounts=0 slots=0 storage=0.00B elapsed="456.708µs"
INFO [01-12|16:18:16.850] Regenerated local transaction journal    transactions=0 accounts=0
INFO [01-12|16:18:16.851] Gasprice oracle is ignoring threshold set threshold=2
WARN [01-12|16:18:16.852] Error reading unclean shutdown markers   error="leveldb: not found"
WARN [01-12|16:18:16.852] Catalyst mode enabled                    protocol=eth
INFO [01-12|16:18:16.853] Starting peer-to-peer node               instance=Geth/v1.10.14-unstable/darwin-arm64/go1.17.2
INFO [01-12|16:18:16.874] Generated state snapshot                 accounts=261 slots=0 storage=9.76KiB elapsed=25.256ms
INFO [01-12|16:18:16.976] New local node record                    seq=1,642,022,296,974 id=2ad1d79bb0cf718c ip=127.0.0.1 udp=30304 tcp=30304
INFO [01-12|16:18:16.977] Started P2P networking                   self=enode://0f703958c65600a2fc893e14c9c95bb302095f7f5efa198fd0ecc8993e7f3a3ffff1716278c67f7ef46a8c186277ffb0ace109b62b164cecd5d38e5602995d94@127.0.0.1:30304
DEBUG[01-12|16:18:16.979] IPCs registered                          namespaces=admin,debug,web3,eth,txpool,personal,ethash,miner,net,engine
INFO [01-12|16:18:16.979] IPC endpoint opened                      url=/Users/maurelian/temp/rollup-node-experiments-1/data_l2/geth.ipc
ERROR[01-12|16:18:16.979] Unavailable modules in HTTP API list     unavailable=[consensus] available="[admin debug web3 eth txpool personal ethash miner net engine]"
ERROR[01-12|16:18:16.980] Unavailable modules in HTTP API list     unavailable=[consensus] available="[admin debug web3 eth txpool personal ethash miner net engine]"
DEBUG[01-12|16:18:16.980] Allowed origin(s) for WS RPC interface [http://localhost http://MacBook-Pro-6.local]
INFO [01-12|16:18:16.980] HTTP server started                      endpoint=127.0.0.1:9000 prefix= cors=* vhosts=localhost
INFO [01-12|16:18:16.981] WebSocket enabled                        url=ws://[::]:9001
DEBUG[01-12|16:18:18.014] Served eth_getBlockByNumber              reqid=47102 t="440.167µs"
INFO [01-12|16:18:19.309] New local node record                    seq=1,642,022,296,975 id=2ad1d79bb0cf718c ip=8.44.63.80 udp=30304 tcp=30304
DEBUG[01-12|16:18:21.041] Served eth_getBlockByNumber              reqid=47103 t="161.042µs"
DEBUG[01-12|16:18:21.689] Served eth_getBlockByNumber              conn=127.0.0.1:51923 reqid=1     t="87.667µs"
DEBUG[01-12|16:18:26.956] Served eth_getBlockByNumber              reqid=1     t="123.208µs"
DEBUG[01-12|16:18:27.038] Served eth_getBlockByNumber              reqid=47104 t="118.25µs"
DEBUG[01-12|16:18:27.038] Served eth_getBlockByNumber              reqid=2     t="105.584µs"
INFO [01-12|16:18:27.041] Setting head                             head=3732a9..469fe1
INFO [01-12|16:18:27.041] Entered PoS stage
INFO [01-12|16:18:27.041] Producing block                          parentHash=3732a9..469fe1
INFO [01-12|16:18:27.042] Created payload                          payloadid=572,113,280,859,045,131
DEBUG[01-12|16:18:27.042] Served engine_forkchoiceUpdatedV1        reqid=3     t=2.223625ms
DEBUG[01-12|16:18:27.043] Served engine_getPayloadV1               reqid=4     t="115.416µs"
INFO [01-12|16:18:27.045] Inserted block                           number=1 hash=a7802d..4c170a txs=1 elapsed=1.244ms
DEBUG[01-12|16:18:27.045] Served engine_executePayloadV1           reqid=5     t=1.594584ms
INFO [01-12|16:18:27.045] Setting head                             head=a7802d..4c170a
INFO [01-12|16:18:27.045] Set the chain head                       number=1 hash=a7802d..4c170a
DEBUG[01-12|16:18:27.045] Served engine_forkchoiceUpdatedV1        reqid=6     t="170.25µs"
DEBUG[01-12|16:18:27.081] Served eth_getBlockByNumber              reqid=7     t="217.583µs"
INFO [01-12|16:18:27.083] Setting head                             head=a7802d..4c170a
INFO [01-12|16:18:27.083] Producing block                          parentHash=a7802d..4c170a
INFO [01-12|16:18:27.083] Created payload                          payloadid=6,663,035,694,114,427,565
DEBUG[01-12|16:18:27.083] Served engine_forkchoiceUpdatedV1        reqid=8     t="240.125µs"
DEBUG[01-12|16:18:27.084] Served engine_getPayloadV1               reqid=9     t="42.125µs"
INFO [01-12|16:18:27.084] Inserted block                           number=2 hash=fea876..0ba638 txs=1 elapsed="226.959µs"
DEBUG[01-12|16:18:27.084] Served engine_executePayloadV1           reqid=10    t="402.833µs"
INFO [01-12|16:18:27.085] Setting head                             head=fea876..0ba638
INFO [01-12|16:18:27.085] Set the chain head                       number=2 hash=fea876..0ba638
DEBUG[01-12|16:18:27.085] Served engine_forkchoiceUpdatedV1        reqid=11    t="154.209µs"
DEBUG[01-12|16:18:27.120] Served eth_getBlockByNumber              reqid=12    t="139µs"
INFO [01-12|16:18:27.122] Setting head                             head=fea876..0ba638
INFO [01-12|16:18:27.122] Producing block                          parentHash=fea876..0ba638
INFO [01-12|16:18:27.122] Created payload                          payloadid=3,717,221,649,718,360,596
DEBUG[01-12|16:18:27.122] Served engine_forkchoiceUpdatedV1        reqid=13    t="242.459µs"
DEBUG[01-12|16:18:27.122] Served engine_getPayloadV1               reqid=14    t="39.875µs"
INFO [01-12|16:18:27.123] Inserted block                           number=3 hash=755dea..d284d4 txs=1 elapsed="215µs"
DEBUG[01-12|16:18:27.123] Served engine_executePayloadV1           reqid=15    t="404.667µs"
INFO [01-12|16:18:27.123] Setting head                             head=755dea..d284d4
INFO [01-12|16:18:27.123] Set the chain head                       number=3 hash=755dea..d284d4
DEBUG[01-12|16:18:27.123] Served engine_forkchoiceUpdatedV1        reqid=16    t="141.083µs"
DEBUG[01-12|16:18:27.159] Served eth_getBlockByNumber              reqid=17    t="137.75µs"
INFO [01-12|16:18:27.161] Setting head                             head=755dea..d284d4
INFO [01-12|16:18:27.161] Producing block                          parentHash=755dea..d284d4
INFO [01-12|16:18:27.162] Created payload                          payloadid=10,259,705,333,447,677,718
DEBUG[01-12|16:18:27.162] Served engine_forkchoiceUpdatedV1        reqid=18    t="249.75µs"
DEBUG[01-12|16:18:27.162] Served engine_getPayloadV1               reqid=19    t="40.208µs"
INFO [01-12|16:18:27.163] Inserted block                           number=4 hash=0391d8..b8566b txs=1 elapsed="268.667µs"
DEBUG[01-12|16:18:27.163] Served engine_executePayloadV1           reqid=20    t="466.75µs"
INFO [01-12|16:18:27.163] Setting head                             head=0391d8..b8566b
INFO [01-12|16:18:27.163] Set the chain head                       number=4 hash=0391d8..b8566b
DEBUG[01-12|16:18:27.163] Served engine_forkchoiceUpdatedV1        reqid=21    t="163.791µs"
DEBUG[01-12|16:18:32.015] Served eth_getBlockByNumber              reqid=47105 t="531.917µs"
INFO [01-12|16:18:33.006] Setting head                             head=0391d8..b8566b
INFO [01-12|16:18:33.006] Producing block                          parentHash=0391d8..b8566b
INFO [01-12|16:18:33.007] Created payload                          payloadid=10,877,485,455,770,647,953
DEBUG[01-12|16:18:33.007] Served engine_forkchoiceUpdatedV1        reqid=22    t="742.75µs"
INFO [01-12|16:18:33.007] Setting head                             head=0391d8..b8566b
INFO [01-12|16:18:33.007] Producing block                          parentHash=0391d8..b8566b
DEBUG[01-12|16:18:33.008] Served engine_getPayloadV1               reqid=23    t="94.458µs"
INFO [01-12|16:18:33.008] Created payload                          payloadid=10,877,485,455,770,647,953
DEBUG[01-12|16:18:33.008] Served engine_forkchoiceUpdatedV1        reqid=47106 t="421.875µs"
DEBUG[01-12|16:18:33.008] Served engine_getPayloadV1               reqid=47107 t="76.959µs"
INFO [01-12|16:18:33.009] Inserted block                           number=5 hash=20eb59..0b7d89 txs=1 elapsed="536.333µs"
DEBUG[01-12|16:18:33.009] Served engine_executePayloadV1           reqid=24    t=1.0775ms
INFO [01-12|16:18:33.010] Setting head                             head=20eb59..0b7d89
INFO [01-12|16:18:33.010] Inserted block                           number=5 hash=20eb59..0b7d89 txs=1 elapsed="414.916µs"
DEBUG[01-12|16:18:33.010] Served engine_executePayloadV1           reqid=47108 t=1.279ms
INFO [01-12|16:18:33.010] Set the chain head                       number=5 hash=20eb59..0b7d89
DEBUG[01-12|16:18:33.025] Served engine_forkchoiceUpdatedV1        reqid=25    t=15.455125ms
INFO [01-12|16:18:33.026] Setting head                             head=20eb59..0b7d89
DEBUG[01-12|16:18:33.026] Served engine_forkchoiceUpdatedV1        reqid=47109 t="65.75µs"
DEBUG[01-12|16:18:35.838] Served eth_getBlockByNumber              reqid=47110 t="236.667µs"
DEBUG[01-12|16:18:35.965] Served eth_getBlockByNumber              reqid=26    t="349.541µs"
INFO [01-12|16:18:39.005] Setting head                             head=20eb59..0b7d89
INFO [01-12|16:18:39.005] Producing block                          parentHash=20eb59..0b7d89
INFO [01-12|16:18:39.005] Setting head                             head=20eb59..0b7d89
INFO [01-12|16:18:39.005] Producing block                          parentHash=20eb59..0b7d89
INFO [01-12|16:18:39.005] Created payload                          payloadid=10,070,489,380,107,968,726
DEBUG[01-12|16:18:39.005] Served engine_forkchoiceUpdatedV1        reqid=47111 t="596.666µs"
INFO [01-12|16:18:39.005] Created payload                          payloadid=10,070,489,380,107,968,726
DEBUG[01-12|16:18:39.005] Served engine_forkchoiceUpdatedV1        reqid=27    t="723µs"
DEBUG[01-12|16:18:39.006] Served engine_getPayloadV1               reqid=47112 t="109.709µs"
DEBUG[01-12|16:18:39.006] Served engine_getPayloadV1               reqid=28    t="78.875µs"
INFO [01-12|16:18:39.007] Inserted block                           number=6 hash=c286aa..80182d txs=1 elapsed="441.375µs"
DEBUG[01-12|16:18:39.007] Served engine_executePayloadV1           reqid=29    t="812.833µs"
INFO [01-12|16:18:39.008] Setting head                             head=c286aa..80182d
INFO [01-12|16:18:39.008] Inserted block                           number=6 hash=c286aa..80182d txs=1 elapsed="373.125µs"
DEBUG[01-12|16:18:39.008] Served engine_executePayloadV1           reqid=47113 t=1.454333ms
INFO [01-12|16:18:39.008] Set the chain head                       number=6 hash=c286aa..80182d
DEBUG[01-12|16:18:39.008] Served engine_forkchoiceUpdatedV1        reqid=30    t="422.625µs"
INFO [01-12|16:18:39.008] Setting head                             head=c286aa..80182d
DEBUG[01-12|16:18:39.008] Served engine_forkchoiceUpdatedV1        reqid=47114 t="112.959µs"
DEBUG[01-12|16:18:41.128] Served eth_getBlockByNumber              reqid=31    t="185.167µs"
INFO [01-12|16:18:45.003] Setting head                             head=c286aa..80182d
INFO [01-12|16:18:45.003] Producing block                          parentHash=c286aa..80182d
INFO [01-12|16:18:45.003] Setting head                             head=c286aa..80182d
INFO [01-12|16:18:45.003] Producing block                          parentHash=c286aa..80182d
INFO [01-12|16:18:45.003] Created payload                          payloadid=39,805,135,060,331,689
DEBUG[01-12|16:18:45.003] Served engine_forkchoiceUpdatedV1        reqid=32    t="489.584µs"
INFO [01-12|16:18:45.003] Created payload                          payloadid=39,805,135,060,331,689
DEBUG[01-12|16:18:45.003] Served engine_forkchoiceUpdatedV1        reqid=47115 t="551.583µs"
DEBUG[01-12|16:18:45.003] Served engine_getPayloadV1               reqid=33    t="89.792µs"
DEBUG[01-12|16:18:45.003] Served engine_getPayloadV1               reqid=47116 t="83.458µs"
INFO [01-12|16:18:45.005] Inserted block                           number=7 hash=eeb9e2..6dca97 txs=1 elapsed="540.333µs"
DEBUG[01-12|16:18:45.005] Served engine_executePayloadV1           reqid=47117 t="903.667µs"
INFO [01-12|16:18:45.005] Setting head                             head=eeb9e2..6dca97
INFO [01-12|16:18:45.005] Inserted block                           number=7 hash=eeb9e2..6dca97 txs=1 elapsed="339.792µs"
DEBUG[01-12|16:18:45.005] Served engine_executePayloadV1           reqid=34    t=1.457792ms
INFO [01-12|16:18:45.006] Set the chain head                       number=7 hash=eeb9e2..6dca97
DEBUG[01-12|16:18:45.017] Served engine_forkchoiceUpdatedV1        reqid=47118 t=11.664083ms
INFO [01-12|16:18:45.006] Setting head                             head=eeb9e2..6dca97
DEBUG[01-12|16:18:45.017] Served engine_forkchoiceUpdatedV1        reqid=35    t=11.11225ms
DEBUG[01-12|16:18:45.523] Served eth_getBlockByNumber              reqid=47119 t="115.583µs"
DEBUG[01-12|16:18:45.649] Served eth_getBlockByNumber              reqid=36    t="256.584µs"
DEBUG[01-12|16:18:46.014] Served eth_getBlockByNumber              reqid=47120 t="287.917µs"
INFO [01-12|16:18:51.007] Setting head                             head=eeb9e2..6dca97
INFO [01-12|16:18:51.007] Producing block                          parentHash=eeb9e2..6dca97
INFO [01-12|16:18:51.007] Setting head                             head=eeb9e2..6dca97
INFO [01-12|16:18:51.007] Producing block                          parentHash=eeb9e2..6dca97
INFO [01-12|16:18:51.007] Created payload                          payloadid=9,752,807,030,860,038,792
INFO [01-12|16:18:51.008] Created payload                          payloadid=9,752,807,030,860,038,792
DEBUG[01-12|16:18:51.008] Served engine_forkchoiceUpdatedV1        reqid=37    t="500.458µs"
DEBUG[01-12|16:18:51.008] Served engine_forkchoiceUpdatedV1        reqid=47121 t="412.042µs"
DEBUG[01-12|16:18:51.008] Served engine_getPayloadV1               reqid=38    t="82.083µs"
DEBUG[01-12|16:18:51.008] Served engine_getPayloadV1               reqid=47122 t="97.709µs"
INFO [01-12|16:18:51.009] Inserted block                           number=8 hash=402f12..00b945 txs=1 elapsed="471.333µs"
DEBUG[01-12|16:18:51.009] Served engine_executePayloadV1           reqid=39    t="860.209µs"
INFO [01-12|16:18:51.010] Setting head                             head=402f12..00b945
INFO [01-12|16:18:51.010] Inserted block                           number=8 hash=402f12..00b945 txs=1 elapsed="423.375µs"
DEBUG[01-12|16:18:51.010] Served engine_executePayloadV1           reqid=47123 t=1.401667ms
INFO [01-12|16:18:51.010] Set the chain head                       number=8 hash=402f12..00b945
INFO [01-12|16:18:51.010] Setting head                             head=402f12..00b945
DEBUG[01-12|16:18:51.032] Served engine_forkchoiceUpdatedV1        reqid=40    t=22.685833ms
DEBUG[01-12|16:18:51.032] Served engine_forkchoiceUpdatedV1        reqid=47124 t=22.033417ms
DEBUG[01-12|16:18:55.129] Served eth_getBlockByNumber              reqid=41    t="652.792µs"
DEBUG[01-12|16:18:56.172] Served eth_getBlockByNumber              reqid=47125 t="380.625µs"
DEBUG[01-12|16:18:56.300] Served eth_getBlockByNumber              reqid=42    t="221.75µs"
INFO [01-12|16:18:57.010] Setting head                             head=402f12..00b945
INFO [01-12|16:18:57.010] Producing block                          parentHash=402f12..00b945
INFO [01-12|16:18:57.010] Setting head                             head=402f12..00b945
INFO [01-12|16:18:57.010] Producing block                          parentHash=402f12..00b945
INFO [01-12|16:18:57.010] Created payload                          payloadid=5,019,808,035,076,001,304
DEBUG[01-12|16:18:57.010] Served engine_forkchoiceUpdatedV1        reqid=47126 t="547.708µs"
INFO [01-12|16:18:57.010] Created payload                          payloadid=5,019,808,035,076,001,304
DEBUG[01-12|16:18:57.010] Served engine_forkchoiceUpdatedV1        reqid=43    t="609.625µs"
DEBUG[01-12|16:18:57.011] Served engine_getPayloadV1               reqid=47127 t="131.583µs"
DEBUG[01-12|16:18:57.011] Served engine_getPayloadV1               reqid=44    t="98.25µs"
INFO [01-12|16:18:57.013] Inserted block                           number=9 hash=cb1b52..17c13f txs=1 elapsed="869.666µs"
DEBUG[01-12|16:18:57.013] Served engine_executePayloadV1           reqid=47128 t=1.253417ms
INFO [01-12|16:18:57.013] Setting head                             head=cb1b52..17c13f
INFO [01-12|16:18:57.014] Inserted block                           number=9 hash=cb1b52..17c13f txs=1 elapsed="552.25µs"
DEBUG[01-12|16:18:57.014] Served engine_executePayloadV1           reqid=45    t=2.025042ms
INFO [01-12|16:18:57.014] Set the chain head                       number=9 hash=cb1b52..17c13f
DEBUG[01-12|16:18:57.014] Served engine_forkchoiceUpdatedV1        reqid=47129 t="484.834µs"
INFO [01-12|16:18:57.014] Setting head                             head=cb1b52..17c13f
DEBUG[01-12|16:18:57.014] Served engine_forkchoiceUpdatedV1        reqid=46    t="61.708µs"
DEBUG[01-12|16:19:00.012] Served eth_getBlockByNumber              reqid=47130 t="159.333µs"
INFO [01-12|16:19:03.004] Setting head                             head=cb1b52..17c13f
INFO [01-12|16:19:03.004] Producing block                          parentHash=cb1b52..17c13f
INFO [01-12|16:19:03.004] Setting head                             head=cb1b52..17c13f
INFO [01-12|16:19:03.004] Producing block                          parentHash=cb1b52..17c13f
INFO [01-12|16:19:03.004] Created payload                          payloadid=11,688,551,707,754,910,319
INFO [01-12|16:19:03.004] Created payload                          payloadid=11,688,551,707,754,910,319
DEBUG[01-12|16:19:03.004] Served engine_forkchoiceUpdatedV1        reqid=47131 t="256.708µs"
DEBUG[01-12|16:19:03.004] Served engine_forkchoiceUpdatedV1        reqid=47    t="257.542µs"
DEBUG[01-12|16:19:03.004] Served engine_getPayloadV1               reqid=47132 t="40.709µs"
DEBUG[01-12|16:19:03.004] Served engine_getPayloadV1               reqid=48    t="30.792µs"
INFO [01-12|16:19:03.005] Inserted block                           number=10 hash=268295..a3c07f txs=1 elapsed="194.042µs"
DEBUG[01-12|16:19:03.005] Served engine_executePayloadV1           reqid=49    t="353.416µs"
INFO [01-12|16:19:03.005] Setting head                             head=268295..a3c07f
INFO [01-12|16:19:03.006] Set the chain head                       number=10 hash=268295..a3c07f
DEBUG[01-12|16:19:03.006] Served engine_forkchoiceUpdatedV1        reqid=50    t="833.459µs"
INFO [01-12|16:19:03.006] Ignoring already processed block         number=10 hash=268295..a3c07f
DEBUG[01-12|16:19:03.006] Served engine_executePayloadV1           reqid=47133 t="92.792µs"
INFO [01-12|16:19:03.006] Setting head                             head=268295..a3c07f
DEBUG[01-12|16:19:03.006] Served engine_forkchoiceUpdatedV1        reqid=47134 t="27.166µs"
DEBUG[01-12|16:19:07.885] Served eth_getBlockByNumber              reqid=47135 t="379.583µs"
DEBUG[01-12|16:19:08.014] Served eth_getBlockByNumber              reqid=51    t="292.75µs"
INFO [01-12|16:19:09.009] Setting head                             head=268295..a3c07f
INFO [01-12|16:19:09.009] Producing block                          parentHash=268295..a3c07f
INFO [01-12|16:19:09.009] Setting head                             head=268295..a3c07f
INFO [01-12|16:19:09.009] Producing block                          parentHash=268295..a3c07f
INFO [01-12|16:19:09.009] Created payload                          payloadid=14,605,999,063,785,613,668
DEBUG[01-12|16:19:09.009] Served engine_forkchoiceUpdatedV1        reqid=47136 t="709.792µs"
INFO [01-12|16:19:09.009] Created payload                          payloadid=14,605,999,063,785,613,668
DEBUG[01-12|16:19:09.009] Served engine_forkchoiceUpdatedV1        reqid=52    t="740.375µs"
DEBUG[01-12|16:19:09.010] Served engine_getPayloadV1               reqid=47137 t="97.791µs"
DEBUG[01-12|16:19:09.010] Served engine_getPayloadV1               reqid=53    t="121.292µs"
INFO [01-12|16:19:09.012] Inserted block                           number=11 hash=fc0766..71c9b0 txs=1 elapsed="922.958µs"
DEBUG[01-12|16:19:09.012] Served engine_executePayloadV1           reqid=47138 t=1.384666ms
INFO [01-12|16:19:09.013] Setting head                             head=fc0766..71c9b0
INFO [01-12|16:19:09.013] Inserted block                           number=11 hash=fc0766..71c9b0 txs=1 elapsed="402.125µs"
DEBUG[01-12|16:19:09.013] Served engine_executePayloadV1           reqid=54    t=2.056292ms
INFO [01-12|16:19:09.013] Set the chain head                       number=11 hash=fc0766..71c9b0
DEBUG[01-12|16:19:09.013] Served engine_forkchoiceUpdatedV1        reqid=47139 t="458.875µs"
INFO [01-12|16:19:09.013] Setting head                             head=fc0766..71c9b0
DEBUG[01-12|16:19:09.020] Served engine_forkchoiceUpdatedV1        reqid=55    t=6.816541ms
DEBUG[01-12|16:19:09.128] Served eth_getBlockByNumber              reqid=56    t="208.125µs"
DEBUG[01-12|16:19:14.010] Served eth_getBlockByNumber              reqid=47140 t="380.791µs"
INFO [01-12|16:19:15.009] Setting head                             head=fc0766..71c9b0
INFO [01-12|16:19:15.009] Producing block                          parentHash=fc0766..71c9b0
INFO [01-12|16:19:15.009] Setting head                             head=fc0766..71c9b0
INFO [01-12|16:19:15.009] Producing block                          parentHash=fc0766..71c9b0
INFO [01-12|16:19:15.009] Created payload                          payloadid=5,054,402,116,750,183,497
DEBUG[01-12|16:19:15.009] Served engine_forkchoiceUpdatedV1        reqid=57    t="646.583µs"
INFO [01-12|16:19:15.009] Created payload                          payloadid=5,054,402,116,750,183,497
DEBUG[01-12|16:19:15.009] Served engine_forkchoiceUpdatedV1        reqid=47141 t="699.625µs"
DEBUG[01-12|16:19:15.010] Served engine_getPayloadV1               reqid=58    t="82.208µs"
DEBUG[01-12|16:19:15.010] Served engine_getPayloadV1               reqid=47142 t="125.083µs"
INFO [01-12|16:19:15.011] Inserted block                           number=12 hash=e1e9cf..30b766 txs=1 elapsed="512.666µs"
DEBUG[01-12|16:19:15.012] Served engine_executePayloadV1           reqid=47143 t="926.75µs"
INFO [01-12|16:19:15.012] Setting head                             head=e1e9cf..30b766
INFO [01-12|16:19:15.012] Inserted block                           number=12 hash=e1e9cf..30b766 txs=1 elapsed="483.875µs"
DEBUG[01-12|16:19:15.012] Served engine_executePayloadV1           reqid=59    t=1.653625ms
INFO [01-12|16:19:15.012] Set the chain head                       number=12 hash=e1e9cf..30b766
DEBUG[01-12|16:19:15.012] Served engine_forkchoiceUpdatedV1        reqid=47144 t="404.5µs"
INFO [01-12|16:19:15.013] Setting head                             head=e1e9cf..30b766
DEBUG[01-12|16:19:15.037] Served engine_forkchoiceUpdatedV1        reqid=60    t=23.852208ms
DEBUG[01-12|16:19:20.774] Served eth_getBlockByNumber              reqid=47145 t="483.791µs"
DEBUG[01-12|16:19:20.902] Served eth_getBlockByNumber              reqid=61    t="208.333µs"
INFO [01-12|16:19:21.007] Setting head                             head=e1e9cf..30b766
INFO [01-12|16:19:21.007] Producing block                          parentHash=e1e9cf..30b766
INFO [01-12|16:19:21.008] Setting head                             head=e1e9cf..30b766
INFO [01-12|16:19:21.008] Producing block                          parentHash=e1e9cf..30b766
INFO [01-12|16:19:21.008] Created payload                          payloadid=3,935,290,890,915,038,142
DEBUG[01-12|16:19:21.008] Served engine_forkchoiceUpdatedV1        reqid=47146 t="570.375µs"
INFO [01-12|16:19:21.008] Created payload                          payloadid=3,935,290,890,915,038,142
DEBUG[01-12|16:19:21.008] Served engine_forkchoiceUpdatedV1        reqid=62    t="298.833µs"
DEBUG[01-12|16:19:21.008] Served engine_getPayloadV1               reqid=47147 t="79.666µs"
DEBUG[01-12|16:19:21.008] Served engine_getPayloadV1               reqid=63    t="62.834µs"
INFO [01-12|16:19:21.009] Inserted block                           number=13 hash=cd443a..d5e80c txs=1 elapsed="406.208µs"
DEBUG[01-12|16:19:21.009] Served engine_executePayloadV1           reqid=47148 t="792.875µs"
INFO [01-12|16:19:21.010] Setting head                             head=cd443a..d5e80c
INFO [01-12|16:19:21.010] Inserted block                           number=13 hash=cd443a..d5e80c txs=1 elapsed="527.125µs"
DEBUG[01-12|16:19:21.010] Served engine_executePayloadV1           reqid=64    t=1.296667ms
INFO [01-12|16:19:21.010] Set the chain head                       number=13 hash=cd443a..d5e80c
DEBUG[01-12|16:19:21.010] Served engine_forkchoiceUpdatedV1        reqid=47149 t="589.875µs"
INFO [01-12|16:19:21.011] Setting head                             head=cd443a..d5e80c
DEBUG[01-12|16:19:21.011] Served engine_forkchoiceUpdatedV1        reqid=65    t="71.792µs"
DEBUG[01-12|16:19:23.127] Served eth_getBlockByNumber              reqid=66    t="598.667µs"
INFO [01-12|16:19:27.008] Setting head                             head=cd443a..d5e80c
INFO [01-12|16:19:27.008] Producing block                          parentHash=cd443a..d5e80c
INFO [01-12|16:19:27.008] Setting head                             head=cd443a..d5e80c
INFO [01-12|16:19:27.009] Producing block                          parentHash=cd443a..d5e80c
fatal error: concurrent map writes

goroutine 459 [running]:
runtime.throw({0x1013d2847, 0x15})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/panic.go:1198 +0x54 fp=0x14001485430 sp=0x14001485400 pc=0x10065a094
runtime.mapassign_fast64(0x10174b240, 0x140000fc4b0, 0xd0c21a3fe849a004)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/map_fast64.go:101 +0x30c fp=0x14001485470 sp=0x14001485430 pc=0x100633ccc
github.com/ethereum/go-ethereum/eth/catalyst.(*ConsensusAPI).ForkchoiceUpdatedV1(0x140000fc4e0, {{0xcd, 0x44, 0x3a, 0x7f, 0x4a, 0xd7, 0xab, 0xe5, 0x14, ...}, ...}, ...)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/catalyst/api.go:222 +0x2a0 fp=0x14001485540 sp=0x14001485470 pc=0x100e294b0
runtime.call256(0x14001114e10, 0x14000d101a8, 0x14000e25220, 0x98, 0x70, 0x98, 0x14001485730)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/asm_arm64.s:418 +0x7c fp=0x14001485650 sp=0x14001485540 pc=0x10068e69c
reflect.Value.call({0x14000bd4600, 0x14000d101a8, 0x13}, {0x1013bd516, 0x4}, {0x140019d0240, 0x3, 0x4})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/reflect/value.go:543 +0x584 fp=0x14001485b30 sp=0x14001485650 pc=0x1006c0884
reflect.Value.Call({0x14000bd4600, 0x14000d101a8, 0x13}, {0x140019d0240, 0x3, 0x4})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/reflect/value.go:339 +0x8c fp=0x14001485bb0 sp=0x14001485b30 pc=0x1006c012c
github.com/ethereum/go-ethereum/rpc.(*callback).call(0x14000d161e0, {0x10194d578, 0x14000e13780}, {0x140019cc160, 0x1a}, {0x140019c1e60, 0x2, 0x2})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/service.go:206 +0x3c8 fp=0x14001485cc0 sp=0x14001485bb0 pc=0x100abdf68
github.com/ethereum/go-ethereum/rpc.(*handler).runMethod(0x14000430c60, {0x10194d578, 0x14000e13780}, 0x140019d4000, 0x14000d161e0, {0x140019c1e60, 0x2, 0x2})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:389 +0x64 fp=0x14001485d30 sp=0x14001485cc0 pc=0x100ab8314
github.com/ethereum/go-ethereum/rpc.(*handler).handleCall(0x14000430c60, 0x140019c1e00, 0x140019d4000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:337 +0x258 fp=0x14001485df0 sp=0x14001485d30 pc=0x100ab7958
github.com/ethereum/go-ethereum/rpc.(*handler).handleCallMsg(0x14000430c60, 0x140019c1e00, 0x140019d4000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:298 +0x90 fp=0x14001485ee0 sp=0x14001485df0 pc=0x100ab6fe0
github.com/ethereum/go-ethereum/rpc.(*handler).handleMsg.func1(0x140019c1e00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:139 +0x3c fp=0x14001485f50 sp=0x14001485ee0 pc=0x100ab5d0c
github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc.func1(0x14000430c60, 0x14001985248)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:226 +0xd8 fp=0x14001485fc0 sp=0x14001485f50 pc=0x100ab6778
runtime.goexit()
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/asm_arm64.s:1133 +0x4 fp=0x14001485fc0 sp=0x14001485fc0 pc=0x100690364
created by github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:222 +0x60

goroutine 1 [chan receive]:
github.com/ethereum/go-ethereum/node.(*Node).Wait(...)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/node/node.go:422
main.geth(0x1400075a000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/cmd/geth/main.go:324 +0x124
gopkg.in/urfave/cli%2ev1.HandleAction({0x1016f7e60, 0x101910880}, 0x1400075a000)
    /Users/maurelian/go-path/pkg/mod/gopkg.in/urfave/cli.v1@v1.20.0/app.go:490 +0x84
gopkg.in/urfave/cli%2ev1.(*App).Run(0x140005821a0, {0x14000182ea0, 0x19, 0x1a})
    /Users/maurelian/go-path/pkg/mod/gopkg.in/urfave/cli.v1@v1.20.0/app.go:264 +0x534
main.main()
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/cmd/geth/main.go:263 +0x50

goroutine 19 [chan receive, 1 minutes]:
github.com/rjeczalik/notify.(*recursiveTree).dispatch(0x1400018e190)
    /Users/maurelian/go-path/pkg/mod/github.com/rjeczalik/notify@v0.9.1/tree_recursive.go:125 +0xac
created by github.com/rjeczalik/notify.newRecursiveTree
    /Users/maurelian/go-path/pkg/mod/github.com/rjeczalik/notify@v0.9.1/tree_recursive.go:119 +0x144

goroutine 20 [syscall, 1 minutes, locked to thread]:
github.com/rjeczalik/notify._Cfunc_CFRunLoopRun()
    _cgo_gotypes.go:229 +0x40
github.com/rjeczalik/notify.init.1.func1()
    /Users/maurelian/go-path/pkg/mod/github.com/rjeczalik/notify@v0.9.1/watcher_fsevents_cgo.go:72 +0x3c
created by github.com/rjeczalik/notify.init.1
    /Users/maurelian/go-path/pkg/mod/github.com/rjeczalik/notify@v0.9.1/watcher_fsevents_cgo.go:64 +0x48

goroutine 21 [chan receive]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x1400021cd00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:63 +0x44
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:55 +0xac

goroutine 22 [chan receive]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x1400021cd00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:63 +0x44
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:55 +0xac

goroutine 23 [chan receive]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x1400021cd00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:63 +0x44
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:55 +0xac

goroutine 24 [chan receive]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x1400021cd00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:63 +0x44
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:55 +0xac

goroutine 25 [chan receive]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x1400021cd00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:63 +0x44
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:55 +0xac

goroutine 26 [chan receive]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x1400021cd00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:63 +0x44
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:55 +0xac

goroutine 27 [chan receive]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x1400021cd00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:63 +0x44
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:55 +0xac

goroutine 28 [chan receive]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x1400021cd00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:63 +0x44
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_cacher.go:55 +0xac

goroutine 29 [chan receive]:
github.com/ethereum/go-ethereum/metrics.(*meterArbiter).tick(0x1023d6d20)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/metrics/meter.go:290 +0x60
created by github.com/ethereum/go-ethereum/metrics.NewMeterForced
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/metrics/meter.go:71 +0xd4

goroutine 30 [select]:
github.com/ethereum/go-ethereum/consensus/ethash.(*remoteSealer).loop(0x140001f00f0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/consensus/ethash/sealer.go:279 +0x154
created by github.com/ethereum/go-ethereum/consensus/ethash.startRemoteSealer
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/consensus/ethash/sealer.go:263 +0x2b8

goroutine 211 [select]:
github.com/ethereum/go-ethereum/rpc.(*websocketCodec).pingLoop(0x1400109fb30)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/websocket.go:283 +0xf4
created by github.com/ethereum/go-ethereum/rpc.newWebsocketCodec
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/websocket.go:255 +0x244

goroutine 145 [chan receive]:
github.com/ethereum/go-ethereum/rpc.(*Server).ServeCodec(0x14000bfb5c0, {0x1019539d8, 0x1400109fb30}, 0x0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/server.go:87 +0x128
github.com/ethereum/go-ethereum/rpc.(*Server).WebsocketHandler.func1({0x10193dd60, 0x14000c74d20}, 0x140010b2000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/websocket.go:64 +0xfc
net/http.HandlerFunc.ServeHTTP(0x14000bc1938, {0x10193dd60, 0x14000c74d20}, 0x140010b2000)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/http/server.go:2046 +0x40
github.com/ethereum/go-ethereum/node.(*httpServer).ServeHTTP(0x140001e5680, {0x10193dd60, 0x14000c74d20}, 0x140010b2000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/node/rpcstack.go:188 +0x154
net/http.serverHandler.ServeHTTP({0x140007196c0}, {0x10193dd60, 0x14000c74d20}, 0x140010b2000)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/http/server.go:2878 +0x444
net/http.(*conn).serve(0x140010992c0, {0x10194d620, 0x1400016d260})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/http/server.go:1929 +0xb6c
created by net/http.(*Server).Serve
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/http/server.go:3033 +0x4b8

goroutine 33 [syscall]:
os/signal.signal_recv()
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/sigqueue.go:166 +0x34
os/signal.loop()
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/os/signal/signal_unix.go:24 +0x20
created by os/signal.Notify.func1.1
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/os/signal/signal.go:151 +0x38

goroutine 7 [select, 1 minutes]:
github.com/ethereum/go-ethereum/accounts.(*Manager).update(0x14000244620)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/accounts/manager.go:134 +0xcc
created by github.com/ethereum/go-ethereum/accounts.NewManager
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/accounts/manager.go:94 +0x290

goroutine 8 [select, 1 minutes]:
github.com/ethereum/go-ethereum/accounts/keystore.(*watcher).loop(0x140005178c0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/accounts/keystore/watch.go:95 +0x458
created by github.com/ethereum/go-ethereum/accounts/keystore.(*watcher).start
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/accounts/keystore/watch.go:53 +0x60

goroutine 9 [select]:
github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).updater(0x140001f0870)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/accounts/keystore/keystore.go:209 +0x8c
created by github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).Subscribe
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/accounts/keystore/keystore.go:196 +0xe4

goroutine 10 [select]:
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0x140001f0960)
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session_util.go:189 +0x390
created by github.com/syndtr/goleveldb/leveldb.newSession
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session.go:93 +0x2ac

goroutine 82 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0x14000778000)
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:91 +0x130
created by github.com/syndtr/goleveldb/leveldb.openDB
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:148 +0x484

goroutine 83 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0x14000778000)
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_state.go:101 +0x98
created by github.com/syndtr/goleveldb/leveldb.openDB
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:149 +0x4a4

goroutine 84 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0x14000778000)
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:836 +0x550
created by github.com/syndtr/goleveldb/leveldb.openDB
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:155 +0x4f4

goroutine 85 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0x14000778000)
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:773 +0xd0
created by github.com/syndtr/goleveldb/leveldb.openDB
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:156 +0x514

goroutine 86 [select]:
github.com/ethereum/go-ethereum/ethdb/leveldb.(*Database).meter(0x1400060e000, 0xb2d05e00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/ethdb/leveldb/leveldb.go:432 +0x168c
created by github.com/ethereum/go-ethereum/ethdb/leveldb.NewCustom
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/ethdb/leveldb/leveldb.go:148 +0x9e8

goroutine 87 [select]:
github.com/ethereum/go-ethereum/core/rawdb.(*freezer).freeze(0x14000130180, {0x101960910, 0x1400060e000})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/rawdb/freezer.go:354 +0x15c
github.com/ethereum/go-ethereum/core/rawdb.NewDatabaseWithFreezer.func1(0x14000130180, {0x101960910, 0x1400060e000})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/rawdb/database.go:221 +0x38
created by github.com/ethereum/go-ethereum/core/rawdb.NewDatabaseWithFreezer
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/rawdb/database.go:220 +0x59c

goroutine 88 [select]:
github.com/ethereum/go-ethereum/consensus/ethash.(*remoteSealer).loop(0x140001f1770)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/consensus/ethash/sealer.go:279 +0x154
created by github.com/ethereum/go-ethereum/consensus/ethash.startRemoteSealer
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/consensus/ethash/sealer.go:263 +0x2b8

goroutine 89 [select]:
github.com/ethereum/go-ethereum/core.(*ChainIndexer).updateLoop(0x140001f1860)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/chain_indexer.go:311 +0xa8
created by github.com/ethereum/go-ethereum/core.NewChainIndexer
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/chain_indexer.go:119 +0x264

goroutine 90 [chan receive]:
github.com/ethereum/go-ethereum/core/state/snapshot.(*diskLayer).generate(0x140001e8b40, 0x14000054a00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/state/snapshot/generate.go:760 +0x6b4
created by github.com/ethereum/go-ethereum/core/state/snapshot.generateSnapshot
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/state/snapshot/generate.go:168 +0x328

goroutine 91 [select]:
github.com/ethereum/go-ethereum/core.(*BlockChain).updateFutureBlocks(0x1400076d000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/blockchain.go:2158 +0xf0
created by github.com/ethereum/go-ethereum/core.NewBlockChain
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/blockchain.go:384 +0x1c7c

goroutine 92 [select]:
github.com/ethereum/go-ethereum/core.(*BlockChain).maintainTxIndex(0x1400076d000, 0x0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/blockchain.go:2285 +0x1d4
created by github.com/ethereum/go-ethereum/core.NewBlockChain
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/blockchain.go:391 +0x1ccc

goroutine 93 [select]:
github.com/ethereum/go-ethereum/trie.(*Database).SaveCachePeriodically(0x14000219a00, {0x140000b8ff0, 0x46}, 0x34630b8a000, 0x14000635320)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/trie/database.go:882 +0xc0
github.com/ethereum/go-ethereum/core.NewBlockChain.func1(0x1400076d000, 0x14000219a00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/blockchain.go:404 +0x7c
created by github.com/ethereum/go-ethereum/core.NewBlockChain
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/blockchain.go:402 +0x1e24

goroutine 94 [select]:
github.com/ethereum/go-ethereum/core.(*ChainIndexer).eventLoop(0x140001f1860, 0x14000340480, 0x140000add40, {0x101939be0, 0x14000514ed0})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/chain_indexer.go:211 +0x1ec
created by github.com/ethereum/go-ethereum/core.(*ChainIndexer).Start
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/chain_indexer.go:152 +0xb8

goroutine 95 [select]:
github.com/ethereum/go-ethereum/core.(*TxPool).scheduleReorgLoop(0x14000326800)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_pool.go:1104 +0x1bc
created by github.com/ethereum/go-ethereum/core.NewTxPool
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_pool.go:312 +0x548

goroutine 96 [select]:
github.com/ethereum/go-ethereum/core.(*TxPool).loop(0x14000326800)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_pool.go:356 +0x1fc
created by github.com/ethereum/go-ethereum/core.NewTxPool
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/core/tx_pool.go:329 +0x7c4

goroutine 97 [select]:
github.com/ethereum/go-ethereum/eth/downloader.(*skeleton).startup(0x14000245960)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/downloader/skeleton.go:227 +0xa0
created by github.com/ethereum/go-ethereum/eth/downloader.newSkeleton
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/downloader/skeleton.go:213 +0x180

goroutine 98 [select]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).stateFetcher(0x140001e4300)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/downloader/statesync.go:47 +0x6c
created by github.com/ethereum/go-ethereum/eth/downloader.New
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/downloader/downloader.go:226 +0x354

goroutine 99 [select]:
github.com/ethereum/go-ethereum/miner.(*worker).mainLoop(0x14000341440)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/miner/worker.go:464 +0x2c8
created by github.com/ethereum/go-ethereum/miner.newWorker
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/miner/worker.go:233 +0x64c

goroutine 100 [select]:
github.com/ethereum/go-ethereum/miner.(*worker).newWorkLoop(0x14000341440, 0xb2d05e00)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/miner/worker.go:392 +0x1ac
created by github.com/ethereum/go-ethereum/miner.newWorker
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/miner/worker.go:234 +0x674

goroutine 101 [select]:
github.com/ethereum/go-ethereum/miner.(*worker).resultLoop(0x14000341440)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/miner/worker.go:614 +0xb0
created by github.com/ethereum/go-ethereum/miner.newWorker
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/miner/worker.go:235 +0x694

goroutine 102 [select]:
github.com/ethereum/go-ethereum/miner.(*worker).taskLoop(0x14000341440)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/miner/worker.go:575 +0xb4
created by github.com/ethereum/go-ethereum/miner.newWorker
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/miner/worker.go:236 +0x6b4

goroutine 103 [select]:
github.com/ethereum/go-ethereum/miner.(*Miner).update(0x1400044f3b0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/miner/miner.go:104 +0x19c
created by github.com/ethereum/go-ethereum/miner.New
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/miner/miner.go:82 +0x1d4

goroutine 104 [chan receive]:
github.com/ethereum/go-ethereum/eth/gasprice.NewOracle.func1(0x140000b0660, 0x140000b0600)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/gasprice/gasprice.go:122 +0x68
created by github.com/ethereum/go-ethereum/eth/gasprice.NewOracle
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/gasprice/gasprice.go:120 +0x780

goroutine 105 [select]:
github.com/ethereum/go-ethereum/eth/downloader.(*PublicDownloaderAPI).eventLoop(0x14000517060)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/downloader/api.go:63 +0x158
created by github.com/ethereum/go-ethereum/eth/downloader.NewPublicDownloaderAPI
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/downloader/api.go:49 +0xf4

goroutine 106 [select]:
github.com/ethereum/go-ethereum/eth/filters.(*EventSystem).eventLoop(0x140001ea4d0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/filters/filter_system.go:459 +0x2c0
created by github.com/ethereum/go-ethereum/eth/filters.NewEventSystem
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/filters/filter_system.go:138 +0x3c0

goroutine 107 [chan receive]:
github.com/ethereum/go-ethereum/eth/filters.(*PublicFilterAPI).timeoutLoop(0x14000055300, 0x45d964b800)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/filters/api.go:79 +0x74
created by github.com/ethereum/go-ethereum/eth/filters.NewPublicFilterAPI
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/filters/api.go:67 +0xe4

goroutine 108 [select]:
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0x140000fe3c0)
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session_util.go:189 +0x390
created by github.com/syndtr/goleveldb/leveldb.newSession
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session.go:93 +0x2ac

goroutine 38 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0x1400014c1c0)
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:91 +0x130
created by github.com/syndtr/goleveldb/leveldb.openDB
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:148 +0x484

goroutine 39 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0x1400014c1c0)
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_state.go:101 +0x98
created by github.com/syndtr/goleveldb/leveldb.openDB
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:149 +0x4a4

goroutine 40 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0x1400014c1c0)
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:836 +0x550
created by github.com/syndtr/goleveldb/leveldb.openDB
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:155 +0x4f4

goroutine 41 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0x1400014c1c0)
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:773 +0xd0
created by github.com/syndtr/goleveldb/leveldb.openDB
    /Users/maurelian/go-path/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:156 +0x514

goroutine 42 [IO wait]:
internal/poll.runtime_pollWait(0x1298a5560, 0x72)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/netpoll.go:229 +0xa4
internal/poll.(*pollDesc).wait(0x14000716798, 0x72, 0x0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x14000716780)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_unix.go:402 +0x1ec
net.(*netFD).accept(0x14000716780)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/fd_unix.go:173 +0x2c
net.(*TCPListener).accept(0x140005f72f0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/tcpsock_posix.go:140 +0x2c
net.(*TCPListener).Accept(0x140005f72f0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/tcpsock.go:262 +0x34
github.com/ethereum/go-ethereum/p2p.(*Server).listenLoop(0x14000538500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/server.go:862 +0x61c
created by github.com/ethereum/go-ethereum/p2p.(*Server).setupListening
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/server.go:680 +0x20c

goroutine 45 [select]:
github.com/ethereum/go-ethereum/p2p/discover.(*Table).loop(0x1400070ab40)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/discover/table.go:236 +0x1ec
created by github.com/ethereum/go-ethereum/p2p/discover.ListenV4
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/discover/v4_udp.go:150 +0x308

goroutine 46 [select]:
github.com/ethereum/go-ethereum/p2p/discover.(*UDPv4).loop(0x140005ed5f0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/discover/v4_udp.go:445 +0x240
created by github.com/ethereum/go-ethereum/p2p/discover.ListenV4
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/discover/v4_udp.go:153 +0x340

goroutine 47 [IO wait]:
internal/poll.runtime_pollWait(0x1298a5478, 0x72)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/netpoll.go:229 +0xa4
internal/poll.(*pollDesc).wait(0x14000716818, 0x72, 0x0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).ReadFrom(0x14000716800, {0x14000512000, 0x500, 0x500})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_unix.go:223 +0x1e0
net.(*netFD).readFrom(0x14000716800, {0x14000512000, 0x500, 0x500})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/fd_posix.go:62 +0x44
net.(*UDPConn).readFrom(0x140007066e8, {0x14000512000, 0x500, 0x500}, 0x1400195c4e0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/udpsock_posix.go:47 +0x44
net.(*UDPConn).readFromUDP(0x140007066e8, {0x14000512000, 0x500, 0x500}, 0x1400195c4e0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/udpsock.go:116 +0x50
net.(*UDPConn).ReadFromUDP(0x140007066e8, {0x14000512000, 0x500, 0x500})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/udpsock.go:108 +0x58
github.com/ethereum/go-ethereum/p2p/discover.(*UDPv4).readLoop(0x140005ed5f0, 0x0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/discover/v4_udp.go:522 +0xe4
created by github.com/ethereum/go-ethereum/p2p/discover.ListenV4
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/discover/v4_udp.go:154 +0x368

goroutine 48 [select]:
github.com/ethereum/go-ethereum/p2p/enode.(*FairMix).runSource(0x1400072e5a0, 0x1400068ea20, 0x140005eefc0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/enode/iter.go:281 +0x134
created by github.com/ethereum/go-ethereum/p2p/enode.(*FairMix).AddSource
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/enode/iter.go:180 +0x234

goroutine 49 [select]:
github.com/ethereum/go-ethereum/p2p.(*dialScheduler).readNodes(0x1400075a160, {0x10193d340, 0x1400072e5a0})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/dial.go:323 +0xf4
created by github.com/ethereum/go-ethereum/p2p.newDialScheduler
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/dial.go:180 +0x334

goroutine 114 [select]:
github.com/ethereum/go-ethereum/p2p.(*dialScheduler).loop(0x1400075a160, {0x10193d340, 0x1400072e5a0})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/dial.go:243 +0x1ec
created by github.com/ethereum/go-ethereum/p2p.newDialScheduler
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/dial.go:181 +0x364

goroutine 115 [select]:
github.com/ethereum/go-ethereum/p2p.(*Server).run(0x14000538500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/server.go:714 +0x39c
created by github.com/ethereum/go-ethereum/p2p.(*Server).Start
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/server.go:487 +0x4d8

goroutine 116 [IO wait]:
internal/poll.runtime_pollWait(0x1298a5390, 0x72)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/netpoll.go:229 +0xa4
internal/poll.(*pollDesc).wait(0x14000bfce98, 0x72, 0x0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x14000bfce80)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_unix.go:402 +0x1ec
net.(*netFD).accept(0x14000bfce80)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/fd_unix.go:173 +0x2c
net.(*UnixListener).accept(0x14000bfab40)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/unixsock_posix.go:167 +0x2c
net.(*UnixListener).Accept(0x14000bfab40)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/unixsock.go:260 +0x34
github.com/ethereum/go-ethereum/rpc.(*Server).ServeListener(0x14000bbd860, {0x10193db80, 0x14000bfab40})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/ipc.go:30 +0x30
created by github.com/ethereum/go-ethereum/rpc.StartIPCEndpoint
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/endpoints.go:50 +0x1c0

goroutine 117 [IO wait]:
internal/poll.runtime_pollWait(0x1298a52a8, 0x72)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/netpoll.go:229 +0xa4
internal/poll.(*pollDesc).wait(0x14000d0b118, 0x72, 0x0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x14000d0b100)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_unix.go:402 +0x1ec
net.(*netFD).accept(0x14000d0b100)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/fd_unix.go:173 +0x2c
net.(*TCPListener).accept(0x14000bc1968)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/tcpsock_posix.go:140 +0x2c
net.(*TCPListener).Accept(0x14000bc1968)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/tcpsock.go:262 +0x34
net/http.(*Server).Serve(0x14000719500, {0x10193db50, 0x14000bc1968})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/http/server.go:3001 +0x37c
created by github.com/ethereum/go-ethereum/node.(*httpServer).start
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/node/rpcstack.go:145 +0x25c

goroutine 118 [IO wait]:
internal/poll.runtime_pollWait(0x1298a51c0, 0x72)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/netpoll.go:229 +0xa4
internal/poll.(*pollDesc).wait(0x14000d0b298, 0x72, 0x0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x14000d0b280)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_unix.go:402 +0x1ec
net.(*netFD).accept(0x14000d0b280)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/fd_unix.go:173 +0x2c
net.(*TCPListener).accept(0x14000bc1980)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/tcpsock_posix.go:140 +0x2c
net.(*TCPListener).Accept(0x14000bc1980)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/tcpsock.go:262 +0x34
net/http.(*Server).Serve(0x140007196c0, {0x10193db50, 0x14000bc1980})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/http/server.go:3001 +0x37c
created by github.com/ethereum/go-ethereum/node.(*httpServer).start
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/node/rpcstack.go:145 +0x25c

goroutine 119 [select]:
github.com/ethereum/go-ethereum/eth/protocols/eth.StartENRUpdater.func1({0x101939be0, 0x14000bc1998}, 0x14000d16360, 0x14000736000, 0x1400076d000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/protocols/eth/discovery.go:48 +0xcc
created by github.com/ethereum/go-ethereum/eth/protocols/eth.StartENRUpdater
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/protocols/eth/discovery.go:45 +0x7c

goroutine 120 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 121 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 122 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 123 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 124 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 125 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 126 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 127 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 128 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 129 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 146 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 147 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 148 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 149 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 150 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 151 [select]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0x140001f1a40, 0x1000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:50 +0x90
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/bloombits.go:48 +0x50

goroutine 152 [select]:
github.com/ethereum/go-ethereum/eth.(*handler).txBroadcastLoop(0x14000021ef0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/handler.go:662 +0xe4
created by github.com/ethereum/go-ethereum/eth.(*handler).Start
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/handler.go:528 +0xe8

goroutine 153 [chan receive]:
github.com/ethereum/go-ethereum/eth.(*handler).minedBroadcastLoop(0x14000021ef0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/handler.go:650 +0x74
created by github.com/ethereum/go-ethereum/eth.(*handler).Start
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/handler.go:533 +0x17c

goroutine 154 [select]:
github.com/ethereum/go-ethereum/eth.(*chainSyncer).loop(0x140000abf80)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/sync.go:119 +0x1d8
created by github.com/ethereum/go-ethereum/eth.(*handler).Start
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/handler.go:537 +0x1b4

goroutine 155 [chan receive]:
github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1(0x1400075a000, 0x140003fcc30)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/cmd/utils/cmd.go:90 +0x290
created by github.com/ethereum/go-ethereum/cmd/utils.StartNode
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/cmd/utils/cmd.go:75 +0xa0

goroutine 11 [select]:
github.com/ethereum/go-ethereum/eth/fetcher.(*BlockFetcher).loop(0x14000245a40)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/fetcher/block_fetcher.go:380 +0x2ac
created by github.com/ethereum/go-ethereum/eth/fetcher.(*BlockFetcher).Start
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/fetcher/block_fetcher.go:232 +0x3c

goroutine 12 [select]:
github.com/ethereum/go-ethereum/eth/fetcher.(*TxFetcher).loop(0x140001ea370)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/fetcher/tx_fetcher.go:354 +0x10c
created by github.com/ethereum/go-ethereum/eth/fetcher.(*TxFetcher).Start
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/fetcher/tx_fetcher.go:336 +0x3c

goroutine 68 [sleep]:
time.Sleep(0xdf8475800)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/time.go:193 +0x138
github.com/ethereum/go-ethereum/cmd/utils.monitorFreeDiskSpace(0x14000690840, {0x14000e00000, 0x3c}, 0x10000000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/cmd/utils/cmd.go:118 +0x3c
created by github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/cmd/utils/cmd.go:87 +0x280

goroutine 156 [chan receive]:
github.com/ethereum/go-ethereum/rpc.(*Server).ServeCodec(0x1400059cc00, {0x101953998, 0x14000705ef0}, 0x0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/server.go:87 +0x128
created by github.com/ethereum/go-ethereum/rpc.DialInProc.func1
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/inproc.go:29 +0xa0

goroutine 109 [select]:
github.com/ethereum/go-ethereum/rpc.(*Client).dispatch(0x140001e8d80, {0x101953998, 0x14000705ef0})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:576 +0x1a8
created by github.com/ethereum/go-ethereum/rpc.initClient
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:240 +0x3b4

goroutine 157 [select]:
github.com/ethereum/go-ethereum/rpc.(*Client).dispatch(0x140005edd40, {0x101953998, 0x14000705f90})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:576 +0x1a8
created by github.com/ethereum/go-ethereum/rpc.initClient
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:240 +0x3b4

goroutine 158 [chan receive]:
main.startNode.func1(0x140003fcc30, 0x14000d16540, 0x14000d101c0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/cmd/geth/main.go:359 +0x6c
created by main.startNode
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/cmd/geth/main.go:351 +0x14c

goroutine 110 [select]:
net.(*pipe).read(0x14000d0b300, {0x14000468a00, 0x200, 0x200})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/pipe.go:159 +0x104
net.(*pipe).Read(0x14000d0b300, {0x14000468a00, 0x200, 0x200})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/pipe.go:142 +0x40
encoding/json.(*Decoder).refill(0x1400070adc0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/encoding/json/stream.go:165 +0x18c
encoding/json.(*Decoder).readValue(0x1400070adc0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/encoding/json/stream.go:140 +0xa4
encoding/json.(*Decoder).Decode(0x1400070adc0, {0x10175ed00, 0x14000515410})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/encoding/json/stream.go:63 +0x68
github.com/ethereum/go-ethereum/rpc.(*jsonCodec).readBatch(0x14000705ef0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/json.go:209 +0x58
github.com/ethereum/go-ethereum/rpc.(*Client).read(0x140001e8d80, {0x101953998, 0x14000705ef0})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:649 +0x6c
created by github.com/ethereum/go-ethereum/rpc.(*Client).dispatch
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:573 +0xd4

goroutine 162 [select]:
net.(*pipe).read(0x14000d0b380, {0x14000440000, 0x200, 0x200})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/pipe.go:159 +0x104
net.(*pipe).Read(0x14000d0b380, {0x14000440000, 0x200, 0x200})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/pipe.go:142 +0x40
encoding/json.(*Decoder).refill(0x1400070af00)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/encoding/json/stream.go:165 +0x18c
encoding/json.(*Decoder).readValue(0x1400070af00)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/encoding/json/stream.go:140 +0xa4
encoding/json.(*Decoder).Decode(0x1400070af00, {0x10175ed00, 0x14000436018})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/encoding/json/stream.go:63 +0x68
github.com/ethereum/go-ethereum/rpc.(*jsonCodec).readBatch(0x14000705f90)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/json.go:209 +0x58
github.com/ethereum/go-ethereum/rpc.(*Client).read(0x140005edd40, {0x101953998, 0x14000705f90})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:649 +0x6c
created by github.com/ethereum/go-ethereum/rpc.(*Client).dispatch
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:573 +0xd4

goroutine 111 [select]:
github.com/ethereum/go-ethereum/p2p/enode.(*DB).expirer(0x140005ee9a0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/enode/nodedb.go:315 +0xb4
created by github.com/ethereum/go-ethereum/p2p/enode.(*DB).ensureExpirer.func1
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/p2p/enode/nodedb.go:306 +0x3c

goroutine 164 [chan receive]:
github.com/ethereum/go-ethereum/rpc.(*Server).ServeCodec(0x14000bfb5c0, {0x1019539d8, 0x14000f228a0}, 0x0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/server.go:87 +0x128
github.com/ethereum/go-ethereum/rpc.(*Server).WebsocketHandler.func1({0x10193dd60, 0x14000f02620}, 0x14000f2c000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/websocket.go:64 +0xfc
net/http.HandlerFunc.ServeHTTP(0x14000bc1938, {0x10193dd60, 0x14000f02620}, 0x14000f2c000)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/http/server.go:2046 +0x40
github.com/ethereum/go-ethereum/node.(*httpServer).ServeHTTP(0x140001e5680, {0x10193dd60, 0x14000f02620}, 0x14000f2c000)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/node/rpcstack.go:188 +0x154
net/http.serverHandler.ServeHTTP({0x140007196c0}, {0x10193dd60, 0x14000f02620}, 0x14000f2c000)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/http/server.go:2878 +0x444
net/http.(*conn).serve(0x14000325b80, {0x10194d620, 0x1400016d260})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/http/server.go:1929 +0xb6c
created by net/http.(*Server).Serve
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/http/server.go:3033 +0x4b8

goroutine 166 [select]:
github.com/ethereum/go-ethereum/rpc.(*websocketCodec).pingLoop(0x14000f228a0)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/websocket.go:283 +0xf4
created by github.com/ethereum/go-ethereum/rpc.newWebsocketCodec
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/websocket.go:255 +0x244

goroutine 167 [select]:
github.com/ethereum/go-ethereum/rpc.(*Client).dispatch(0x14000430bd0, {0x1019539d8, 0x14000f228a0})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:576 +0x1a8
created by github.com/ethereum/go-ethereum/rpc.initClient
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:240 +0x3b4

goroutine 168 [IO wait]:
internal/poll.runtime_pollWait(0x1298a50d8, 0x72)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/netpoll.go:229 +0xa4
internal/poll.(*pollDesc).wait(0x14000f12a98, 0x72, 0x0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000f12a80, {0x14000f32000, 0x400, 0x400})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_unix.go:167 +0x1dc
net.(*netFD).Read(0x14000f12a80, {0x14000f32000, 0x400, 0x400})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/fd_posix.go:56 +0x44
net.(*conn).Read(0x140002ae0f0, {0x14000f32000, 0x400, 0x400})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/net.go:183 +0x4c
bufio.(*Reader).fill(0x1400040b680)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/bufio/bufio.go:101 +0x10c
bufio.(*Reader).Peek(0x1400040b680, 0x2)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/bufio/bufio.go:139 +0x6c
github.com/gorilla/websocket.(*Conn).read(0x140002bc840, 0x2)
    /Users/maurelian/go-path/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:370 +0x34
github.com/gorilla/websocket.(*Conn).advanceFrame(0x140002bc840)
    /Users/maurelian/go-path/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:798 +0x8c
github.com/gorilla/websocket.(*Conn).NextReader(0x140002bc840)
    /Users/maurelian/go-path/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:980 +0xb4
github.com/gorilla/websocket.(*Conn).ReadJSON(0x140002bc840, {0x10175ed00, 0x14001985230})
    /Users/maurelian/go-path/pkg/mod/github.com/gorilla/websocket@v1.4.2/json.go:50 +0x28
github.com/ethereum/go-ethereum/rpc.(*jsonCodec).readBatch(0x14000443220)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/json.go:209 +0x58
github.com/ethereum/go-ethereum/rpc.(*Client).read(0x14000430bd0, {0x1019539d8, 0x14000f228a0})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:649 +0x6c
created by github.com/ethereum/go-ethereum/rpc.(*Client).dispatch
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:573 +0xd4

goroutine 212 [select]:
github.com/ethereum/go-ethereum/rpc.(*Client).dispatch(0x14001084ea0, {0x1019539d8, 0x1400109fb30})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:576 +0x1a8
created by github.com/ethereum/go-ethereum/rpc.initClient
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:240 +0x3b4

goroutine 213 [IO wait]:
internal/poll.runtime_pollWait(0x1298a4ff0, 0x72)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/runtime/netpoll.go:229 +0xa4
internal/poll.(*pollDesc).wait(0x14000c71998, 0x72, 0x0)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000c71980, {0x140010b6000, 0x400, 0x400})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/internal/poll/fd_unix.go:167 +0x1dc
net.(*netFD).Read(0x14000c71980, {0x140010b6000, 0x400, 0x400})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/fd_posix.go:56 +0x44
net.(*conn).Read(0x14000172098, {0x140010b6000, 0x400, 0x400})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/net/net.go:183 +0x4c
bufio.(*Reader).fill(0x14000c73320)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/bufio/bufio.go:101 +0x10c
bufio.(*Reader).Peek(0x14000c73320, 0x2)
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/bufio/bufio.go:139 +0x6c
github.com/gorilla/websocket.(*Conn).read(0x14000c646e0, 0x2)
    /Users/maurelian/go-path/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:370 +0x34
github.com/gorilla/websocket.(*Conn).advanceFrame(0x14000c646e0)
    /Users/maurelian/go-path/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:798 +0x8c
github.com/gorilla/websocket.(*Conn).NextReader(0x14000c646e0)
    /Users/maurelian/go-path/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:980 +0xb4
github.com/gorilla/websocket.(*Conn).ReadJSON(0x14000c646e0, {0x10175ed00, 0x14001946c90})
    /Users/maurelian/go-path/pkg/mod/github.com/gorilla/websocket@v1.4.2/json.go:50 +0x28
github.com/ethereum/go-ethereum/rpc.(*jsonCodec).readBatch(0x1400109ca50)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/json.go:209 +0x58
github.com/ethereum/go-ethereum/rpc.(*Client).read(0x14001084ea0, {0x1019539d8, 0x1400109fb30})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:649 +0x6c
created by github.com/ethereum/go-ethereum/rpc.(*Client).dispatch
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/client.go:573 +0xd4

goroutine 445 [runnable]:
bytes.(*Buffer).Write(0x1400195cc90, {0x14000409560, 0x33, 0x60})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/bytes/buffer.go:168 +0xe8
fmt.Fprintf({0x10192a7a0, 0x1400195cc90}, {0x1013cf4d7, 0x13}, {0x14001912c08, 0x4, 0x4})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/fmt/print.go:205 +0x84
github.com/ethereum/go-ethereum/log.TerminalFormat.func1(0x14001945500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/format.go:132 +0x28c
github.com/ethereum/go-ethereum/log.formatFunc.Format(0x140005a2630, 0x14001945500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/format.go:68 +0x30
github.com/ethereum/go-ethereum/log.StreamHandler.func1(0x14001945500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/handler.go:43 +0x48
github.com/ethereum/go-ethereum/log.funcHandler.Log(0x1400059c4b0, 0x14001945500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/handler.go:31 +0x30
github.com/ethereum/go-ethereum/log.SyncHandler.func1(0x14001945500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/handler.go:57 +0xd8
github.com/ethereum/go-ethereum/log.funcHandler.Log(0x140005b4160, 0x14001945500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/handler.go:31 +0x30
github.com/ethereum/go-ethereum/log.LazyHandler.func1(0x14001945500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/handler.go:299 +0x300
github.com/ethereum/go-ethereum/log.funcHandler.Log(0x14000738f18, 0x14001945500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/handler.go:31 +0x30
github.com/ethereum/go-ethereum/log.(*GlogHandler).Log(0x1400033b880, 0x14001945500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/handler_glog.go:202 +0x474
github.com/ethereum/go-ethereum/log.(*swapHandler).Log(0x1400019abe0, 0x14001945500)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/handler_go14.go:15 +0x88
github.com/ethereum/go-ethereum/log.(*logger).write(0x1023a9a80, {0x1013c9c60, 0xf}, 0x3, {0x14001913518, 0x2, 0x2}, 0x2)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/logger.go:134 +0x27c
github.com/ethereum/go-ethereum/log.Info(...)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/log/root.go:44
github.com/ethereum/go-ethereum/eth/catalyst.(*ConsensusAPI).ForkchoiceUpdatedV1(0x140000fc4e0, {{0xcd, 0x44, 0x3a, 0x7f, 0x4a, 0xd7, 0xab, 0xe5, 0x14, ...}, ...}, ...)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/eth/catalyst/api.go:223 +0x354
reflect.Value.call({0x14000bd4600, 0x14000d101a8, 0x13}, {0x1013bd516, 0x4}, {0x14001943380, 0x3, 0x4})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/reflect/value.go:543 +0x584
reflect.Value.Call({0x14000bd4600, 0x14000d101a8, 0x13}, {0x14001943380, 0x3, 0x4})
    /opt/homebrew/Cellar/go/1.17.2/libexec/src/reflect/value.go:339 +0x8c
github.com/ethereum/go-ethereum/rpc.(*callback).call(0x14000d161e0, {0x10194d578, 0x140016f9980}, {0x140017ffe60, 0x1a}, {0x1400195c5a0, 0x2, 0x2})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/service.go:206 +0x3c8
github.com/ethereum/go-ethereum/rpc.(*handler).runMethod(0x14001084f30, {0x10194d578, 0x140016f9980}, 0x14001956310, 0x14000d161e0, {0x1400195c5a0, 0x2, 0x2})
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:389 +0x64
github.com/ethereum/go-ethereum/rpc.(*handler).handleCall(0x14001084f30, 0x1400195c540, 0x14001956310)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:337 +0x258
github.com/ethereum/go-ethereum/rpc.(*handler).handleCallMsg(0x14001084f30, 0x1400195c540, 0x14001956310)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:298 +0x90
github.com/ethereum/go-ethereum/rpc.(*handler).handleMsg.func1(0x1400195c540)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:139 +0x3c
github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc.func1(0x14001084f30, 0x14001946ca8)
    /Users/maurelian/temp/rollup-node-experiments-1/reference-optimistic-geth/rpc/handler.go:226 +0xd8
created by github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc

[protolambda]

Thank you for the detailed logs! The bug is a concurrent map write to preparedBlocks: https://github.com/ethereum-optimism/reference-optimistic-geth/blob/2ffee1a8d355874d22090dec1e722d70b85009d7/eth/catalyst/api.go#L222

Somewhere two block-producing forkchoice updates were running in parallel (maybe the first timed out, and the second was started) and ended up writing a block to the same map. Or maybe you were running two rollup nodes?

I'll look through the latest Geth fork for the Kintsugi testnet, this looks like an upstream bug! (which might have been fixed, the current optimism diff is extending a Merge branch from a few weeks ago)

[protolambda]

https://github.com/MariusVanDerWijden/go-ethereum/blob/63a89820ca9721ef2256859391db5a5920faf16c/eth/catalyst/api.go#L221

The problem also exists in upstream geth, will try submit a PR tomorrow

[protolambda]

Fixed in https://github.com/ethereum/go-ethereum/pull/24236

We'll rebase the L2 diff when it gets merged. (and squash the L2 changes, minimal simple diff ftw)