Cannon fraud proofs have debug_dbGet dependency

[jyellick]

I was a little unsure whether to classify this issue as a bug or a feature, so please feel free to convert.

There are a number of client diversity projects underway to bring Optimism support to Ethereum clients other than Geth. The community seems to recognize the importance of having multiple client implementations for the health of the overall Optimism ecosystem.

However, it seems that the Cannon fraud proofs are being built with a strong dependency on Geth specific APIs. In particular, the fraudproof_test.go e2e tests seem to ultimately be using the debug_dbGet API. Based on my reading of these tests, this API is not simply being used by the test, but is actually relied on by the implementation. This feels like a serious problem because this API is not part of the formal Ethereum JSON RPC spec, so its behavior is not defined. Additionally because there is no real versioning guarantees over the layout of the Geth database, it is also inherently not stable.

I've not had a chance to really dig into exactly how this API is being used in Cannon, but based on other usage seen in the e2e tests, I suspect that its usage might be related to retrieving nodes of the Merkle Trie. Could we instead use the eth_getProof API instead? Do we need to put an EIP together for a new well specified API that other clients can implement? Is the debug_dbGet usage simply intended to be transient and replaced with something else ultimately? Is the coupling of the on-chain executing mini-Geth so tight that Cannon will never be able to be used with other Ethereum clients?

It would be great to hear some clarification on the above questions and to learn whether changes to eliminate this dependency would be welcomed.

[protolambda]

I definitely hope to support all clients with standard RPC in the future, the debug method is there because it simplified and derisked the MPT a lot, to unblock development of other parts of the proof stack.

Previous versions of Cannon have tried eth_getProof instead, but this added significant complexity at the time, since eth_getProof is meant to provide MPT data only as part of a branch to a leaf node. Edge-cases related to retrieving the right pre-images, even throughout MPT changes (node deletions etc. included), are challenging and resulted in several bugs.

debug_dbGet is thus used instead to retrieve the MPT nodes, providing a much simpler hash->preimage lookup for the MPT nodes (at least when using the old geth tree based DB scheme).

However, with new improvements to the FP design, there may be a way to make eth_getProof work: by utilizing the preimage-hinting system. I.e. the op-program client can write a new type of hint, that essentially says to the server: "I'll need all preimages of state-root X and path y", and the op-program host can prepare the preimages. The in-memory op-program oracle-backed state DB can then fetch the preimages lazily as necessary, and retain a copy of them to then build any new branches on top of, and cache for future use, without interrupting future preimage lookups, since hints are always written with full path/state-root information, unlike previous versions in the original minigeth that had to deal with MPT edits/deletions to fetch the right preimages.

So yes, I believe we can move it back to eth_getProof thanks to newer features that were introduced after the original MPT-proving issues with legacy Cannon/minigeth setup, it will still add a little bit of complexity, but remove the debug_dbGet dependency.