search

ethereum / py-evm

A Python implementation of the Ethereum Virtual Machine
https://py-evm.readthedocs.io/en/latest/
MIT License
2.27k stars 654 forks source link

Weird balance overrides when forking `py-evm` #1960

Open Pet3ris opened 4 years ago

Pet3ris commented 4 years ago
Pip freeze output

``` aniso8601==7.0.0 appdirs==1.4.4 appnope==0.1.0 argon2-cffi==20.1.0 asttokens==2.0.3 async-generator==1.10 attrdict==2.0.1 attrs==20.2.0 Babel==2.8.0 backcall==0.2.0 base58==2.0.1 bitarray==1.2.2 black==20.8b1 blake2b-py==0.1.3 bleach==3.2.1 cached-property==1.5.2 certifi==2020.6.20 cffi==1.14.3 chardet==3.0.4 click==7.1.2 cytoolz==0.11.0 decorator==4.4.2 defusedxml==0.6.0 entrypoints==0.3 eth-abi==2.1.1 eth-account==0.5.4 eth-bloom==1.0.3 eth-hash==0.2.0 eth-keyfile==0.5.1 eth-keys==0.3.3 eth-rlp==0.2.1 eth-tester==0.5.0b2 eth-typing==2.2.2 eth-utils==1.9.5 fastdiff==0.2.0 Flask==1.1.2 Flask-Cors==3.0.9 Flask-GraphQL==2.0.1 future==0.18.2 gevent==20.9.0 graphene==2.1.8 graphql-core==2.3.2 graphql-relay==2.0.1 graphql-server-core==1.2.0 greenlet==0.4.17 gunicorn==20.0.4 hexbytes==0.2.1 idna==2.10 importlib-metadata==2.0.0 iniconfig==1.0.1 ipfshttpclient==0.6.1 ipykernel==5.3.4 ipython==7.18.1 ipython-genutils==0.2.0 isort==5.5.3 itsdangerous==1.1.0 jedi==0.17.2 Jinja2==2.11.2 json5==0.9.5 jsonschema==3.2.0 jupyter-client==6.1.7 jupyter-core==4.6.3 jupyter-server==1.0.0rc16 jupyterlab==2.2.8 jupyterlab-pygments==0.1.1 jupyterlab-server==1.2.0 lru-dict==1.1.6 MarkupSafe==1.1.1 mistune==0.8.4 more-itertools==8.5.0 multiaddr==0.0.9 mypy==0.782 mypy-extensions==0.4.3 nbclassic==0.2.0rc7 nbclient==0.5.0 nbconvert==6.0.5 nbformat==5.0.7 nest-asyncio==1.4.0 netaddr==0.8.0 notebook==6.1.4 packaging==20.4 pandocfilters==1.4.2 parsimonious==0.8.1 parso==0.7.1 pathspec==0.8.0 pexpect==4.8.0 pickleshare==0.7.5 pluggy==0.13.1 prometheus-client==0.8.0 promise==2.3 prompt-toolkit==3.0.7 protobuf==3.13.0 ptyprocess==0.6.0 py==1.9.0 py-ecc==4.1.0 py-evm==0.3.0a19 py-geth==2.4.0 py-solc===3.2.0-fixedstdin py-solc-x==1.0.0 pycparser==2.20 pycryptodome==3.9.8 pyethash==0.1.27 pyevmasm==0.2.3 Pygments==2.7.1 pyparsing==2.4.7 pyrsistent==0.17.3 pysha3==1.0.2 pytest==6.0.2 pytest-tornasync==0.6.0.post2 python-dateutil==2.8.1 python-dotenv==0.14.0 python-json-logger==2.0.1 pytz==2020.1 pyzmq==19.0.2 regex==2020.7.14 requests==2.24.0 rlp==2.0.0a1 rusty-rlp==0.1.15 Rx==1.6.1 semantic-version==2.8.5 Send2Trash==1.5.0 six==1.15.0 snapshottest==0.5.1 sortedcontainers==2.2.2 termcolor==1.1.0 terminado==0.9.1 testpath==0.4.4 toml==0.10.1 toolz==0.11.1 tornado==6.0.4 traitlets==5.0.4 trie==2.0.0a4 typed-ast==1.4.1 typing-extensions==3.7.4.3 urllib3==1.25.11 varint==1.0.2 vyper==0.2.7 wasmer==0.4.1 wcwidth==0.2.5 web3==5.12.2 webencodings==0.5.1 websockets==8.1 Werkzeug==1.0.1 zipp==3.3.1 zope.event==4.5.0 zope.interface==5.1.2 ```

What is wrong?

I'm running a notebook shown below (exported in md for simplicity).

More about how this works:

How is the fork itself set up:

Fork testing notebook

Designed to explore why account balances do not preserve correctly when forking py-evm.

from typing import Any, Iterable, Optional, Type

from eth.abc import BlockAPI, ExecutionContextAPI
from eth.rlp.blocks import BaseBlock
from eth.vm.forks.muir_glacier import MuirGlacierVM
from eth.vm.forks.muir_glacier.blocks import MuirGlacierBlock
from eth.vm.forks.muir_glacier.headers import (
    compute_muir_glacier_difficulty,
    configure_muir_glacier_header,
    create_muir_glacier_header_from_parent,
)
from eth.vm.forks.muir_glacier.state import MuirGlacierState
from eth.vm.state import BaseState
from eth_hash.auto import keccak
from eth_typing import Address, BlockNumber, Hash32
from eth_utils import to_checksum_address

def get_fallback_vm_configuration():
    balances_set = set()
    ran = [0]

    class FallbackState(MuirGlacierState):
        def get_balance(self, address: Address) -> int:
            adx = to_checksum_address(address.hex())
            if address in balances_set or adx != "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf":
                balance = self._account_db.get_balance(address)
                if address in balances_set:
                    print(f"updated get_balance({adx}) = {balance}")
                else:
                    print(f"original get_balance({adx}) = {balance}")
                return balance

            # New idea: commit fallback balance to current chain when retrieving
            if adx != "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf":
                raise Exception("impossible")
            balance = 2 * 10**18
            self.set_balance(address, balance)
            print(f"override get_balance({adx}) = {balance}")
            return balance

        def set_balance(self, address: Address, balance: int) -> None:
            adx = to_checksum_address(address.hex())
            if ran[0] < 1 and adx == "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf":
                ran[0] += 1
                print(f"ignored set_balance({adx}) := {balance}")
                return # ignore first run

            print(f"set_balance({adx}) := {balance}")
            balances_set.add(address)

            self._account_db.set_balance(address, balance)

    class FallbackVM(MuirGlacierVM):
        """Fallback virtual machine that forks from a given web3 network."""

        # fork name
        fork = "muir-glacier-mainnet-fallback"

        # classes
        block_class: Type[BaseBlock] = MuirGlacierBlock
        _state_class: Type[BaseState] = FallbackState

        # Methods
        create_header_from_parent = staticmethod(create_muir_glacier_header_from_parent)  # type: ignore
        compute_difficulty = staticmethod(compute_muir_glacier_difficulty)  # type: ignore
        configure_header = configure_muir_glacier_header

    no_proof_vms = ((0, FallbackVM),)
    return no_proof_vms
import web3
import eth_tester

vm_configuration = get_fallback_vm_configuration()
backend = eth_tester.PyEVMBackend(vm_configuration=vm_configuration)
w3 = web3.Web3(web3.Web3.EthereumTesterProvider(backend))
ignored set_balance(0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf) := 1000000000000000000000000
set_balance(0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF) := 1000000000000000000000000
set_balance(0x6813Eb9362372EEF6200f3b1dbC3f819671cBA69) := 1000000000000000000000000
set_balance(0x1efF47bc3a10a45D4B230B5d10E37751FE6AA718) := 1000000000000000000000000
set_balance(0xe1AB8145F7E55DC933d51a18c793F901A3A0b276) := 1000000000000000000000000
set_balance(0xE57bFE9F44b819898F47BF37E5AF72a0783e1141) := 1000000000000000000000000
set_balance(0xd41c057fd1c78805AAC12B0A94a405c0461A6FBb) := 1000000000000000000000000
set_balance(0xF1F6619B38A98d6De0800F1DefC0a6399eB6d30C) := 1000000000000000000000000
set_balance(0xF7Edc8FA1eCc32967F827C9043FcAe6ba73afA5c) := 1000000000000000000000000
set_balance(0x4CCeBa2d7D2B4fdcE4304d3e09a1fea9fbEb1528) := 1000000000000000000000000
w3.eth.coinbase
'0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf'
w3.eth.getBalance(w3.eth.coinbase)
set_balance(0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf) := 2000000000000000000
override get_balance(0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf) = 2000000000000000000

2000000000000000000
w3.eth.accounts
['0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf',
 '0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF',
 '0x6813Eb9362372EEF6200f3b1dbC3f819671cBA69',
 '0x1efF47bc3a10a45D4B230B5d10E37751FE6AA718',
 '0xe1AB8145F7E55DC933d51a18c793F901A3A0b276',
 '0xE57bFE9F44b819898F47BF37E5AF72a0783e1141',
 '0xd41c057fd1c78805AAC12B0A94a405c0461A6FBb',
 '0xF1F6619B38A98d6De0800F1DefC0a6399eB6d30C',
 '0xF7Edc8FA1eCc32967F827C9043FcAe6ba73afA5c',
 '0x4CCeBa2d7D2B4fdcE4304d3e09a1fea9fbEb1528']
starting_bal = w3.eth.getBalance(w3.eth.accounts[0])
updated get_balance(0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf) = 0

How can it be fixed

This may not be a bug, but I'd love to hear how to ensure that the balances stay consistent between transactions if I do override them.

carver commented 4 years ago

I haven't dived deep into what's going on with this code yet. This isn't really an answer to your question, but an alternative would be to follow the model of TheDAO fork: https://github.com/ethereum/py-evm/blob/239c72c202cb6371fd0059436152fda91592124e/eth/vm/forks/homestead/headers.py#L85-L105

Pet3ris commented 4 years ago

@carver this makes sense and thanks so much for highlighting the example, it's quite elegant. Unfortunately, I'm making a dynamic fork to simulate mainnet calls so I'm following an approach similar to ganache which responds and fills in gaps lazily as they are requested by specific transactions rather than pre-loading state at once. The idea is that without running the transactions in the first place, I don't really know what state to set.

flux627 commented 2 years ago

Did you ever figure this out?

Pet3ris commented 2 years ago

Unfortunately I didn't, I parked this until I have more observability infrastructure to be able to debug it better but one thing that helped with related issues was being thorough in incorporating all the API functions. For example for nonces, in addition to set_nonce and get_nonce there is increment_nonce and important to update all of them.

Pet3ris commented 2 years ago

@flux627 are you running into a similar issue?

flux627 commented 2 years ago

I'm still in a research phase for tooling- looking to see if it was possible to fork mainnet with this like Ganache, in hopes that this implementation is faster. But, it seems that this tooling isn't really meant for this. Also looking at hevm, but I don't know Haskell and it doesn't have any bindings. Any suggestions for performant mainnet forking tests are welcome.

carver commented 2 years ago

Unfortunately, we aren't currently putting any resources toward new features like this in py-evm (though forking mainnet is definitely a cool one that we've talked about, and would like some day).

Though it's fairly straightforward to think of "forking mainnet" (without having the full state) as a kind of variant of Beam Sync. So you can check out how Beam Sync is implemented in trinity, especially the pausing_vm_decorator and how it overwrites VMState. Note that it overrides all the methods (like increment_nonce) in a similar way.

See https://github.com/ethereum/trinity/blob/eaa3b040ffdf0848b8e00a5f329480662ecc7c11/trinity/sync/beam/importer.py#L190