API Endpoints not honored when using multiple instances of EthereumTesterProvider

[antazoey]

• Version: 6.9.0
• Python: 3.10
• OS: macos
• pip freeze output

abnf==1.1.1
-e git+ssh://git@github.com/antazoey/afplay-py.git@aa33599426158e1faf2ced81d8b6b07bb670a095#egg=afplay_py
aiohttp==3.8.5
aiosignal==1.3.1
aiosqlite==0.18.0
alabaster==0.7.13
alembic==1.11.3
annotated-types==0.5.0
anyio==3.7.1
-e git+ssh://git@github.com/antazoey/ape-alchemy.git@975b01be439b6ccf4c80894a4f7ee85076273096#egg=ape_alchemy
-e git+ssh://git@github.com/unparalleled-js/ape-arbitrum.git@8e7026d789c00479881b17a64a1975766320f449#egg=ape_arbitrum
ape-avalanche==0.6.3
ape-base==0.6.0
ape-bsc==0.6.1
ape-ens==0.6.1
-e git+ssh://git@github.com/unparalleled-js/ape-etherscan.git@412dc6bd33982ecc3bc2dd02ee39b96653398e54#egg=ape_etherscan
-e git+ssh://git@github.com/unparalleled-js/ape-foundry.git@3a717172f193b00e07fbe687ba04e5777604cf17#egg=ape_foundry
ape-ganache==0.6.8
-e git+ssh://git@github.com/antazoey/ape-hardhat.git@e6b818ded4f70b2e86740cc0ca696289b1432c12#egg=ape_hardhat
ape-infura==0.6.3
ape-ledger==0.6.2
ape-optimism==0.6.2
-e git+ssh://git@github.com/unparalleled-js/ape-polygon.git@fbebb9684eace5c73c005fecd9771e641d41720e#egg=ape_polygon
-e git+ssh://git@github.com/antazoey/ape-safe.git@3c40b5d251b496874be3d9c989599486a6096db5#egg=ape_safe
ape-solidity @ file:///Users/jules/PycharmProjects/ape-solidity
-e git+ssh://git@github.com/antazoey/ape-vyper.git@185594024924b8310e3a3cb26e67c8e2e03cb7f7#egg=ape_vyper
apepay==0.1.1
appnope==0.1.3
argcomplete==2.0.6
asttokens==2.2.1
async-generator==1.10
async-timeout==4.0.3
asyncer==0.0.2
attrs==23.1.0
awscli==1.29.29
Babel==2.13.0
backcall==0.2.0
base58==1.0.3
bcrypt==4.0.1
bitarray==2.8.1
black==23.9.1
bleak==0.21.1
boto3==1.28.29
botocore==1.31.29
build==0.10.0
CacheControl==0.13.1
cached-property==1.5.2
cbor2==5.4.6
certifi==2023.7.22
cffi==1.15.1
cfgv==3.4.0
charset-normalizer==2.1.1
chompjs==1.2.2
cleo==2.0.1
click==8.1.7
colorama==0.4.4
commitizen==2.40.0
commonmark==0.9.1
coverage==7.3.0
crashtest==0.4.1
cryptography==41.0.3
cytoolz==0.12.2
dataclassy==0.11.1
decli==0.5.2
decorator==5.1.1
Deprecated==1.2.14
distlib==0.3.7
dnspython==2.4.2
docopt==0.6.2
docutils==0.18.1
dulwich==0.21.5
ECPy==1.2.5
eip712==0.2.1
email-validator==2.0.0.post2
eth-abi==4.2.0
eth-account==0.8.0
-e git+ssh://git@github.com/antazoey/ape.git@f9c8183b0c646e1d555dac255f922af4f4065bc4#egg=eth_ape
eth-bloom==2.0.0
eth-hash==0.5.2
eth-keyfile==0.6.1
eth-keys==0.4.0
eth-pydantic-types @ file:///Users/jules/PycharmProjects/eth-pydantic-types
eth-rlp==0.3.0
eth-tester==0.9.1b1
eth-typing==3.5.0
eth-utils==2.2.0
ethpm-types @ file:///Users/jules/PycharmProjects/ethpm-types
evm-trace @ file:///Users/jules/PycharmProjects/evm-trace
exceptiongroup==1.1.3
execnet==2.0.2
executing==1.2.0
Faker==18.13.0
fastapi==0.92.0
fastapi-login==1.9.1
filelock==3.12.2
flake8==6.1.0
flake8-breakpoint==1.1.0
flake8-plugin-utils==1.3.3
flake8-print==4.0.1
frozenlist==1.4.0
future==0.18.3
greenlet==2.0.2
h11==0.14.0
hexbytes==0.3.1
hidapi==0.14.0
httpcore==0.16.3
httpx==0.23.3
hypothesis==6.86.2
hypothesis-jsonschema==0.19.0
identify==2.5.27
idna==3.4
ijson==3.2.3
imagesize==1.4.1
importlib-metadata==6.8.0
iniconfig==2.0.0
installer==0.7.0
ipython==8.14.0
isort==5.10.1
jaraco.classes==3.3.0
jedi==0.19.0
Jinja2==3.1.2
jmespath==1.0.1
jsonschema==4.19.0
jsonschema-specifications==2023.7.1
keyring==24.2.0
lazyasd==0.1.4
ledgerblue==0.1.48
ledgereth==0.8.1
libusb1==3.0.0
linkify-it-py==2.0.2
lru-dict==1.2.0
Mako==1.2.4
markdown-it-py==2.2.0
MarkupSafe==2.1.3
matplotlib-inline==0.1.6
mccabe==0.7.0
mdformat==0.7.17
mdformat-gfm==0.3.5
mdformat_frontmatter==2.0.1
mdformat_pyproject==0.0.1
mdformat_tables==0.4.1
mdit-py-plugins==0.3.5
mdurl==0.1.2
more-itertools==10.1.0
morphys==1.0
msgpack==1.0.5
msgspec==0.18.1
multidict==6.0.4
mypy==1.5.1
mypy-extensions==1.0.0
myst-parser==1.0.0
ndeflib==0.3.3
nfcpy==1.0.4
nodeenv==1.8.0
numpy==1.25.2
outcome==1.2.0
packaging==23.1
pandas==1.5.3
pandas-stubs==1.2.0.62
parsimonious==0.9.0
parso==0.8.3
passlib==1.7.4
pathspec==0.11.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==10.0.1
pkginfo==1.9.6
platformdirs==3.10.0
pluggy==1.3.0
pockets==0.9.1
poetry==1.6.1
poetry-core==1.7.0
poetry-plugin-export==1.5.0
pre-commit==3.3.3
prometheus-client==0.17.1
prompt-toolkit==3.0.39
protobuf==4.24.0
psycopg2-binary==2.9.7
ptyprocess==0.7.0
pure-eval==0.2.2
py-cid==0.3.0
py-ecc==6.0.0
py-evm==0.7.0a4
py-geth==3.13.0
py-multibase==1.0.3
py-multicodec==0.2.1
py-multihash==0.2.3
py-solc-x==1.1.1
pyasn1==0.5.0
pycodestyle==2.11.0
pycparser==2.21
pycron==3.0.0
pycryptodome==3.18.0
pycryptodomex==3.19.0
pydantic==2.4.2
pydantic-settings==2.0.3
pydantic_core==2.10.1
pyDes==2.0.1
pyelftools==0.30
pyethash==0.1.27
pyflakes==3.1.0
PyGithub==1.59.1
Pygments==2.16.1
PyJWT==2.8.0
PyNaCl==1.5.0
pyobjc-core==9.2
pyobjc-framework-Cocoa==9.2
pyobjc-framework-CoreBluetooth==9.2
pyobjc-framework-libdispatch==9.2
pyproject_hooks==1.0.0
pyrsistent==0.19.3
pyserial==3.5
pysha3==1.0.2
pytest==7.4.0
pytest-cov==4.1.0
pytest-mock==3.11.1
pytest-watch==4.2.0
pytest-xdist==3.3.1
python-baseconv==1.2.2
python-dateutil==2.8.2
python-dotenv==1.0.0
python-gnupg==0.5.1
python-multipart==0.0.6
python-u2flib-host==3.0.3
pytz==2023.3
pyunormalize==15.0.0
pyupgrade==3.13.0
PyYAML==6.0.1
questionary==1.10.0
rapidfuzz==2.15.1
referencing==0.30.2
regex==2023.8.8
requests==2.31.0
requests-toolbelt==1.0.0
rfc3986==1.5.0
rfc3987==1.3.8
rich==12.6.0
rlp==3.0.0
rpds-py==0.9.2
rsa==4.7.2
ruamel.yaml==0.17.32
ruamel.yaml.clib==0.2.7
s3transfer==0.6.2
safe-pysha3==1.0.4
semantic-version==2.10.0
setuptools-scm==8.0.1
shellingham==1.5.3
siwe==2.2.0
six==1.16.0
sniffio==1.3.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
Sphinx==6.2.1
sphinx-click==4.4.0
sphinx-plausible==0.1.2
sphinx-rtd-theme==1.3.0
sphinxcontrib-applehelp==1.0.7
sphinxcontrib-devhelp==1.0.5
sphinxcontrib-htmlhelp==2.0.4
sphinxcontrib-jquery==4.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-napoleon==0.7
sphinxcontrib-qthelp==1.0.6
sphinxcontrib-serializinghtml==1.1.9
SQLAlchemy==1.4.41
sqlalchemy2-stubs==0.0.2a35
sqlmodel==0.0.8
stack-data==0.6.2
starlette==0.25.0
taskiq==0.6.0
taskiq-dependencies==1.3.1
termcolor==2.3.0
tokenize-rt==5.2.0
-e git+ssh://git@github.com/unparalleled-js/py-tokenlists.git@b4d681f940bd06df041b26f23bcb82a59fbdcd7a#egg=tokenlists
tomli==2.0.1
tomlkit==0.12.1
toolz==0.12.0
tqdm==4.66.1
traitlets==5.9.0
trie==2.1.1
trio==0.21.0
trove-classifiers==2023.8.7
types-PyYAML==6.0.12.11
types-requests==2.31.0.2
types-setuptools==68.1.0.0
types-SQLAlchemy==1.4.53.38
types-urllib3==1.26.25.14
typing_extensions==4.8.0
uc-micro-py==1.0.2
urllib3==1.26.16
uvicorn==0.20.0
varint==1.0.2
virtualenv==20.24.3
vvm==0.2.0
vyper==0.3.9
watchdog==3.0.0
wcwidth==0.2.6
web3==6.9.0
websocket-client==1.6.3
websockets==11.0.3
wrapt==1.15.0
xattr==0.10.1
yarl==1.9.2
zipp==3.16.2

What was wrong?

When trying to use multiple instances of web3 with different eth-testers, the API endpoints are not respected in each instance. Here is a simple repro:

from eth_tester.backends import PyEVMBackend
from web3.providers.eth_tester.defaults import API_ENDPOINTS, static_return

from web3 import EthereumTesterProvider, Web3

def make_web3(chain_id):
    evm_backend = PyEVMBackend.from_mnemonic(
        mnemonic="test test test test test test test test test test test junk",
        num_accounts=5,
    )
    endpoints = {**API_ENDPOINTS}
    endpoints["eth"]["chainId"] = static_return(chain_id)
    tester = EthereumTesterProvider(ethereum_tester=evm_backend, api_endpoints=endpoints)
    return Web3(tester)

def get_chain_id(web3):
    return web3.provider.make_request("eth_chainId", [])

web3_0 = make_web3(1337)
web3_1 = make_web3(1338)

for web3 in (web3_0, web3_1):
    print(get_chain_id(web3))

# prints:
# {'id': 1, 'jsonrpc': '2.0', 'result': 1338}
# {'id': 1, 'jsonrpc': '2.0', 'result': 1338}

notice the chain ID is the same for web3s even though they are separate instances. The API Endoints is not honored

How can it be fixed?

update: I was able to fix the issue by changing the defaults file to be like this:

def get_endpoints():
    return { ... }

API_ENDPOINTS = get_endpoints()

and in my script, I call the method instead of using the constant, so i get a new object every time. Note: I tried using copy and deepcopy but those had other issues.

---

Note: We prefer to use issues to track our work. If you think you've encountered a bug in web3py or have a feature request, you're in the right place. If you have implementation or usage questions, please refer to our documentation and/or join the conversation on discord.

[antazoey]

OK, it seems like the line endpoints["eth"]["chainId"] = static_return(chain_id) also updates the global API_ENDPOINTS dict for some reason

Edit: turns out the issue was because API_ENDPOINTS is not copy-able, it is C stuff i think... so i fixed it by making a method in web3.py to return the api endpoints.

[fselmo]

Yeah I think that, due to the functools methods that it uses as values, the dictionary is not deep-copyable and so it really only ever has references to the values. What a funky quirk you found that I've never ran into 😅 .

This would also work:

...
from toolz import merge

endpoints = {**API_ENDPOINTS}
endpoints["eth"] = merge(endpoints["eth"], {"chainId": static_return(chain_id)})

[edited after the correction below]

[fselmo]

closing as it doesn't appear to be an issue with the library

[antazoey]

endpoints = merge( API_ENDPOINTS, {"eth": {"chainId": static_return(chain_id)}}, )

This isn't quite right (just as FYI)! You lose the rest of eth:

(Pdb) endpoints["eth"]
{'chainId': <function static_return.<locals>.inner at 0x11f876e60>}

I'll update with the correct code soon.

[antazoey]

This is working I think:

def get_endpoints(chain_id: int):
    # HACK: Until https://github.com/ethereum/web3.py/issues/3136 resolved.
    eth_endpoints = merge(API_ENDPOINTS["eth"], {"chainId": static_return(chain_id)})
    return merge(API_ENDPOINTS, {"eth": eth_endpoints})

Thanks @fselmo for your help! Hopefully if someone else encountered this issue, they find this GH issue!

[fselmo]

Ah yes, I saw it on my end too. Thought I updated it here. I edited the original just in case anyone stumbles on it. Thanks for the nudge 👍🏼