Function `create_access_list` - incorrect behavior and documentation

barakman commented 6 months ago

What happened?

These are the input and output of function create_access_list according to the official documentation:

Input Example:

{'from': '0x0', 'data': '0x0', 'type': '0x1'}

Output Example:

{
    'accessList': (
        {
            'address': '0xde0b295669a9fd93d5f28d9ec85e40f4cb697bae',
            'storageKeys': (
                '0x0000000000000000000000000000000000000000000000000000000000000003',
                '0x0000000000000000000000000000000000000000000000000000000000000007',
            )
        },
        {
            'address': '0xbb9bc244d798123fde783fcc1c72d3bb8c189413',
            'storageKeys': ()
        },
    ),
    'gas': '21000'
}

There are several problems here, both in the documentation, and in the actual behavior of the function.

Problem 1 (should be fixed in the documentation):

The input should be the entire transaction dictionary, and not just the keys in the example, including:

'gas'
'chainId'
'from'
'to'
'data'
'value'
'nonce'
'type'
'gasPrice' in the case of type 1, or 'maxPriorityFeePerGas' and 'maxFeePerGas' in the case of type 2

It is possible that some of them, like 'from' and 'chainId', can be omitted. But it seems that at least the majority of them should be included. Otherwise, the function throws errors such as intrinsic gas too low, etc.

Problem 2 (should be fixed either in the documentation or in the code):

The output dictionary does not have a key named 'gas', but a key named 'gasUsed'.

Problem 3 (should be fixed in the code):

The value of the output 'accessList' is not a list of dict items, but a list of AttributeDict items. Moreover, the value of each 'storageKeys' is not a list of str items, but a list of HexBytes items. This makes it impossible to just take the output 'accessList' and add it to the input transaction like this:

result = web3.eth.create_access_list(tx)
if tx['gas'] > result['gasUsed']:
    tx['gas'] = result['gasUsed']
    tx['accessList'] = result['accessList']

Instead, one needs to do something like:

result = web3.eth.create_access_list(tx)
if tx['gas'] > result['gasUsed']:
    tx['gas'] = result['gasUsed']
    tx['accessList'] = [
        {
            'address': item['address'],
            'storageKeys': [key.hex() for key in item['storageKeys']]
        }
        for item in result['accessList']
    ]

Code that produced the error

No response

Full error output

No response

Fill this section in if you know how this could or should be fixed

No response

web3 Version

6.16.0

Python Version

3.9.13

Operating System

MacOS

Output from `pip freeze`

aiofiles==22.1.0
aiohttp==3.8.1
aiosignal==1.2.0
aiosqlite==0.18.0
alabaster==0.7.12
alchemy-sdk==0.1.1
anyio==3.6.1
appnope==0.1.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.2
async-timeout==4.0.2
attrs==21.4.0
Babel==2.10.3
backcall==0.2.0
backoff==2.2.1
bancor-research==2.0.3
base58==2.1.1
beautifulsoup4==4.11.1
binaryornot==0.4.4
bitarray==2.6.0
black==22.6.0
bleach==5.0.1
branca==0.5.0
cachetools==5.2.0
cairo-lang==0.9.1
carbon-simulator==2.0
certifi==2022.6.15
cffi==1.15.1
chardet==5.0.0
charset-normalizer==2.1.0
click==8.1.3
cloudpickle==2.1.0
colorama==0.4.5
contextvars==2.4
contourpy==1.2.0
cookiecutter==2.1.1
cramjam==2.5.0
cvxpy==1.3.2
cycler==0.11.0
cytoolz==0.12.0
dataclass-wizard==0.22.2
datascience==0.17.5
debugpy==1.6.2
decorator==5.1.1
defusedxml==0.7.1
docutils==0.17.1
ecdsa==0.18.0
ecos==2.0.12
entrypoints==0.4
eth-abi==4.1.0
eth-account==0.9.0
eth-hash==0.5.2
eth-keyfile==0.6.1
eth-keys==0.4.0
eth-rlp==0.3.0
eth-typing==3.4.0
eth-utils==2.2.0
fastecdsa==2.2.3
fastjsonschema==2.16.1
fastlane-bot==3.1.1
fastparquet==0.8.1
folium==0.12.1.post1
fonttools==4.34.4
fqdn==1.5.1
frozendict==1.2
frozenlist==1.3.1
fsspec==2022.7.1
gitdb==4.0.9
GitPython==3.1.27
greenlet==1.1.2
hexbytes==0.3.1
idna==3.3
imagesize==1.4.1
immutables==0.18
importlib-metadata==4.12.0
importlib-resources==6.1.1
iniconfig==1.1.1
ipfshttpclient==0.8.0a2
ipykernel==6.15.1
ipython==7.34.0
ipython-genutils==0.2.0
ipywidgets==7.7.1
isoduration==20.11.0
jedi==0.18.1
Jinja2==3.1.2
jinja2-time==0.2.0
joblib==1.2.0
json5==0.9.11
jsonpointer==2.3
jsonschema==4.17.3
jupyter==1.0.0
jupyter-black==0.3.1
jupyter-book==0.13.0
jupyter-cache==0.4.3
jupyter-console==6.4.4
jupyter-events==0.6.3
jupyter-server-mathjax==0.2.6
jupyter-sphinx==0.3.2
jupyter-ydoc==0.2.2
jupyter_client==8.0.3
jupyter_core==5.2.0
jupyter_server==2.3.0
jupyter_server_fileid==0.8.0
jupyter_server_terminals==0.4.4
jupyter_server_ydoc==0.6.1
jupyterlab==3.6.1
jupyterlab-pygments==0.2.2
jupyterlab-widgets==1.1.1
jupyterlab_server==2.19.0
jupytext==1.16.1
kiwisolver==1.4.4
lark-parser==0.12.0
latexcodec==2.0.1
linkify-it-py==1.0.3
lru-dict==1.1.8
markdown-it-py==1.1.0
MarkupSafe==2.1.1
marshmallow==3.17.0
marshmallow-dataclass==8.5.8
marshmallow-enum==1.5.1
marshmallow-oneofschema==3.0.1
matplotlib==3.7.4
matplotlib-inline==0.1.3
mdit-py-plugins==0.2.8
Mesa==1.0.0
mistune==0.8.4
mplfinance==0.12.10b0
mpmath==1.2.1
multiaddr==0.0.9
multidict==6.0.2
mypy-extensions==0.4.3
myst-nb==0.13.2
myst-parser==0.15.2
nbclassic==0.5.2
nbclient==0.5.13
nbconvert==6.5.0
nbdime==3.1.1
nbformat==5.4.0
nest-asyncio==1.5.8
netaddr==0.8.0
networkx==3.2.1
notebook==6.4.12
notebook_shim==0.2.2
numpy==1.23.1
orjson==3.9.10
osqp==0.6.3
packaging==21.3
pandas==1.5.3
pandocfilters==1.5.0
parsimonious==0.9.0
parso==0.8.3
pathspec==0.9.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.2.0
pipdeptree==2.2.1
platformdirs==2.5.2
plotly==5.9.0
pluggy==1.0.0
prometheus-client==0.14.1
prompt-toolkit==3.0.30
protobuf==4.24.4
psutil==5.9.7
ptyprocess==0.7.0
py==1.11.0
pyarrow==11.0.0
pybtex==0.24.0
pybtex-docutils==1.0.2
pycparser==2.21
pycryptodome==3.15.0
pydantic==1.9.1
pydata-sphinx-theme==0.8.1
Pygments==2.12.0
pyparsing==3.0.9
pyrsistent==0.18.1
pytest==7.1.2
pytest-asyncio==0.19.0
python-dateutil==2.8.2
python-dotenv==0.16.0
python-json-logger==2.0.7
python-slugify==6.1.2
pytz==2022.1
pyunormalize==15.0.0
PyYAML==6.0
pyzmq==25.0.0
qdldl==0.1.7.post0
qtconsole==5.4.0
QtPy==2.3.0
regex==2023.8.8
requests==2.31.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rlp==3.0.0
scipy==1.9.0
scs==3.2.4.post1
Send2Trash==1.8.0
six==1.16.0
smmap==5.0.0
sniffio==1.2.0
snowballstemmer==2.2.0
soupsieve==2.3.2.post1
Sphinx==4.5.0
sphinx-book-theme==0.3.3
sphinx-comments==0.0.3
sphinx-copybutton==0.5.0
sphinx-external-toc==0.2.4
sphinx-jupyterbook-latex==0.4.6
sphinx-multitoc-numbering==0.1.3
sphinx-thebe==0.1.2
sphinx-togglebutton==0.3.2
sphinx_design==0.1.0
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-bibtex==2.4.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
SQLAlchemy==1.4.39
sympy==1.10.1
tabulate==0.9.0
tenacity==8.0.1
terminado==0.15.0
text-unidecode==1.3
tinycss2==1.1.1
tokenize-rt==4.2.1
toml==0.10.2
tomli==2.0.1
toolz==0.12.0
tornado==6.2
tqdm==4.64.1
traitlets==5.9.0
typeguard==2.13.3
typing-inspect==0.7.1
typing_extensions==4.7.1
uc-micro-py==1.0.1
uri-template==1.2.0
urllib3==1.26.11
varint==1.0.2
wcwidth==0.2.5
web3==6.16.0
webcolors==1.12
webencodings==0.5.1
websocket-client==1.3.3
websockets==11.0.3
widgetsnbextension==3.6.1
y-py==0.5.9
yarl==1.8.1
ypy-websocket==0.8.2
zipp==3.8.1

barakman commented 6 months ago

Alchemy node used for testing all of this. See the relevant documentation here.

fselmo commented 6 months ago

Thanks for the detailed issue @barakman. I have a draft PR up for getting this resolved. I should be able to wrap it up by tomorrow.

barakman commented 6 months ago

Thanks for the detailed issue @barakman. I have a draft PR up for getting this resolved. I should be able to wrap it up by tomorrow.

Thank you for your quick handling of this. Note that this is probably an API breaking change for anyone using that function, since as far as I understand, you've changed the structure of its output.

BTW, if that is indeed the case, then you can probably revert some of your fixes in the documentation. Specifically, where the documentation describes this is part in the output of that function:

            'accessList': [
                AttributeDict({
                    'address': '0xde0B295669a9FD93d5F28D9Ec85E40f4cb697BAe',
                    'storageKeys': [
                        HexBytes('0x0000000000000000000000000000000000000000000000000000000000000003'),
                        HexBytes('0x0000000000000000000000000000000000000000000000000000000000000007'),
                    ]
                }),
                AttributeDict({
                    'address': '0xBB9bc244D798123fDe783fCc1C72d3Bb8C189413',
                    'storageKeys': []
                }),
            ],

If the AttributeDict and HexBytes are really part of this output, then I'd expect txn to become illegal (cannot be unmarshalled by Go, or something similar to that) as a result of setting txn['accessList'] to that output.

fselmo commented 6 months ago

since as far as I understand, you've changed the structure of its output.

This is not the case. Request formatters were added to properly handle the case where storage keys may be bytes or HexBytes.

Nov1kov commented 4 months ago

Should eth-account support this merge request?

In my case, even eth-account == 0.13.0 couldn't validate the HexBytes in "storageKeys"

    return transaction, self.bot_account.sign_transaction(transaction).raw_transaction
.venv-311\Lib\site-packages\eth_account\signers\local.py:84: in sign_transaction
    return self._publicapi.sign_transaction(transaction_dict, self.key, blobs=blobs)
.venv-311\Lib\site-packages\eth_utils\decorators.py:20: in _wrapper
    return self.method(objtype, *args, **kwargs)
.venv-311\Lib\site-packages\eth_account\account.py:801: in sign_transaction
    ) = sign_transaction_dict(account._key_obj, sanitized_transaction, blobs=blobs)
.venv-311\Lib\site-packages\eth_account\_utils\signing.py:32: in sign_transaction_dict
    unsigned_transaction = serializable_unsigned_transaction_from_dict(
.venv-311\Lib\site-packages\eth_account\_utils\legacy_transactions.py:43: in serializable_unsigned_transaction_from_dict
    return TypedTransaction.from_dict(transaction_dict, blobs=blobs)
.venv-311\Lib\site-packages\eth_account\typed_transactions\typed_transaction.py:97: in from_dict
    transaction=transaction.from_dict(dictionary, blobs=blobs),
.venv-311\Lib\site-packages\eth_account\typed_transactions\dynamic_fee_transaction.py:147: in from_dict
....
    cls.assert_valid_fields(dictionary)
        if not all(valid_fields.values()):
            invalid = {
                key: dictionary[key] for key, valid in valid_fields.items() if not valid
            }
>           raise TypeError(f"Transaction had invalid fields: {repr(invalid)}")
E           TypeError: Transaction had invalid fields: {'accessList': [AttributeDict({'address': ....

.venv-311\Lib\site-packages\eth_account\typed_transactions\dynamic_fee_transaction.py:133: TypeError

That check doesn't allow HexBytes in storageKey.

ethereum / web3.py