ethereum / eth-account

Account abstraction library for web3.py
http://eth-account.readthedocs.io/
MIT License
257 stars 155 forks source link

Unable to RLP encode to unsigned EIP-1559 transaction #146

Closed antazoey closed 2 months ago

antazoey commented 2 years ago

What was wrong?

An unsigned transaction's hash is wrong and not signable

transaction = DynamicFeeTransaction.from_dict(txn)
encoded_txn = transaction.hash()   #  <---not right hash

My transaction does not have V, R, or S (it is unsigned). It also does not have any access lists. The tests and docs are not helping either:

The hash I am getting does not seem correct because of its length and because the Ledger dongle does not accept it

MORE

This is my transaction dict:

{'chainId': 61, 'to': b'', 'value': 0, 'data': b'`\x80`@R4\x80\x15a\x00\x10W`\x00\x80\xfd[P3`\x00\x80a\x01\x00\n\x81T\x81s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x02\x19\x16\x90\x83s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x02\x17\x90UP`\x01`\x03`\x00a\x01\x00\n\x81T\x81`\xff\x02\x19\x16\x90\x83\x15\x15\x02\x17\x90UPa\x07\xca\x80a\x00{`\x009`\x00\xf3\xfe`\x80`@R`\x046\x10a\x00pW`\x005`\xe0\x1c\x80c>G\xd6\xf3\x11a\x00NW\x80c>G\xd6\xf3\x14a\x00\xe9W\x80c\x8d\xa5\xcb[\x14a\x01NW\x80c\xb6\rB\x88\x14a\x01\x8fW\x80c\xdc\r=\xff\x14a\x01\x99Wa\x00pV[\x80c\x12)\xdc\x9e\x14a\x00uW\x80c#\x8d\xaf\xe0\x14a\x00\xb2W\x80c<\xcf\xd6\x0b\x14a\x00\xdfW[`\x00\x80\xfd[4\x80\x15a\x00\x81W`\x00\x80\xfd[Pa\x00\xb0`\x04\x806\x03` \x81\x10\x15a\x00\x98W`\x00\x80\xfd[\x81\x01\x90\x80\x805\x15\x15\x90` \x01\x90\x92\x91\x90PPPa\x01\xfeV[\x00[4\x80\x15a\x00\xbeW`\x00\x80\xfd[Pa\x00\xc7a\x02\xdcV[`@Q\x80\x82\x15\x15\x81R` \x01\x91PP`@Q\x80\x91\x03\x90\xf3[a\x00\xe7a\x02\xefV[\x00[4\x80\x15a\x00\xf5W`\x00\x80\xfd[Pa\x018`\x04\x806\x03` \x81\x10\x15a\x01\x0cW`\x00\x80\xfd[\x81\x01\x90\x80\x805s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x90` \x01\x90\x92\x91\x90PPPa\x05\x10V[`@Q\x80\x82\x81R` \x01\x91PP`@Q\x80\x91\x03\x90\xf3[4\x80\x15a\x01ZW`\x00\x80\xfd[Pa\x01ca\x05(V[`@Q\x80\x82s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x81R` \x01\x91PP`@Q\x80\x91\x03\x90\xf3[a\x01\x97a\x05LV[\x00[4\x80\x15a\x01\xa5W`\x00\x80\xfd[Pa\x01\xd2`\x04\x806\x03` \x81\x10\x15a\x01\xbcW`\x00\x80\xfd[\x81\x01\x90\x80\x805\x90` \x01\x90\x92\x91\x90PPPa\x06pV[`@Q\x80\x82s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x81R` \x01\x91PP`@Q\x80\x91\x03\x90\xf3[`\x00\x80T\x90a\x01\x00\n\x90\x04s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x163s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x14a\x02\xbfW`@Q\x7f\x08\xc3y\xa0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x81R`\x04\x01\x80\x80` \x01\x82\x81\x03\x82R`\x0b\x81R` \x01\x80\x7f!authorized\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x81RP` \x01\x91PP`@Q\x80\x91\x03\x90\xfd[\x80`\x03`\x00a\x01\x00\n\x81T\x81`\xff\x02\x19\x16\x90\x83\x15\x15\x02\x17\x90UPPV[`\x03`\x00\x90T\x90a\x01\x00\n\x90\x04`\xff\x16\x81V[`\x00\x80T\x90a\x01\x00\n\x90\x04s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x163s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x14a\x03\xb0W`@Q\x7f\x08\xc3y\xa0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x81R`\x04\x01\x80\x80` \x01\x82\x81\x03\x82R`\x0b\x81R` \x01\x80\x7f!authorized\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x81RP` \x01\x91PP`@Q\x80\x91\x03\x90\xfd[`\x03`\x00\x90T\x90a\x01\x00\n\x90\x04`\xff\x16a\x03\xc9W`\x00\x80\xfd[3s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16a\x08\xfcG\x90\x81\x15\x02\x90`@Q`\x00`@Q\x80\x83\x03\x81\x85\x88\x88\xf1\x93PPPP\x15\x80\x15a\x04\x0fW=`\x00\x80>=`\x00\xfd[P`\x00[`\x02\x80T\x90P\x81\x10\x15a\x04\xafW`\x00`\x02\x82\x81T\x81\x10a\x04/W\xfe[\x90`\x00R` `\x00 \x01`\x00\x90T\x90a\x01\x00\n\x90\x04s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x90P`\x00`\x01`\x00\x83s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x81R` \x01\x90\x81R` \x01`\x00 \x81\x90UPP\x80\x80`\x01\x01\x91PPa\x04\x13V[P`\x00g\xff\xff\xff\xff\xff\xff\xff\xff\x81\x11\x80\x15a\x04\xc8W`\x00\x80\xfd[P`@Q\x90\x80\x82R\x80` \x02` \x01\x82\x01`@R\x80\x15a\x04\xf7W\x81` \x01` \x82\x02\x806\x837\x80\x82\x01\x91PP\x90P[P`\x02\x90\x80Q\x90` \x01\x90a\x05\r\x92\x91\x90a\x06\xacV[PV[`\x01` R\x80`\x00R`@`\x00 `\x00\x91P\x90PT\x81V[`\x00\x80T\x90a\x01\x00\n\x90\x04s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x81V[`\x03`\x00\x90T\x90a\x01\x00\n\x90\x04`\xff\x16a\x05eW`\x00\x80\xfd[`\x004\x11a\x05\xbeW`@Q\x7f\x08\xc3y\xa0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x81R`\x04\x01\x80\x80` \x01\x82\x81\x03\x82R`#\x81R` \x01\x80a\x07r`#\x919`@\x01\x91PP`@Q\x80\x91\x03\x90\xfd[4`\x01`\x003s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x81R` \x01\x90\x81R` \x01`\x00 `\x00\x82\x82T\x01\x92PP\x81\x90UP`\x023\x90\x80`\x01\x81T\x01\x80\x82U\x80\x91PP`\x01\x90\x03\x90`\x00R` `\x00 \x01`\x00\x90\x91\x90\x91\x90\x91a\x01\x00\n\x81T\x81s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x02\x19\x16\x90\x83s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x02\x17\x90UPV[`\x02\x81\x81T\x81\x10a\x06}W\xfe[\x90`\x00R` `\x00 \x01`\x00\x91PT\x90a\x01\x00\n\x90\x04s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x81V[\x82\x80T\x82\x82U\x90`\x00R` `\x00 \x90\x81\x01\x92\x82\x15a\x07%W\x91` \x02\x82\x01[\x82\x81\x11\x15a\x07$W\x82Q\x82`\x00a\x01\x00\n\x81T\x81s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x02\x19\x16\x90\x83s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x16\x02\x17\x90UP\x91` \x01\x91\x90`\x01\x01\x90a\x06\xccV[[P\x90Pa\x072\x91\x90a\x076V[P\x90V[[\x80\x82\x11\x15a\x07mW`\x00\x81\x81a\x01\x00\n\x81T\x90s\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x02\x19\x16\x90UP`\x01\x01a\x077V[P\x90V\xfeFund amount must be greater than 0.\xa2dipfsX"\x12  \xbcL\xe7gG\xd6\x88*\xee\x91\x96\x8b\x7fV\xed\xdb\xe5 \xd4\x0eJ\xb6\x13s\x97w8\xe6w.\x92dsolcC\x00\x06\x0c\x003', 'accessList': [], 'nonce': 0, 'gas': 537114, 'maxPriorityFeePerGas': 875000000, 'maxFeePerGas': 0, 'type': 2}

This is the return value from hash():

b' U:\xe9\x00\xe3O\xfc\\>\x82\x92\x97\xaf9\x9d\x1a\xed\x0f)\xc6%\xbf\\\x08\xc4\x95\t\xce.\xcc7'

Environment

# run this:
$ python -m eth_utils

# then copy the output here:
Python version:
3.7.9 (v3.7.9:13c94747c7, Aug 15 2020, 01:31:08) 
[Clang 6.0 (clang-600.0.57)]

Operating System: Darwin-21.3.0-x86_64-i386-64bit

pip freeze result:
aiohttp==3.8.1
aiosignal==1.2.0
alabaster==0.7.12
-e git+ssh://git@github.com/ApeWorX/ape-ledger.git@a7d0fbc2ba083f75054e02fe9a9622d5c895acbc#egg=ape_ledger
ape-solidity==0.1.0b5
appnope==0.1.2
argcomplete==1.12.3
async-timeout==4.0.2
asynctest==0.13.0
attrs==21.4.0
Babel==2.9.1
backcall==0.2.0
backports.cached-property==1.0.1
base58==2.1.1
bitarray==1.2.2
black==21.12b0
bleach==4.1.0
cached-property==1.5.2
certifi==2021.10.8
cffi==1.15.0
cfgv==3.3.1
charset-normalizer==2.0.11
click==8.0.3
colorama==0.4.4
commitizen==2.19.0
commonmark==0.9.1
coverage==6.3.1
cytoolz==0.11.2
dataclassy==0.10.4
decli==0.5.2
decorator==5.1.1
Deprecated==1.2.13
distlib==0.3.4
docopt==0.6.2
docutils==0.16
ecdsa==0.17.0
eip712==0.1.0
eth-abi==2.1.1
eth-account==0.5.7
-e git+ssh://git@github.com/unparalleled-js/ape.git@93f5987b50ff5964f32f73d5dd5f4562d7ace691#egg=eth_ape
eth-bloom==1.0.4
eth-hash==0.3.2
eth-keyfile==0.5.1
eth-keys==0.3.4
eth-rlp==0.2.1
eth-tester==0.6.0b6
eth-typing==2.3.0
eth-utils==1.10.0
ethpm-types==0.1.0b7
execnet==1.9.0
fastecdsa==2.2.3
filelock==3.4.2
flake8==3.9.2
flake8-breakpoint==1.1.0
flake8-plugin-utils==1.3.2
flake8-print==4.0.0
frozenlist==1.3.0
hexbytes==0.2.2
hidapi==0.10.1
hypothesis==6.36.1
hypothesis-jsonschema==0.19.0
identify==2.4.8
idna==3.3
imagesize==1.3.0
importlib-metadata==4.10.1
iniconfig==1.1.1
ipdb==0.13.9
ipfshttpclient==0.8.0a2
ipython==7.31.1
isort==5.10.1
jedi==0.18.1
Jinja2==3.0.3
jsonschema==3.2.0
keyring==23.5.0
lru-dict==1.1.7
markdown-it-py==2.0.1
MarkupSafe==2.0.1
matplotlib-inline==0.1.3
mccabe==0.6.1
mdit-py-plugins==0.3.0
mdurl==0.1.0
mpmath==1.2.1
multiaddr==0.0.9
multidict==6.0.2
mypy==0.931
mypy-extensions==0.4.3
myst-parser==0.16.0
netaddr==0.8.0
nodeenv==1.6.0
numpy==1.21.5
packaging==20.9
pandas==1.3.5
pandas-stubs==1.2.0.49
parsimonious==0.8.1
parso==0.8.3
pathspec==0.9.0
pexpect==4.8.0
pickleshare==0.7.5
pkginfo==1.8.2
platformdirs==2.4.1
pluggy==0.13.1
pockets==0.9.1
pre-commit==2.17.0
prompt-toolkit==3.0.26
protobuf==3.19.4
ptyprocess==0.7.0
py==1.11.0
py-ecc==4.1.0
py-evm==0.5.0a3
py-geth==3.7.0
py-solc-x==1.1.1
pycodestyle==2.7.0
pycparser==2.21
pycryptodome==3.14.1
pydantic==1.9.0
pyethash==0.1.27
pyflakes==2.3.1
pygit2==1.8.0
PyGithub==1.55
Pygments==2.11.2
PyJWT==2.3.0
PyNaCl==1.5.0
pyparsing==3.0.7
pyrsistent==0.18.1
pysha3==1.0.2
pytest==6.2.5
pytest-cov==3.0.0
pytest-forked==1.4.0
pytest-mock==3.7.0
pytest-watch==4.2.0
pytest-xdist==2.5.0
python-dateutil==2.8.2
pytz==2021.3
PyYAML==5.4.1
questionary==1.10.0
readme-renderer==32.0
requests==2.27.1
requests-toolbelt==0.9.1
rfc3986==2.0.0
rich==10.16.2
rlp==2.0.1
semantic-version==2.8.5
setuptools-scm==6.4.2
singledispatchmethod==1.0
six==1.16.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
Sphinx==3.5.4
sphinx-click==3.1.0
sphinx-rtd-theme==0.5.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-napoleon==0.7
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
sympy==1.9
termcolor==1.1.0
tokenlists==0.1.1
toml==0.10.2
tomli==1.2.3
tomlkit==0.9.0
toolz==0.11.2
tqdm==4.62.3
traitlets==5.1.1
trie==2.0.0a5
twine==3.8.0
typed-ast==1.5.2
types-PyYAML==6.0.4
types-requests==2.27.8
types-urllib3==1.26.9
typing-extensions==3.10.0.2
urllib3==1.26.8
varint==1.0.2
virtualenv==20.13.0
vvm==0.1.0
watchdog==2.1.6
wcwidth==0.2.5
web3==5.27.0
webencodings==0.5.1
websockets==9.1
wrapt==1.13.3
yarl==1.7.2
zipp==3.7.0
antazoey commented 2 years ago

Also note that I need to have this transaction encoded while it is unsigned because I am sending it off to my Ledger device to get signed and the device expects the transaction to be RLP encoded.

kclowes commented 2 years ago

Thanks for raising this. Allowing an unsigned DynamicFeeTransaction looks like it's a gap in our functionality. I'll leave this open to track.

The hash I am getting does not seem right at all

What value do you expect?

I did some testing, and found a couple of workarounds:

from eth_account._utils.legacy_transactions import serializable_unsigned_transaction_from_dict
h = serializable_unsigned_transaction_from_dict(txn).hash() # returns b' U:\xe9\x00\xe3O\xfc\\>\x82\x92\x97\xaf9\x9d\x1a\xed\x0f)\xc6%\xbf\\\x08\xc4\x95\t\xce.\xcc7', same as yours above
HexBytes(h) # returns HexBytes('0x20553ae900e34ffc5c3e829297af399d1aed0f29c625bf5c08c49509ce2ecc37')
antazoey commented 2 years ago

Hello @kclowes ! Thank you for the response and for looking this so much :)

What value do you expect?

I am not 100% sure on the exact value, however it does not seem like enough bytes. All the examples I have seen were 98 characters long, and the one I am getting is 66 characters long, so that made me think that this is not the right hash I need.

Here is an example in the Ledger app-ethereum of an already-encoded EIP-1559 hash ready to be signed: https://github.com/LedgerHQ/app-ethereum/pull/273/files

Also, I am working with Mike Shultz a little and testing out his PR: https://github.com/LedgerHQ/app-ethereum/pull/273/files Setting breakpoints when using that lib, I can see the hashes look like the example in appe-ethereum.

The hashes I get from eth-account here are not accepted by the Ledger device either.

And unsigned transactions are working great, thanks for the warning on the usage of the protected _utils namespace.

kclowes commented 2 years ago

Hi :wave: ! No problem! I'm a little confused. To clarify, the UnsignedTransaction is working correctly but needs legacy gas fields. But you think DynamicFeeTransaction.from_dict(txn).hash() returns the wrong value?

The string you linked to in that PR (https://github.com/LedgerHQ/app-ethereum/pull/273/files) looks to me like an rlp encoding, rather than a keccak hash. A keccak hash is what is returned from WhateverTransaction.from_dict(txn).hash(). Is the value you're handing to the Ledger an rlp encoded transaction or a keccak hash of the rlp encoded transaction? If it's the rlp encoded transaction, you may be able to use pyrlp directly. Docs to encode custom objects are here. Let me know if (what) I'm misunderstanding!

antazoey commented 2 years ago

Is the value you're handing to the Ledger an rlp encoded transaction or a keccak hash of the rlp encoded transaction?

Yes, I am looking for the RLP. That explains my confusion! I was definitely the one with the misunderstanding but now I know. I can try to use the pyrlp lib.

This ticket can probably be closed, unless anything can come out of this.

I do find serialized_unsigned_transaction_from_dict() to be a bit strange because the legacy txn is returns is RLP encode-able but the typed transaction it returns is not (no sedes).

kclowes commented 2 years ago

Glad you figured it out :boom:!

I'm going to leave this open for now because there are some inconsistencies I'd like to look into, like returning an RLP encodable TypedTransaction, returning hash as either HexBytes or a bytestring consistently.

antazoey commented 2 years ago

It was weird because these transaction classes look like RLP encode-able transactions, but yeah I think they just assume signed? I am not sure! But I can use the unsigned legacy transaction in rlp.encode() but I can't use the unsigned EIP-1559 transactions that way.

For now, I just made my own rlp objects, but they look a like the ones in this package, so I am not sure what exactly is going on. Is this just because of the v r s fields?

kclowes commented 2 years ago

Yeah, I think the TypedTransactions do assume they're signed. I don't see a way to work with unsigned TypedTransactions, but I also only have surface-level knowledge of this part of the library, so I could be wrong. We do some mucking around with the access list parameters (converting them from a dict to a list and vice versa) and so I wonder if the timing of that is the problem - maybe they're getting passed to rlp as a dict which doesn't have sedes?

fselmo commented 2 years ago

💯 ... Thanks for the conversation around this.

I think there's a lot to be improved in this process for sure. Having recently dived into rlp encoding for typed transactions, it was very difficult to navigate as it currently stands. Might be nice to separate these different ideas into bite-sized issues to better track them. I can start doing that and touch base to see if I missed anything.

fselmo commented 2 years ago

One thing that could currently work, and I think is what you're looking for unless I'm totally mistaken, is... using your example above:

import rlp
from eth.vm.forks.london.transactions import LondonTransactionBuilder, DynamicFeeTransaction

# notice the snake case since it's a py-evm model :(
tx = {
    'chain_id': 61,
    'to': b'',
    'value': 0,
    'data': b'`\x80`@R4\x80\x15a\x00\x10W`\x00\x80\xfd[P3`\x00\x00\x00...,
    'access_list': [],
    'nonce': 0,
    'gas': 537114,
    'max_priority_fee_per_gas': 875000000,
    'max_fee_per_gas': 0,

    # notice no `type` here since that would raise:
    # 'type': 2 
}

unsigned_tx = LondonTransactionBuilder.new_unsigned_dynamic_fee_transaction(**tx)

just_the_rlp_part = rlp.encode(unsigned_tx)  # rlp(tx) -- in bytes
type_with_rlp = unsigned_tx.get_message_for_signing()  # 0x02 || rlp(tx) -- in bytes
pacrob commented 2 months ago

Closing as stale