search

ethereum / web3.py

A python interface for interacting with the Ethereum blockchain and ecosystem.
http://web3py.readthedocs.io
MIT License
4.94k stars 1.69k forks source link

Extend Web3.toJSON json encoder (Web3JsonEncoder) to support bytes type #2584

Closed abarbatei closed 11 months ago

abarbatei commented 2 years ago 
aiohttp==3.7.4.post0
aiosignal==1.2.0
altgraph==0.17.2
ansicon==1.89.0
appdirs==1.4.4
async-timeout==3.0.1
atomicwrites==1.4.1
attrs==20.3.0
Automat==20.2.0
bankreader==0.2.3
base58==2.1.1
beautifulsoup4==4.9.3
bitarray==1.2.2
black==22.6.0
blessed==1.19.0
Brotli==1.0.9
bscscan-python==2.0.0
certifi==2020.12.5
cffi==1.14.5
cfscrape==2.1.1
chardet==4.0.0
charset-normalizer==2.0.9
click==8.0.3
cloudscraper==1.2.60
colorama==0.4.4
coloredlogs==15.0.1
colorlog==6.6.0
config-checker==0.0.1
constantly==15.1.0
cryptography==3.4.7
cssselect==1.1.0
cssselect2==0.4.1
cytoolz==0.11.2
db-ops==0.1.1
decompyle3==3.8.0
discord.py-self==1.9.2
docopt==0.6.2
et-xmlfile==1.1.0
eth-abi==2.1.1
eth-account==0.5.7
eth-hash==0.3.2
eth-keyfile==0.5.1
eth-keys==0.3.4
eth-rlp==0.2.1
eth-typing==2.2.2
eth-utils==1.10.0
etherscan-python==2.1.0
ethplorer-py==1.0
ezenv==0.92
Flask==2.1.3
fonttools==4.23.1
frozenlist==1.2.0
future==0.18.2
h2==3.2.0
halo==0.0.31
hexbytes==0.2.2
hpack==3.0.0
html5lib==1.1
humanfriendly==10.0
hyperframe==5.2.0
hyperlink==21.0.0
idna==2.10
img2pdf==0.4.1
importlib-metadata==4.12.0
incremental==21.3.0
influxdb==5.3.1
iniconfig==1.1.1
inquirer==2.8.0
install==1.3.5
ipfshttpclient==0.8.0a2
itemadapter==0.2.0
itemloaders==1.0.4
itsdangerous==2.1.2
Jinja2==3.1.2
jinxed==1.1.0
jmespath==0.10.0
jsonschema==3.2.0
keyboard==0.13.5
log-symbols==0.0.14
lru-dict==1.1.7
lxml==4.6.3
markdown2==2.4.0
MarkupSafe==2.1.1
md2pdf==0.5
morse-talk==0.2
msgpack==1.0.2
multiaddr==0.0.9
multidict==5.2.0
mypy-extensions==0.4.3
netaddr==0.8.0
numpy==1.20.3
oauthlib==3.1.1
openpyxl==3.0.7
packaging==21.3
pandas==1.2.4
panoramix-decompiler-abi==0.5.2
parsel==1.6.0
parsimonious==0.8.1
pathspec==0.9.0
pefile==2021.9.3
pika==1.3.0
pika-stubs==0.1.3
pikepdf==2.12.0
Pillow==8.2.0
platformdirs==2.5.2
pluggy==1.0.0
priority==1.3.0
prompt-toolkit==1.0.14
Protego==0.1.16
protobuf==3.19.1
py==1.11.0
pyaes==1.6.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycoingecko==1.4.1
pycparser==2.20
pycryptodome==3.12.0
PyDispatcher==2.0.5
pydyf==0.0.3
Pygments==2.10.0
PyInquirer==1.0.3
pyinstaller==4.9
pyinstaller-hooks-contrib==2022.2
pymongo==4.2.0
pyonepassword==2.0.0
pyOpenSSL==20.0.1
pyparsing==3.0.8
Pyphen==0.10.0
pyreadline3==3.3
pyRFC3339==1.1
pyrsistent==0.18.0
PySocks==1.7.1
pytest==7.1.2
pytest-mock==3.8.2
python-dateutil==2.8.1
python-editor==1.0.4
pytz==2021.1
pywin32==302
pywin32-ctypes==0.2.0
PyYAML==6.0
qrcode==7.3.1
queuelib==1.5.0
readchar==2.0.1
regex==2021.11.10
requests==2.26.0
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
rlp==2.0.1
rsa==4.8
Scrapy==2.5.0
scribd-downloader==1.3.1
selenium==3.141.0
service-identity==18.1.0
six==1.15.0
soupsieve==2.2.1
spark-parser==1.8.9
spinners==0.0.24
Telethon==1.24.0
termcolor==1.1.0
timeout-decorator==0.5.0
tinycss2==1.1.0
tomli==2.0.1
toolz==0.11.2
tweepy==3.10.0
Twisted==21.2.0
twisted-iocpsupport==1.0.1
typing_extensions==4.0.1
uncompyle6==3.8.0
urllib3==1.26.4
varint==1.0.2
w3lib==1.22.0
wcwidth==0.2.5
weasyprint==53.0b1
web3==5.30.0
webencodings==0.5.1
websockets==9.1
Werkzeug==2.1.2
wget==3.2
whale-alert==0.0.4
xdis==6.0.3
xlrd==2.0.1
xortool==1.0.1
yarl==1.7.2
zipp==3.8.0
zope.interface==5.3.0
zopfli==0.1.8

What was wrong?

While attempting to convert some filtered events of LogReceipt type (https://github.com/ethereum/web3.py/blob/master/web3/types.py#L164 as returned by https://web3py.readthedocs.io/en/latest/filters.html#web3.utils.filters.Filter.get_new_entries) to json using Web3.toJSON, I encountered events with bytes as type, that resulted in an TypeError exception being thrown:

Traceback (most recent call last):
  File "C:\Python\lib\site-packages\web3\_utils\encoding.py", line 206, in _friendly_json_encode
    encoded = json.dumps(obj, cls=cls)
  File "C:\Python\lib\json\__init__.py", line 234, in dumps
    return cls(
  File "C:\Python\lib\json\encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Python\lib\json\encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "C:\Python\lib\site-packages\web3\_utils\encoding.py", line 285, in default
    return json.JSONEncoder.default(self, obj)
  File "C:\Python\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

I would consider extending the Web3.toJSON used JsonEncoder (https://github.com/ethereum/web3.py/blob/master/web3/_utils/encoding.py#L286) to also support bytes.

How can it be fixed?

Resolving this can be very easy just by reusing the available types and adding this check in https://github.com/ethereum/web3.py/blob/master/web3/_utils/encoding.py#L290 :

if isinstance(obj, bytes):
    return HexStr(HexBytes(obj).hex())

My reasoning is that, a specific toJSON function for web3py should be able to process anything that other web3py APIs return (TypeDicts or AttributeDict at least).

Can be reproduced by parsing the OrderFulfilled event from the (OpenSea: Seaport 1.1 - Etherscan contract 0x00000000006c3852cbEf3e08E8dF289169EdE581) for example.

Note: We prefer to use issues to track our work. If you think you've encountered a bug in web3py or have a feature request, you're in the right place. If you have implementation or usage questions, please refer to our documentation and/or join the conversation on discord.

abarbatei commented 11 months ago

thank you!

fselmo commented 11 months ago

@abarbatei merged and released! You can find this feature in the newly released v6.10.0.