search

ethereum / web3.py

A python interface for interacting with the Ethereum blockchain and ecosystem.
http://web3py.readthedocs.io
MIT License
4.89k stars 1.67k forks source link

HTTPProvider timeout is not working for chunked responses #3418

Open Uxio0 opened 2 weeks ago

Uxio0 commented 2 weeks ago

What happened?

This is a problem with urllib3 and not with web3.py. If server answers with a chunked response, timeout will not be respected. More info:

Code that produced the error

from web3 import Web3
w3 = Web3(Web3.HTTPProvider("http://127.0.0.1:8545", request_kwargs={'timeout': 60}))

A big query to `tracing.trace_filter` using an Erigon node. Erigon answers with a `Transfer-Encoding: chunked` header and starts slowly streaming the result, not letting the timeout trigger, query takes more than 10 minutes.

Full error output

No error. Code hangs when `Web3.py` does the request until Erigon (or other RPC) ends streaming the response.

Fill this section in if you know how this could or should be fixed

Using requests.post(url, json=data, stream=True) would allow Web3.py to control for how long the request is stuck, and process the chunked data in a more optimal way. Also, if the data is not chunked it shouldn't be an issue at all.

web3 Version

6.19.0

Python Version

3.12.4

Operating System

linux

Output from pip freeze

aiohttp==3.9.5
aiosignal==1.3.1
amqp==5.2.0
asgiref==3.7.2
astroid==3.1.0
asttokens==2.4.1
attrs==23.2.0
billiard==4.2.0
bitarray==2.9.2
black==24.4.2
boto3==1.34.117
botocore==1.34.126
CacheControl==0.14.0
cached-property==1.5.2
cachetools==5.3.3
celery==5.4.0
certifi==2024.2.2
cffi==1.16.0
cfgv==3.4.0
charset-normalizer==3.3.2
ckzg==1.0.1
click==8.1.7
click-didyoumean==0.3.1
click-plugins==1.1.1
click-repl==0.3.0
coverage==7.5.3
cron-descriptor==1.4.3
cryptography==42.0.5
cytoolz==0.12.3
debugpy==1.8.1
decorator==5.1.1
dill==0.3.8
distlib==0.3.8
Django==5.0.6
django-appconf==1.0.6
django-cache-memoize==0.2.0
django-celery-beat==2.6.0
django-cors-headers==4.3.1
django-db-geventpool==4.0.5
django-debug-toolbar==4.3.0
django-debug-toolbar-force==0.2
django-environ==0.11.2
django-extensions==3.2.3
django-filter==24.2
django-imagekit==5.0.0
django-model-utils==4.5.1
django-nine==0.2.7
django-redis==5.4.0
django-s3-storage==0.15.0
django-stubs==5.0.0
django-stubs-ext==5.0.0
django-test-migrations==1.3.0
django-timezone-field==6.1.0
djangorestframework==3.15.1
djangorestframework-camel-case==1.4.2
docker==7.0.0
docutils==0.21.2
drf-yasg==1.21.7
eth-account==0.11.2
eth-bloom==3.0.1
eth-hash==0.7.0
eth-keyfile==0.8.1
eth-keys==0.5.1
eth-rlp==1.0.1
eth-typing==4.1.0
eth-utils==4.1.0
eth_abi==5.1.0
executing==2.0.1
factory-boy==3.3.0
Faker==25.8.0
filelock==3.14.0
firebase-admin==6.5.0
flake8==7.0.0
flower==2.0.1
frozenlist==1.4.1
gevent==24.2.1
google-api-core==2.19.0
google-api-python-client==2.127.0
google-auth==2.29.0
google-auth-httplib2==0.2.0
google-cloud-core==2.4.1
google-cloud-firestore==2.16.0
google-cloud-storage==2.16.0
google-crc32c==1.5.0
google-resumable-media==2.7.0
googleapis-common-protos==1.63.0
greenlet==3.0.3
grpcio==1.63.0
grpcio-status==1.62.2
gunicorn==22.0.0
hexbytes==0.3.1
hiredis==2.3.2
httplib2==0.22.0
humanize==4.9.0
identify==2.5.36
idna==3.7
inflection==0.5.1
iniconfig==2.0.0
ipdb==0.13.13
ipython==8.24.0
isort==5.13.2
jedi==0.19.1
jmespath==1.0.1
jsonschema==4.22.0
jsonschema-specifications==2023.12.1
kombu==5.3.7
lru-dict==1.2.0
matplotlib-inline==0.1.7
mccabe==0.7.0
msgpack==1.0.8
multidict==6.0.5
mypy==1.10.0
mypy-extensions==1.0.0
nodeenv==1.8.0
packaging==24.0
parsimonious==0.10.0
parso==0.8.4
pathspec==0.12.1
pexpect==4.9.0
pika==1.3.2
pilkit==3.0
pillow==10.3.0
platformdirs==4.2.1
pluggy==1.5.0
pre-commit==3.7.0
prometheus_client==0.20.0
prompt-toolkit==3.0.43
proto-plus==1.23.0
protobuf==4.25.3
psutil==5.9.8
psycogreen==1.0.2
psycopg2==2.9.9
ptyprocess==0.7.0
pur==7.3.1
pure-eval==0.2.2
py==1.11.0
py-ecc==7.0.1
py-evm==0.10.1b1
pyasn1==0.6.0
pyasn1_modules==0.4.0
pycodestyle==2.11.1
pycparser==2.22
pycryptodome==3.20.0
pyflakes==3.2.0
Pygments==2.17.2
PyJWT==2.8.0
pylint==3.1.0
pylint-django==2.5.5
pylint-plugin-utils==0.8.2
pyparsing==3.1.2
pytest==8.2.1
pytest-celery==1.0.0
pytest-django==4.8.0
pytest-docker-tools==3.1.3
pytest-env==1.1.3
pytest-rerunfailures==14.0
pytest-sugar==1.0.0
python-crontab==3.0.0
python-dateutil==2.9.0.post0
pytz==2024.1
pyunormalize==15.1.0
PyYAML==6.0.1
redis==5.0.6
referencing==0.35.1
regex==2024.4.28
requests==2.32.3
retry==0.9.2
rlp==4.0.1
rpds-py==0.18.0
rsa==4.9
s3transfer==0.10.1
safe-eth-py==6.0.0b31
safe-pysha3==1.0.4
setuptools==69.5.1
six==1.16.0
sortedcontainers==2.4.0
sqlparse==0.5.0
stack-data==0.6.3
swagger-spec-validator==3.0.3
termcolor==2.4.0
tomlkit==0.12.4
toolz==0.12.1
tornado==6.4
traitlets==5.14.3
trie==3.0.1
types-pytz==2024.1.0.20240417
types-PyYAML==6.0.12.20240311
typing_extensions==4.11.0
tzdata==2024.1
uritemplate==4.1.1
urllib3==2.2.1
vine==5.1.0
virtualenv==20.26.1
wcwidth==0.2.13
web3==6.19.0
websockets==12.0
wheel==0.43.0
yarl==1.9.4
zope.event==5.0
zope.interface==6.3
kclowes commented 2 weeks ago

You should be able to pass in stream=True as a request_kwarg to the provider. If that doesn't work and we need to dig in further, please reopen :)

Uxio0 commented 2 weeks ago

@kclowes Even if I pass stream=True, as the response is not iterated and just parsed as a not streamed one in Web3.py that makes no difference. Web3.py is blocked until all the chunks in the response are received and parsed

Uxio0 commented 2 weeks ago

(I don't have permissions to reopen the issue)

kclowes commented 2 weeks ago

Ah, interesting. I'll take a deeper look. Thanks!

Uxio0 commented 1 week ago

I feel maybe https://github.com/ethereum/web3.py/issues/1368 can be implemented also when fixing this, as the timeout can be used in this case. Let me see if I can get some time to draft a PR, at least a PoC of my idea

Uxio0 commented 1 week ago

@kclowes this is a very quick PoC that I think illustrates my idea: https://github.com/ethereum/web3.py/pull/3428