Open fridary opened 4 months ago
Hey @fridary.
Could you share logs having both moments when request is served OK and then served with error?
You could send your requests in batches and set --rpc.batch.limit=x
for erigon.
@awskii thanks for reply. I searched for how to close connections for web3py and have found no information about it. So I guess there is no way how to close connections. Is that right? Here is example with pure http request on python that I do:
requests.post(testnet, json={"method":"eth_getTransactionByHash","params":[hash_],"id":1,"jsonrpc":"2.0"}, headers={"Content-Type": "application/json"}).json()['result']
and w3 examples with contract data:
w3 = Web3(Web3.HTTPProvider('http://127.0.0.1:8545/?t'))
contract = w3.eth.contract(address=address, abi=abi_address)
balance = contract.functions.balanceOf(pair_address).call(block_identifier=block)
Yes, for sure I am not referring old blocks. Moreover, in case of old block there will be an error block was not found, but not an error like mine with fully exiting application.
I will try send logs both ways later.
Erigon version:
erigon version 2.59.3-088fd8ef
OS & Version: Ubuntu 20.04.6 LTS, CPU E5-2695 v4, 72 processes, 192 Gb RAM Disk: NVMe HPE MT006400KWHAE 6.4 Tb Erigon Command (with flags/config):ExecStart=/home/root/erigon/build/bin/erigon --internalcl --datadir=/disk_sde/erigon --http.api=eth,erigon,engine,web3,net,debug,trace,txpool --authrpc.jwtsecret=/home/root/erigon/jwtsecret --metrics --prune.h.before=13916166 --prune.r.before=13916166 --prune.t.before=13916166 --prune.c.before=13916166 --torrent.download.rate=128mb
Chain/Network: mainnetError
Steps to reproduce the behaviour
I do high load multiprocessing in 72 processes (72 parallel jobs) to query Ethereum node with RPC.
web3py
lib with multiprocessing, that's why every request I do with a new connection and unique http url likew3 = Web3(Web3.HTTPProvider(f"{testnet}/?{''.join(random.choices(string.ascii_letters + string.digits, k=20))}"))
. So 1 mill requests make 1 millWeb3()
connections (there is no other choice).After around more than some millions requests in 15 mins I get error above. Any ideas what can be wrong? Maybe I need to change Erigon flags to allow more requests? Or problem with reading disk database so often?
Logs
Last 200 logs during executing.