Closed fridary closed 2 months ago
Update: I figured out that if I don't declare w3
outside the calculate_transaction()
function, I mean if I remove this code:
w3 = Web3(Web3.HTTPProvider(testnet))
if not w3.is_connected():
exit("w3 not connected")
and if I declare w3
each time/process in calculate_transaction()
, error will gone. So the problem is in defining w3
in/outside pool function. But how to fix I don't know, because in real program I have to define w3
also outside the pool processing.
The only reason could be to define w3
once at start and send w3
in func params like def calculate_transaction(w3, i, hash_)
, but then I will get this error:
Traceback (most recent call last):
File "/home/fridary/python/ethereum/parallel_4.py", line 53, in <module>
pool.starmap_async(calculate_transaction, params).get()
File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 774, in get
raise self._value
File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
put(task)
File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'construct_web3_formatting_middleware.<locals>.formatter_middleware'
Maybe there is any way to close connection w3
?
Update: this solution works, but it's ugly option. Creating new w3
every step on another address. Any other ideas?
testnet = 'http://127.0.0.1:8545/'
def calculate_transaction(i):
w3 = Web3(Web3.HTTPProvider(testnet + f'?{i}'))
On first glance, it looks like something to do with how we generate the cache key and multiprocessing. I'm glad you found a workaround though. We'll put this in our queue to look into. Thanks for the issue!
@fridary when I run your example, I do error out but with the following message:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
To fix this issue, refer to the "Safe importing of main module"
section in https://docs.python.org/3/library/multiprocessing.html
When I do run with the if __name__ == '__main__':
pattern I do not get any errors. Try this quick and dirty refactor of your example to see if that resolves it:
from web3 import Web3
from multiprocessing import Pool, cpu_count
w3 = Web3(Web3.HTTPProvider(HTTP_LOCAL))
def calculate_transaction(i, hash_):
global w3
# gives error
transaction = w3.eth.get_transaction(hash_)
print(i, transaction['hash'].hex())
def main():
global w3
if not w3.is_connected():
exit("w3 not connected")
block = w3.eth.get_block('latest')
transactions = block['transactions'][:min(cpu_count(), 50)]
params = []
for i, hash_ in enumerate(transactions):
params.append((i, hash_))
pool = Pool(min(cpu_count(), 50))
pool.starmap_async(calculate_transaction, params).get()
pool.close()
if __name__ == "__main__":
main()
I'm going to close this as I can't reproduce it. If that does not resolve the issue and you see that something else is going on, please reach back out to re-open.
If it helps, my HTTP_LOCAL
is a Nethermind client.
What happened?
Erigon: 2.59.3-088fd8ef (last March 2024)
I have a problem to parallel querying local Erigon node. By the end of execution there is an error
ValueError: {'code': -32700, 'message': 'parse error'}
. Error is always returned on last request, whether I query 4 or 50 transactions inPool
. Although, if I test code on public node (https://eth-pokt.nodies.app
in example) instead of'http://127.0.0.1:8545'
, problem is gone. Yes, perhaps problem is in Erigon. But if I dorequests.post()
, there is no error. If I do another query likew3.eth.get_block(block)
, error is the same. Any ideas what's wrong? Update: it seems problem is in declaringw3
outside pool process together with inside (calculate_transaction()
) for local nodes.Erigon:
/home/fridary/erigon/build/bin/erigon --internalcl --datadir=/disk_sde/erigon --http.api=eth,erigon,engine,web3,net,debug,trace,txpool --authrpc.jwtsecret=/home/fridary/erigon/jwtsecret --metrics --prune.h.before=13916166 --prune.r.before=13916166 --prune.t.before=13916166 --prune.c.before=13916166 --torrent.download.rate=128mb
Code that produced the error
Full error output