Closed kimroniny closed 4 months ago
We don't currently support multiprocessing. You may be able to sort of work around this by using two separate instances of w3, however.
We don't currently support multiprocessing. You may be able to sort of work around this by using two separate instances of w3, however.
@kclowes
The problem is that even though I use two seperate instances of w3 in different processes, this ERROR still exists just like the above example.
Though it doesn't support multiprocessing, it shouldn't occur any errors when using seperate instances in different processes.
The key BUG is about when to initialize SimpleCache()
and how to set the cache key. The cache key is only identical to the threading identifier. That's NOT enough in python.
Got it. We'll triage and put it in our queue. Thanks for reporting!
@kimroniny this is not reproducible, though I agree that a session cache should exist per provider instance even. If you have another reproducible example, it would help quite a bit. I did start a PR towards strapping a session cache manager to each instance of an http provider (or async http provider) which I think is the most reasonable approach and I hope that solves whatever problem you are seeing.
However, on the current v6
branch, with no changes to the session caching, I am getting different sessions:
Process: MainProcess
cache: odict_items([('46b10f397f77283a83ce39cad6f8520b', <requests.sessions.Session object at 0x114b0c1d0>)])
Process: Process-2
cache: odict_items([('46b10f397f77283a83ce39cad6f8520b', <requests.sessions.Session object at 0x10e1847d0>)])
Process: Process-1
cache: odict_items([('46b10f397f77283a83ce39cad6f8520b', <requests.sessions.Session object at 0x104eb05d0>)])
with the following code:
from web3 import Web3, HTTPProvider
import multiprocessing
from web3._utils.request import (
_session_cache
)
def conn():
w3 = Web3(HTTPProvider(endpoint_uri=HTTP_LOCAL, request_kwargs={"timeout": 1000}))
w3.eth.block_number
print(f"Process: {multiprocessing.current_process().name}")
print(f"cache: {_session_cache._data.items()}")
if __name__ == "__main__":
conn()
proc1 = multiprocessing.Process(target=conn)
proc2 = multiprocessing.Process(target=conn)
proc1.start()
proc2.start()
proc1.join()
proc2.join
Similar results on main
branch (v7) and similar results on the refactor PR I created. Please provide a more clear and reproducible code snippet if you can.
pip freeze
outputWhat was wrong?
Please include any of the following that are applicable:
The full output of the error
What type of node you were connecting to.
How can it be fixed?
explain the output
I print session id in the format
"{current_process().name}, session: {id(session)}"
.The exception happens when querying
eth.accounts
. The response shoule betype: List[Account]
, while it istype: bool
which should be the response type ofunlock_account
.The detailed exception line is the code. It has obtained the response but faces the error type.
reason
When sending requests, web3 uses the cached session with
SimpleCache()
. 'SimpleCache()' is instantiated upon importing the web3 and won't be re-instantiated whenever creating a newWeb3()
.In the main process, after executing
conn()
the first session is cached. When forking the children process, all objects including the cached session in the main process are also forked. However, the underlying connections of this session will be shared across all processes, which could cause the responses mixed up in different processes.It is an absolutely dangerous BUG !!!
solution
The easiest solution is to change the cache key. Now the key is just tied to the thread identifier which is identical in the above codes. Just add the process identifier into the key.
Solution 1 is not the best because all process are still using the same one
SimpleCache()
. I think, it should check whether it is in a new forked process or not. If it is, then a new SimpleCache()` should be instantiated.