TypeError: cannot pickle '_thread.RLock' object

wardspan commented 3 years ago

Getting the same error - rebooted system - have tried pip install as well as installing from git clone - neither works

search.py started with 15 processing units...
Traceback (most recent call last):
  File "/usr/local/bin/onionsearch", line 33, in <module>
    sys.exit(load_entry_point('onionsearch==1', 'console_scripts', 'onionsearch')())
  File "/usr/local/lib/python3.9/site-packages/onionsearch/core.py", line 938, in scrape
    with Pool(units, initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),)) as p:
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object

v4lerio commented 3 years ago

Issue is with the Pool initializer using the progress bar at line 938

change the code from with Pool(units, initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),)) as p: to with Pool(units) as p:

and you still have the progress bar working for each search engine.

Please note that you may need to set your proxy from 127.0.0.1:9050 (default) to 127.0.0.1:9150

megadose commented 3 years ago

Okay, thank you, it's done https://github.com/megadose/OnionSearch/commit/f15606323f013e15735200193287114c0ce95aa9

megadose / OnionSearch

TypeError: cannot pickle '_thread.RLock' object #5