ben-denham / labtech

Easily run experiment permutations with multi-processing and caching
https://ben-denham.github.io/labtech/
GNU General Public License v3.0
7 stars 1 forks source link

README example raises AttributeError: Can't pickle local object #1

Closed nathanjmcdougall closed 6 months ago

nathanjmcdougall commented 6 months ago

Hi Ben,

I have finally got a chance to give labtech a go for a situation where I think it's exactly what I need.

Unfortunately, on my Windows machine, I get this error trace when running the example in the README (as well as other cases I have tried):

Experiment: 0/25 [00:00, ?tasks/s] |                                                                        |   0% [? remaining]
Traceback (most recent call last):
  File "c:\Users\namc\Repositories\p-1017473-vuln-dist-convolution\src\scripts\labtechtest.py", line 43, in <module>
    results = lab.run_tasks(experiments)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\namc\Repositories\p-1017473-vuln-dist-convolution\.venv\Lib\site-packages\labtech\lab.py", line 474, in run_tasks
    return runner.run(tasks)
           ^^^^^^^^^^^^^^^^^
  File "C:\Users\namc\Repositories\p-1017473-vuln-dist-convolution\.venv\Lib\site-packages\labtech\lab.py", line 343, in run     
    future = future_executor.submit(self.run_or_load_task, process_name, task)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\namc\.pyenv\pyenv-win\versions\3.11.5\Lib\concurrent\futures\process.py", line 791, in submit
    self._adjust_process_count()
  File "C:\Users\namc\.pyenv\pyenv-win\versions\3.11.5\Lib\concurrent\futures\process.py", line 750, in _adjust_process_count    
    self._spawn_process()
  File "C:\Users\namc\.pyenv\pyenv-win\versions\3.11.5\Lib\concurrent\futures\process.py", line 768, in _spawn_process
    p.start()
  File "C:\Users\namc\.pyenv\pyenv-win\versions\3.11.5\Lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\namc\.pyenv\pyenv-win\versions\3.11.5\Lib\multiprocessing\context.py", line 336, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\namc\.pyenv\pyenv-win\versions\3.11.5\Lib\multiprocessing\popen_spawn_win32.py", line 94, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\namc\.pyenv\pyenv-win\versions\3.11.5\Lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'TaskRunner.get_executor.<locals>.init_worker'
PS C:\Users\namc\Repositories\p-1017473-vuln-dist-convolution> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\namc\.pyenv\pyenv-win\versions\3.11.5\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\namc\.pyenv\pyenv-win\versions\3.11.5\Lib\multiprocessing\spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
EOFError: Ran out of input
ben-denham commented 6 months ago

Thanks for trying labtech and providing this feedback @nathanjmcdougall !

This issue has now been fixed in: https://github.com/ben-denham/labtech/commit/e86367862fcf5c3ea830f539d33f3bc6967d7fa6

Please update to version 0.4.2, which includes this fix as well as some helpful documentation for using labtech from Python scripts on Windows and macOS: https://ben-denham.github.io/labtech/cookbook/#why-do-i-see-the-following-error-an-attempt-has-been-made-to-start-a-new-process-before-the-current-process-has-finished