search

cdanielmachado / carveme

CarveMe: genome-scale metabolic model reconstruction
Other
147 stars 50 forks source link

Error when multiprocessing with -r -- AttributeError: Can't pickle local object 'main.<locals>.f' #99

Closed silvtal closed 3 years ago

silvtal commented 3 years ago

Hi, I'm trying to use the -r option to parallelize model reconstruction, since I need to create hundreds of models. However, whenever I try to use it I get this error:

s$ carve -r iJ*.xml -o . Traceback (most recent call last): File "/home/urihs/.local/bin/carve", line 8, in sys.exit(main()) File "/home/urihs/.local/lib/python3.6/site-packages/carveme/cli/carve.py", line 432, in main p.map(f, args.input) File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get raise self._value File "/usr/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks put(task) File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) AttributeError: Can't pickle local object 'main..f'

Do you have any idea why this could be happening? Is it a bug?

Thank you in advance

cdanielmachado commented 3 years ago

Yes, it does seem to be a bug, probably related to our migration from python 2 to python 3.

Unfortunately, I am currently quite busy, and will only be able to work on this in about 2-3 weeks.

You can easily write your own script to call carveme multiple times, and if you run stuff on a cluster that's a lot more efficient than using the python multiprocessing library. I am actually considering to remove this '-r' option at some point.

silvtal commented 3 years ago

Ok, thank you! I'll explore alternatives then

EmmettPeng commented 3 years ago

Yes, it does seem to be a bug, probably related to our migration from python 2 to python 3.

Unfortunately, I am currently quite busy, and will only be able to work on this in about 2-3 weeks.

You can easily write your own script to call carveme multiple times, and if you run stuff on a cluster that's a lot more efficient than using the python multiprocessing library. I am actually considering to remove this '-r' option at some point.

Hi Daniel,

I know this issue has been closed but I wonder if this bug is fixed. I encountered the same bug using python3.7 and carveme v1.5.1. Can you re-check this problem? Thanks!

Emmett Peng