Pathos still giving pickle error

[NeelKanwal]

Hi,

I switches to Pathos.multiprocessing instead of multiprocessing but still getting a similar error with pickle dump.

Here is simplifed version of the code

def create_patches(location, file, path, 
    patch_folder, workers=1, patch_size=224, mask_overlap= 95.0):
    # Calculate something
    # pass input to next function.

  def extract_and_save_patch(x_cord, y_cord, file_path=file_pth, file_name=file, mask_path=mask_path,
                                 patch_folder=patch_folder, patch_size=patch_size, mask_overlap=mask_overlap):
          # extract a small peice of image based on x_cord and y_cord from big iage
          # save the image

    def multiprocess_passer(x_cord, y_cord):
        print(f"creating patches for ({x_cord},{y_cord})")
        extract_and_save_patch(x_cord, y_cord)

     list_cord = [(x, y) for x in range(0, n_across) for y in range(0, n_down)]
     with Pool(processes=workers) as p:
            p.map(extract_and_save_patch, list_cord)

create_patches(location, file, path, patch_folder, workers=num_workers)

I want to pass a tuple to function every time and leave other argument default which load the files and mask. But still getting the same error which appears in multiprocesisng library.

This the error trace.

Traceback (most recent call last):
  File "multiprocess.py", line 137, in <module>
    create_patches(location, file, path, patch_folder, workers=num_workers)
  File "multiprocess.py", line 130, in create_patches
    p.map(multiprocess_passer, list_cord)
  File "/home/neel/miniconda3/envs/process/lib/python3.7/site-packages/pathos/multiprocessing.py", line 135, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "/home/neel/miniconda3/envs/process/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/neel/miniconda3/envs/process/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/home/neel/miniconda3/envs/process/lib/python3.7/multiprocessing/pool.py", line 431, in _handle_tasks
    put(task)
  File "/home/neel/miniconda3/envs/process/lib/python3.7/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/neel/miniconda3/envs/process/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'starargs.<locals>.<lambda>'

Anyone faced a similar issue or solved it?

Thanks

[mmckerns]

What version of pathos are you using? What version of multiprocess and dill are you using? If I look at the line numbers in your traceback, they don't correspond with the current code for pathos. I also don't see how your code is falling back to multiprocessing instead of multiprocess. multiprocessing doesn't use dill, it uses pickle... and thus you'd see the picking error. Is multiprocess (and dill) installed? I don't see why your code should be using multiprocessing unless multiprocess isn't installed. Also, it'd be good if you could post a minimal working example so that I can actually test it and see if I get the same behavior. If I can reproduce what you have, it's easier to solve why it's occurring. Without a minimal working example, it's much harder to diagnose what is going on.

[dougwood]

I am also getting a can't pickle an object error using pathos ProcessPool . But my example is pretty complex, so it is very hard to create something smaller and reproducible.

I am using pathos 0.3.0 and multiprocess 0.70.14

The failure I am having is in trying to pickle an class from the module pylibczirw

I have included the traceback below if that is any help.

I do notice from the traceback that the root exception is coming from line 388 in _dill and that looks like some sort of attempt to try using StockPIckler.save(), so it would appear that dill does use pickle in some situations.

File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 578, in save rv = reduce(self.proto) TypeError: cannot pickle '_pylibCZIrw.czi_reader' object

and if you look at the code in _dill.py near line 388 (last line in the snippet below) you can see:

    if GENERATOR_FAIL and type(obj) == GeneratorType:
        msg = "Can't pickle %s: attribute lookup builtins.generator failed" % GeneratorType
        raise PicklingError(msg)
    StockPickler.save(self, obj, save_persistent_id)

Here is a more complete traceback:

File "C:\dev\ultistacker.ai\ultistacker\Stacker.py", line 923, in process_tiles self.process_tiles_parallel(round_index, def_field, A_global) File "C:\dev\ultistacker.ai\ultistacker\Stacker.py", line 1247, in process_tiles_parallel res = pp.map(self._process_one_tile, args) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\pathos\multiprocessing.py", line 135, in map return _pool.map(star(f), zip(args)) # chunksize File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\multiprocess\pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\multiprocess\pool.py", line 771, in get raise self._value File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\multiprocess\pool.py", line 537, in _handle_tasks put(task) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\multiprocess\connection.py", line 214, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\multiprocess\reduction.py", line 54, in dumps cls(buf, protocol, args, *kwds).dump(obj) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 394, in dump StockPickler.dump(self, obj) File "C:\Program Files\Python39\lib\pickle.py", line 487, in dump self.save(obj) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Program Files\Python39\lib\pickle.py", line 901, in save_tuple save(element) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Program Files\Python39\lib\pickle.py", line 886, in save_tuple save(element) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Program Files\Python39\lib\pickle.py", line 886, in save_tuple save(element) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1824, in save_function _save_with_postproc(pickler, (_create_function, ( File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1089, in _save_with_postproc pickler.save_reduce(reduction) File "C:\Program Files\Python39\lib\pickle.py", line 692, in save_reduce save(args) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Program Files\Python39\lib\pickle.py", line 886, in save_tuple save(element) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1427, in save_instancemethod0 pickler.save_reduce(MethodType, (obj.func, obj.self), obj=obj) File "C:\Program Files\Python39\lib\pickle.py", line 692, in save_reduce save(args) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Program Files\Python39\lib\pickle.py", line 886, in save_tuple save(element) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 603, in save self.save_reduce(obj=obj, rv) File "C:\Program Files\Python39\lib\pickle.py", line 717, in save_reduce save(state) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1186, in save_module_dict StockPickler.save_dict(pickler, obj) File "C:\Program Files\Python39\lib\pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "C:\Program Files\Python39\lib\pickle.py", line 997, in _batch_setitems save(v) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Program Files\Python39\lib\pickle.py", line 931, in save_list self._batch_appends(obj) File "C:\Program Files\Python39\lib\pickle.py", line 955, in _batch_appends save(x) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 603, in save self.save_reduce(obj=obj, rv) File "C:\Program Files\Python39\lib\pickle.py", line 717, in save_reduce save(state) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1186, in save_module_dict StockPickler.save_dict(pickler, obj) File "C:\Program Files\Python39\lib\pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "C:\Program Files\Python39\lib\pickle.py", line 997, in _batch_setitems save(v) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "C:\Program Files\Python39\lib\pickle.py", line 717, in save_reduce save(state) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1186, in save_module_dict StockPickler.save_dict(pickler, obj) File "C:\Program Files\Python39\lib\pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "C:\Program Files\Python39\lib\pickle.py", line 997, in _batch_setitems save(v) File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "C:\Program Files\Python39\lib\pickle.py", line 578, in save rv = reduce(self.proto) TypeError: cannot pickle '_pylibCZIrw.czi_reader' object python-BaseException

Thank you for your help with this! -Doug

[mmckerns]

so it would appear that dill does use pickle in some situations.

yes, that's true. What I said was multiprocessing uses pickle not dill, so the OP's traceback with multiprocessing in it is unexpected unless multiprocess was not installed.

@dougwood: so this seems like a different issue than the OP, so probably belongs in it's own issue. Again, there's limited ability to debug without a minimal example that demonstrates the same error. However, it looks like you are running into an object that dill nor pickle knows how to serialize. You can try a few things: (1) try the dill serialization variant dill.settings['recurse'] = True, (2) try to diagnose why it fails, or how it gets to the failure point, but using the pickle debugger dill.detect.trace(True). You'd want to post the results of the trace so we can help interpreting it. Think about opening a new ticket.