io.UnsupportedOperation: seek

[lazy-pi1]

Hi all, I am new to parallel python programming and I am trying to run one function in parallel. Previously I tried with multiprocessing library but it threw an error regarding pickling so, I tried with pathos and the code was working. But then all of a sudden it has started giving me this error. the function definitions are as follows:

from pathos.multiprocessing import ProcessingPool as Pool pullParallel = partial(stockPullWithDateLookup, engine, dataTable, fromdate) pool.map(pullParallel, chunkedMcscrips)

and the pool.map is giving the error. Also, stockPullWithDateLookup() is working in the serial mode. So I am not able to identify what exactly went wrong. Any help is appreciated.

Traceback (most recent call last): File "mcEodDataUpdate.py", line 24, in loadAllScriptsEodData() File "/home/braineater/maangeLLP/dataFetch/mcDataPull.py", line 388, in loadAllScriptsEodData pool.map(pullParallel, chunkedMcscrips) File "/usr/local/lib/python3.5/dist-packages/pathos/multiprocessing.py", line 136, in map return _pool.map(star(f), zip(args)) # chunksize File "/usr/local/lib/python3.5/dist-packages/multiprocess/pool.py", line 260, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/local/lib/python3.5/dist-packages/multiprocess/pool.py", line 608, in get raise self._value File "/usr/local/lib/python3.5/dist-packages/multiprocess/pool.py", line 385, in _handle_tasks put(task) File "/usr/local/lib/python3.5/dist-packages/multiprocess/connection.py", line 209, in send self._send_bytes(ForkingPickler.dumps(obj)) File "/usr/local/lib/python3.5/dist-packages/multiprocess/reduction.py", line 53, in dumps cls(buf, protocol).dump(obj) File "/usr/lib/python3.5/pickle.py", line 408, in dump self.save(obj) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python3.5/pickle.py", line 740, in save_tuple save(element) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python3.5/pickle.py", line 725, in save_tuple save(element) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python3.5/pickle.py", line 725, in save_tuple save(element) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python3.5/dist-packages/dill/dill.py", line 1306, in save_function obj.dict), obj=obj) File "/usr/lib/python3.5/pickle.py", line 599, in save_reduce save(args) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python3.5/pickle.py", line 740, in save_tuple save(element) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python3.5/pickle.py", line 725, in save_tuple save(element) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python3.5/dist-packages/dill/dill.py", line 1057, in save_cell pickler.save_reduce(_create_cell, (obj.cell_contents,), obj=obj) File "/usr/lib/python3.5/pickle.py", line 599, in save_reduce save(args) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python3.5/pickle.py", line 725, in save_tuple save(element) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python3.5/dist-packages/dill/dill.py", line 968, in save_functor obj.keywords), obj=obj) File "/usr/lib/python3.5/pickle.py", line 599, in save_reduce save(args) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python3.5/pickle.py", line 740, in save_tuple save(element) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python3.5/pickle.py", line 725, in save_tuple save(element) File "/usr/lib/python3.5/pickle.py", line 520, in save self.save_reduce(obj=obj, rv) File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce save(state) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python3.5/dist-packages/dill/dill.py", line 841, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/lib/python3.5/pickle.py", line 810, in save_dict self._batch_setitems(obj.items()) File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems save(v) File "/usr/lib/python3.5/pickle.py", line 520, in save self.save_reduce(obj=obj, rv) File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce save(state) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python3.5/dist-packages/dill/dill.py", line 841, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/lib/python3.5/pickle.py", line 810, in save_dict self._batch_setitems(obj.items()) File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems save(v) File "/usr/lib/python3.5/pickle.py", line 520, in save self.save_reduce(obj=obj, rv) File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce save(state) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python3.5/dist-packages/dill/dill.py", line 841, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/lib/python3.5/pickle.py", line 810, in save_dict self._batch_setitems(obj.items()) File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems save(v) File "/usr/lib/python3.5/pickle.py", line 520, in save self.save_reduce(obj=obj, rv) File "/usr/lib/python3.5/pickle.py", line 599, in save_reduce save(args) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python3.5/pickle.py", line 725, in save_tuple save(element) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python3.5/pickle.py", line 770, in save_list self._batch_appends(obj) File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends save(tmp[0]) File "/usr/lib/python3.5/pickle.py", line 520, in save self.save_reduce(obj=obj, rv) File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce save(state) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python3.5/dist-packages/dill/dill.py", line 841, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/lib/python3.5/pickle.py", line 810, in save_dict self._batch_setitems(obj.items()) File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems save(v) File "/usr/lib/python3.5/pickle.py", line 520, in save self.save_reduce(obj=obj, *rv) File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce save(state) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python3.5/dist-packages/dill/dill.py", line 841, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/lib/python3.5/pickle.py", line 810, in save_dict self._batch_setitems(obj.items()) File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems save(v) File "/usr/lib/python3.5/pickle.py", line 475, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python3.5/dist-packages/dill/dill.py", line 920, in save_file f = _save_file(pickler, obj, open) File "/usr/local/lib/python3.5/dist-packages/dill/dill.py", line 894, in _save_file position = obj.tell() io.UnsupportedOperation: seek

[mmckerns]

@lazy-pi1: This seems like (from the UnsupportedOperation Error, that you are trying to read and write to a file in parallel with different processes. While this is ok, it can also go badly many ways. A better choice is often to read from one file, and write to another then rename it to be the old filename. If this isn't what you are doing, then pardon me. It's hard to tell from your post what exactly you are doing. What would help is if you can post a simple example that reproduces your error. Something that I can try, and thus better diagnose it.

Also, there are some tools that can help uncover what the root of the error is. You can check if it's a pickling problem with dill.check and dill.detect.trace, however I don't think it is. It seems to be an issue with working with files in parallel. And in that case, your best bet is either to post a reduced code example that produces the same error... or editing your code to use a strategy like the one I describe above.

[mmckerns]

@lazy-pi1: Your issue resolved itself?

[lazy-pi1]

@mmckerns I found out the issue was related to the sharing of sqlAlchemy pool with each worker so I shifted to MySQL connector and created one connection in each worker which resolved the issue. Cheers!

[mmckerns]

Ok, thanks for the follow-up.