search

LLNL / maestrowf

A tool to easily orchestrate general computational workflows both locally and on supercomputers
https://maestrowf.readthedocs.io
MIT License
134 stars 43 forks source link

dill>0.3.5.1 unable to pickle CompiledFFI object #406

Open Jmast opened 1 year ago

Jmast commented 1 year ago

fails starting with dill==0.3.6. Not sure what changed in dill; however, at the very least the maestro deps need to be dill<=0.3.5.1

pickler._batch_setitems(iter(source.items()))

File "/usr/lib/python3.8/pickle.py", line 997, in _batch_setitems save(v) File "/home/mast2/spl_flux_test/local/lib/python3.8/site-packages/dill/_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "/usr/lib/python3.8/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/usr/lib/python3.8/pickle.py", line 717, in save_reduce save(state) File "/home/mast2/spl_flux_test/local/lib/python3.8/site-packages/dill/_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "/usr/lib/python3.8/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/mast2/spl_flux_test/local/lib/python3.8/site-packages/dill/_dill.py", line 1186, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/lib/python3.8/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/usr/lib/python3.8/pickle.py", line 997, in _batch_setitems save(v) File "/home/mast2/spl_flux_test/local/lib/python3.8/site-packages/dill/_dill.py", line 388, in save StockPickler.save(self, obj, save_persistent_id) File "/usr/lib/python3.8/pickle.py", line 578, in save rv = reduce(self.proto) TypeError: cannot pickle 'CompiledFFI' object 2022-11-22 17:20:10,629:name : 453:ERRO: Maestro workflow finished with exit status: 1

FrankD412 commented 1 year ago

Thanks for the issue @Jmast -- I'll fix this by the end of the weekend. Is this holding you up at all? Do you need something sooner?

Jmast commented 1 year ago

Thanks, no rush, @FrankD412 , but there are a few other things:

  1. I have a issue outstanding regarding corona flux version and maestro being merged that I commented on in PR: "added flux adaptor v0.31.0 #395") comment is here
  2. Until a more suitable approach is implemented, the version checks for flux probably need to be entirely removed as I have found that the "strict" version throws an exception for "unreleased" installs of flux. Ultimately I think that the flux plugin needs to support a range of versions and should just be backward compatible as much as possible with logic based on version numbers. Right now in my forked version I just commented out the entire try/except block in connect_to_flux in maestrowf/abstracts/interfaces/flux.py because the StrictVersion() call throws an exception for versions like 0.42.0-84-g1dcbcd97d
FrankD412 commented 1 year ago

Oh -- sorry about the delayed fix. Slipped through the cracks on my end. I can take a look at that soon.