search

materialsproject / pymatgen

Python Materials Genomics (pymatgen) is a robust materials analysis code that defines classes for structures and molecules with support for many electronic structure codes. It powers the Materials Project.
https://pymatgen.org
Other
1.49k stars 857 forks source link

@cached_class decorator prevents class methods of Compatibility objects from being used in multiprocessing as they can't be pickled #3898

Closed CompRhys closed 2 months ago

CompRhys commented 3 months ago

The use of this decorator leads to the error message:

self = <joblib.parallel.BatchCompletionCallBack object at 0x3376ec590>
    def _return_or_raise(self):
        try:
            if self.status == TASK_ERROR:
>               raise self._result
E               AttributeError: Can't pickle local object 'cached_class.<locals>._decorated'

I'm unsure what problem this decorator is solving in this instance. If there's no reason to keep these decorators I can to submit PR of local edits to use joblib to parallelize applying compatibility settings to large numbers of entries.

https://github.com/materialsproject/pymatgen/blob/fbf8ec0a5a9c0fb649975d66dedb31b17bcc9dfc/pymatgen/entries/compatibility.py#L838-L840

CompRhys commented 3 months ago

This seems to be the motivation for the decorator: https://github.com/materialsproject/pymatgen/pull/2450 suggesting it may be necessary, pinging @munrojm for input on whether this is likely still potential issue given changes to the api/rester

CompRhys commented 3 months ago

Talked to @janosh and he thinks we can probably resolve this with a PR to monty to sort out the pickling behavior of the decorator

CompRhys commented 2 months ago

Closing as fixed by https://github.com/materialsproject/pymatgen/pull/2450