The pickled object could not be unpickled..

[superstar54]

Here is the script to reproduce

# test_shell.py
from aiida_shell import ShellJob
from aiida.engine import submit
from aiida import load_profile
from aiida_shell.data import PickledData
from aiida_shell.launch import prepare_code

load_profile()

echo_code = prepare_code("echo")

def test_shell():

    def parser(self, dirpath):
        from aiida.orm import Str
        return {'string': Str((dirpath / 'stdout').read_text().strip())}

    n = PickledData(parser)
    n.base.attributes.get("unpickler_module")

    inputs = {'code': echo_code,
             'arguments': 'some output',
             'parser': PickledData(parser),
            }
    submit(ShellJob, inputs = inputs)

Run pytest test_shell.py, the process is excepted

$ verdi process report 33902
*** 33902: None
*** (empty scheduler output file)
*** (empty scheduler errors file)
*** 1 LOG MESSAGES:
+-> WARNING at 2024-05-07 22:26:52.813038+02:00
 | output parser returned exit code<310>: Callable specified in the `parser` input excepted: The pickled object could not be unpickled..

However, if I run test_shell() function directly inside the script using python test_shell.py instead of pytest, it works without error.

[sphuber]

The difference is in how the parser function is pickled by dill. When running through pytest, it is pickled as <function run_pickle.test_shell.<locals>.parser(self, dirpath)>. But when running the script directly, it is stored as <function __main__.test_shell.<locals>.parser(self, dirpath)>. Notice that in the latter case, the "module" is __main__ because the script is the main entry point. However, when running through pytest, the script is treated as a module and so the module string starts with run_pickle (which is the name I gave to the script).

When dill tries to unpickle the pytest case, it will try to load the run_pickle module, but this is likely not in the path and so it fails.

I don't think this is a bug in aiida-shell. Why are you running a script through pytest? If you want to write unit tests that include a dynamic parser using pytest, you can see how it is done here: https://github.com/sphuber/aiida-shell/blob/f68ea013a7392bb0000b85bc6a266f7dce694cf1/tests/test_launch.py#L272

That works just fine. Closing this for now. Feel free to reopen if you think there still is a problem in functionality

[superstar54]

Thanks for the comment. I used it in the unit test, and I think I did the same as in the aiida-shell/test_launch.py. I will investigate it more and will reopen it if needed.