Unable to load a pickled object

[luke-leiter]

We are using a graph to do some pathfinding for us in python. The graph is fairly large, so we are generating the graph elsewhere and pickling the python object to be loaded later. AnyLogic is one of our test beds for testing the output of the pathfinding algorithms. We need to load the pickled object but have been unsuccessful. The error given:

Failed to run python code; feedback: AttributeError("Can't get attribute 'CustomClass' on <module '__main__' from 'C:\\\\Users\\\\REDACTED\\\\AppData\\\\Local\\\\Temp\\\\pypeline-server_13616648004425206534.py'>")
    at com.anylogic.engine.Engine.error(Unknown Source)

I've imported the class in the error message using a from statement. I'm running the pickle.load command right in AnyLogic.

pyComm.run("with open('python_files/package/src/package/pickled_object.pkl', 'rb') as file:",
" custom_class_object = pickle.load(file)");

I'm able to declare a new object of the class as well.

pyComm.run("custom_class_object=CustomClass()");

Is there a limitation with "where" on my computer pypeline actually runs? This is my first guess based on the output of the error showing it's running main from a temp directory.

Any help is appreciated!

[t-wolfeadam]

Interesting use case!

Is there a limitation with "where" on my computer pypeline actually runs?

I thought the answer was "no": before this problem, I would've touted that it runs the same way as if you execute a script or use in an interactive terminal relative to the model's directory. Turns out pickling is an exception!

To test this, I started by creating a setup (assumingly) similar to yours (see here) and generated the pkl object from running the shown script directly.

I then added a 'helper' script in the model directory for streamlining the python code needed in AL and for testing outside of AL (see here) - as you can see, running that directly does work. However, if I try importing the helper from a separate interactive terminal, it fails for a similar reason it does in AL (see here). Double however, the successful "direct loading" seen in the screenshot fails when using Pypeline -- that, I don't know why.

I did find a workaround though: you just need a custom unpickler that redirects the class lookup to the correct module. Updating the 'helper.py' script to look like the following allows it to work from anywhere.

class MyCustomUnpickler(pickle.Unpickler):
    def find_class(self, module, name):
        if module == "__main__":
            module = "pickler"  # script name that generated the file
        return super().find_class(module, name)

def get_obj():
    with open("python_files/pickling_test/pickled_object.pkl", "rb") as f:
        unpickler = MyCustomUnpickler(f)
        obj = unpickler.load()
    return obj

In case you want to explore yourself: Model323.zip

[luke-leiter]

@t-wolfeadam I just figured this out. I did exactly what you put here. I didn't know that pickle saved the path to the module used.

[t-wolfeadam]

@t-wolfeadam I just figured this out. I did exactly what you put here. I didn't know that pickle saved the path to the module used.

There's that, but still the issue of a discrepancy between loading from the active environment in an interactive shell (which works) versus when called via Pypeline (which does not) when the idea is that they should behave the same.

After looking into it more, the way Pypeline creates the Python environment causes __name__ to be set to "builtins" (vs "__main__" in an interactive shell). The same thing happens to PyCharm's shell (SO ref) - something for me to look into at some point