eriknw / afar

Run code on a Dask cluster via a context manager or IPython magic
Other
30 stars 0 forks source link

Try to pickle nicely #5

Closed eriknw closed 3 years ago

eriknw commented 3 years ago

We weren't able to pickle the function we obtained from compile and exec. This PR gets around this by running compile and exec again upon deserializing. This was the easiest option to implement.

Another option (instead of calling compile and exec) would be to serialize the components that make up the code object, then create the code object and function using types.CodeType and types.FunctionType. Similarly, we don't need to recreate the ScopedFunction from innerscope either, but doing so is the easiest.

In other words, I have put very little effort in optimizing for performance.