Closed gcroci2 closed 3 months ago
- I fixed it by using
dill
to serialize and deserialize the functions, which appears to be more reliable.It's not completely clear to me what this does, but if it works now, I'm happy to accept the change.
inspect.getsource
(old way) retrieves the source code of the function or lambda expression passed to it.
It looks up the function definition in the current environment. Lambda functions are often anonymous and defined inline, making it challenging for inspect
to locate their source code reliably, especially if they're not defined in a module or a directly accessible script. When I run the code on Snellius (I get the error only in that case), it's very likely that the exact location or the context where the lambda was defined might not be accessible or might not be present in the expected form.dill.dumps
) serializes the lambda function into a byte stream. The deserialization (dill.loads
) converts the byte stream back into the original function object. This means that even if the function's source code is not directly accessible, dill
(that captures the complete context of the function) can reconstruct it from the serialized data. The deserialized function retains the context and source information (captured in the serialized data) that inspect
can then access.I hope it's clearer but in case we can have a chat about this :)
Given that we have much less time for this project now, maybe we can discuss fixing versions of many packages to avoid similar problems in the future? Also given that development of this package will slow down, at least for the moment, and current package versions work well with what we are currently doing, it may be less important to stay up to date with newest package versions. This would be a separate PR though.
Very good point, I opened issue #607 :)
While training a big network I got:
OSError: could not get source code
from the_save_model()
method of theTrainer
class, in particular when we dostr_expr = inspect.getsource(key["transform"])
. The error indicates that the inspect module is unable to retrieve the source code for the lambda function stored inkey["transform"]
.dill
to serialize and deserialize the functions, which appears to be more reliable.scipy.signal.bspline
import toscipy.interpolate.BSpline
to be up to date with a more recent version of scipy (minimum 1.13.1) as indicated in one of their latest releases (1.13.0). I did this since we prefer to not fix the versions, but just to indicate the minimum, and the previous requirement on scipy made the tests break because of this expired deprecation.