Unable to access `lf.source_directory()` in federated execution

[Depetrol]

The lf.source_directory() cannot be accessed in federated execution. Example program:

target Python {
    coordination: decentralized
}
reactor Server{
reaction (startup){=
    print("LF source directory", lf.source_directory())
=}
}
federated reactor {
    server = new Server()
}

Error:

    print("LF source directory", lf.source_directory())
RuntimeError: LF_SOURCE_DIRECTORY constant is not defined.

[lhstrh]

This is one of the reasons the feature was questioned in the first place. The circumstances under which you can possibly know where your source directory is located are very limited. For one, this won't work if the binary gets transferred to a remote system and the sources are not. It also won't work if you create a Docker image that doesn't make the sources available in the runner stage. If the path is absolute, it won't work in someone else's file system. If it is relative, it won't work if you move the binary. Etc...

I honestly still doubt there is a way to make this "work" universally, so likely we'll have to narrow down scope. We had a discussion about this recently, and I recall @edwardalee advocated that the feature "as-is" was incomplete but useful enough in the near term to be merged and could be fixed later.

Could I ask what are you trying to accomplish with this feature?

[Depetrol]

In this specific case, I'm trying to install a the custom python serializer package using pip in the preamble (#2375). This would require locating the python package on the file system and installing it with pip. However the python package path could only be relative, because we want the path to work in different environments. I'm trying to use lf.source_directory() as the base for the relative import.

However I have encountered the need for something similar to lf.source_directory() in various other cases, and I think the functionality could be quite useful. Because the generated python files are located in the src-gen directory, it would inevitably break relative paths.

• Loading config files. For example, using Hydra.
• Accessing data files for ML applications.
• Importing relative python files

In these applications I used workarounds such as changing the operating directory with os.chdir to a absolute location on my file system before imports.

[cmnrd]

For Python there is already established, well-thought-out, and hardened tooling for all of this. The key for solving this issue would be a proper integration with the Python ecosystem (in particular packaging) and using importlib.resources instead of a homebrew mechanism.