Fix loading C++ plugins from Python

[feltech]

What

Allow C++ plugins to be loaded from a Python environment for both CMake shared library installs as well as PyPI pip installs.

Why

Currently, C++ plugins cannot be loaded by our PyPI package, because the Python extension module openassetio._openassetio statically links in the core C++ library, and hides all its symbols, so the plugin cannot link when loaded at runtime.

There are three issues at play here

• The _openassetio library hides all symbols from the (static) core C++ library, so the plugin cannot link when loaded.
• Python (on POSIX) loads with RTLD_LOCAL, meaning the symbols are unavailable to subsequently loaded libraries, even if they are public.
• Ideally the plugin would have no dependency on the core OpenAssetIO C++ library, and would instead rely on symbols being already available in the process environment when it is loaded.
  • However, even if the _openassetio library and core C++ library are split into two shared libraries, RTLD_LOCAL is transitive, so would apply to the core C++ library too. So the plugin cannot rely on the symbols being present in the environment.
  • Windows is another question, and probably enforces RTLD_LOCAL-like behaviour regardless (needs more research).
  • A workaround, therefore, and possibly necessary (depending on Windows), is that the plugin must explicitly depend on the OpenAssetIO core C++ library.

Original investigation: https://github.com/OpenAssetIO/OpenAssetIO/issues/1324#issuecomment-2197460862

[feltech]

Add a CI job that pip installs OpenAssetIO and uses simpleResolver to resolve from SimpleCppManager. This will test linkage etc.

[feltech]

Note that building as multiple shared libraries is insufficient to provide the ideal plugin loading environment.

For example when split into multiple shared libraries, Python will transitively load libopenassetio but still not expose its symbols (RTLD_LOCAL is transitive). So C++ plugins still won't load unless the plugin has its own dependency on libopenassetio (which is at least possible when using a shared library build). I.e. the plugin cannot trust that the necessary symbols will be available in the process environment.

[feltech]

Updated description to focus specifically on the Python package, since that's the end goal. We should still consider warning users that static library builds are unsupported.

[feltech]

Relevant Windows documentation for Python library loading: https://docs.python.org/3/whatsnew/3.8.html#bpo-36085-whatsnew

the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies [...] Specifically, PATH and the current working directory are no longer used

Note this is since 3.8. We still support 3.7, which we either stop supporting, or we use the intersection of old and new behaviour, which seems to leave

[...] the directory containing the DLL or PYD file [...]

That is, assuming we split out core C++ library from the Python extension module, then the core C++ library will have to live in the same directory as the Python extension module on Windows.

This doesn't answer the question of how plugins will find the core C++ library, though...

[feltech]

I tested loading the SimpleCppManager C++ plugin in Windows, with split core library and Python module, where the two libraries live in the same directory, and it loads fine. I.e. the SimpleCppManager plugin has an explicit dependency on openassetio.dll, which is already loaded into the process when the plugin is loaded, so it doesn't have to search for it.

Ideally we'd compile in a Windows manifest to mirror the Linux and macOS RPATH mechanism, giving some consistency between platforms.

[feltech]

Interesting stackoverflow for Windows equivalent of RPATH

[feltech]

As a good example of prior art, I checked pip install PySide2 on both Linux and Windows. In both cases, PySide2 must also install Qt (C++ library) to the Python environment...

On Linux it installs Qt to a Qt subdirectory inside the site-packages/PySide2 directory. The Python extension modules are built with a RUNPATH of $ORIGIN/Qt, so the Qt subdirectory is added to the library search path.

On Windows, all the Qt dlls are dumped directly inside the site-packages/PySide2 directory, i.e. in the same directory as the Python extension modules. So the Qt dlls are on the search path augmented by Python (as explained above https://github.com/OpenAssetIO/OpenAssetIO/issues/1340#issuecomment-2203488458)