[FEATURE] Dealing with dynamic dependencies

[stan423321]

Describe the scenario and circumstances that should be addressed

As currently documented, dds build system has a fairly rigid approach to dealing with dependencies, focusing on static archives for self-managed libraries and allowing system defaults for compiler's built-in standard ones.

However, for legal and/or technical reasons, dynamic linking is sometimes required. Instances of legal reasons are fairly obvious for anything deployed on end user machines, otherwise LGPL would not be considered different from GPL. The technical reasons are present in case of, for example, video games interfacing with rendering APIs, which have varying implementations depending on hardware and firmware setup.

Describe alternatives you've considered

For dependencies that one cannot (drivers) or otherwise doesn't need to modify, one could subvert the dependency management aspect of dds and copy the required header files into, depending on the application, src/ or include/ directory of their project, and then manipulate the linker flags accordingly. Compiler flags could also be used to include headers, which may or may not work depending on how it defines a compiler-defined header. One of the points of dds usage is dependency management, so these two are not satisfactory.

In case where one wants a modified DLL/SO/... and is not satisfied with static linking either due to legal obligations or other reasons (such as a wish to present a failsafe customization point), there doesn't seem to be an obvious way to do that with dds at the moment.

Describe the solution you'd like

These appear to be two separate problems which just happen to share a common root issue, and as such it appears to me that providing two separate solutions may be adequate.

For things which can be built, but should be put into a separate object for one reason or another, a configuration key that would force it to work that way would do that. This, in turn, presents another issue, as the dds packages are not supposed to be particularly configurable. The most satisfying solution, from the viewpoint of application developer, would be to allow multiple options, such as forcing static linkage of an entire library, forcing dynamic linkage of library and letting it decide about its dependencies, or mixing a selection of libraries into a single DLL/SO/..., which is in direct conflict to that. Alternatively, a library could enforce its preferred linking setup, and optionally present different versions of itself (or the package) to allow options, though given indirect dependencies and uniform library identifiers this could get out of control as well.

For things that simply cannot be built, but could benefit from being treated as a managed library, a more orthogonal solution would be needed, such as allowing import libraries / SOs as a special type of sources.

Additional context

The SDL2 library is interesting in that it uses manual dynamic linking of its dependencies, including, optionally, newer versions of itself when linked statically. However, this is not accomplished through an in-archive list of needed symbols in such modules, as its current build system requires some version of such dependencies to be installed in order to detect them at runtime. What is exactly happening there could warrant further investigation.

[JPGygax68]

My own use case for DLLs (under Windows) is a main program written in Fortran that I and a few other developers are writing DLLs for. Of course those DLLs all expose a strictly "C" interface, which the main program consumes via manual binding (LoadLibrary() + GetProcAddress()).

I fully appreciate the idea of radical simplification, which I agree is direly needed, behind the decision to focus on libraries. However, shared libraries seem to me such a widespread use case that it should be supported.

Merely thinking aloud here: how about supporting them in a way that is similar to how application entry points are? Instead of having foo.main.cpp, we could have something like foo.exports.lst.

I may be speaking out of ignorance here. I'm a Windows developer and have only used C++ on Linux very superficially, so I'm not familiar with how shared libraries work there.

[vector-of-bool]

Creating dynamic libraries with dds is an intended future feature, but one that will need a lot of careful consideration. While static libraries behave fully within the bounds of the standard phases-of-translation defined by the language and behave mostly uniformly between toolchains, dynamic libraries often side-step these semantics (and in different ways on different platforms).

Dynamic libraries also bring a bundle of new questions to the table, such as "How do I find my dynamic dependencies (RPATH, assembly manifest)?" "How do I encode the link-name (SONAME) of my dependencies?" "How do I encode my own link-name?" and "How do I define the symbols I export?"

Regarding one potential design: The toolchain file is the primary customization point for a dds build at the moment, but is going to eventually encompass much more than just the compiler and linker. It may warrant a rename. For the case of building dynamic libs, it is simple enough to say "build my own project as a dynamic lib instead of a static one," but to say "build dependency XYZ as a dynamic lib instead of a static one" is a bit more tricky. A "toolchain" file might be the answer, encoding it as:

# ... toolchain stuff ...

# The `ACME/Widgets` and `ACME/Gizmos` libraries will be built as dynamic libraries, 
# instead of static ones
Build-Dynamic: ACME/Widgets
Build-Dynamic: ACME/Gizmos

Of course, a library might not support being used as a dynamic library (e.g. it may be missing symbol visibility). In addition, a library that can be built in either mode might need to be told which mode it is being built for.

One of the pending features is to support some minimal amount of configurability for individual libraries to expose to their consumers, and I believe that will be a prerequisite to exposing dynamic libs. I have a design in mind of how this will look, but I'll cover that in another ticket :slightly_smiling_face:.

[JPGygax68]

This is one of the things that CMake makes look easy - at first glance at any rate; just replace STATIC with SHARED and you're done ... er, no quite.

I guess this brings up the question of extensibility. One of the (few) things I like about CMake is that, having a turing-complete language, it allows you to create functions that will hide the complexities from your users (and yourself). In fact, I've been musing about creating something I'd have liked to call "high-level CMake", using which a library or executable could be configured with a single CMake function call. Of course, that could only work by imposing certain restrictions, which is why I got so interested in DDS.

How would I go about integrating DDS into a bigger build/development setup, such that I could append DLL creation to the build chain? What language/tool(s) would you recommend?

[stan423321]

The default Linux setup's most obvious difference is that everything is exported, unless code explicitly says it isn't, but projects using the reverse, Windows-ish, setup exist. The funny thing is that, as far as I am aware, you switch them with compiler flags. Same compiler flags everywhere? That means the same export setup.

The other difference is that Windows has no RPATH and instead uses its algorithm for constructing a collection of paths where autoloaded DLLs may be. From build system point of view, that actually may mean less work, new DLLs must be next to an EXE, but EXE does not need adjusting when they're all being moved.

There are a few other details, such as Linux using position-independent code and Windows using relocatable code instead, as well as SO being its own "import library" when linking for autoloading. These don't seem to be particularly problematic differences when it comes to build systems, as far as I'm aware, but this isn't my area of expertise either.

[vector-of-bool]

@JPGygax68 I haven't fully documented the "how to use it with CMake" stuff yet, but it is possible. You can at least see a small example of how it is done in my blog post, under the heading Example: Using it from CMake. I haven't added this process to the main docs yet, though (See #8).

@stan423321 Regarding symbol visibility, I'll probably be adding -fvisibility=hidden to the default GCC/Clang toolchains, which will mean that all common platforms will require visibility attributes or toolchain-overrides in the case that one wishes to generate a static library. Also, Windows gained some limited RPATH-like functionality recently, employed using an attribute in the binary manifest. It's still pretty new and I'm not sure there are many people making use of it.

The static libraries that dds generates are compiled as relocatable (with -fPIC), so it might be possible to simply (in theory) "convert" a static library into a shared one by linking the "main" archive with --whole-archive (I've never tried this, but my intuition is that it will "just work"? If you happen to try it, get back to me on that).

[nyanpasu64]

Is linking to Qt GUI libraries a case of dynamic dependencies? Is that something that can be reasonably handled using DDS?

[vector-of-bool]

@nyanpasu64 Qt is almost always consumed through dynamic libraries, yes, but it isn't mandatory except for licensing oddities.

As for whether dds can handle Qt: Not at present. Such behemoth frameworks are often the most difficult to deal with.

[tomboehmer]

Here are some ideas for your consideration.

Linking an external library

I see two possibilities:

• add the necessary information manually to link_flags in the toolchain file
• provide a key for convenience to be used like so:

  Link-Dynamic: "/absolute/path/to/libmylib.so"
  Link-Dynamic: "mylib"

  In the second case it is up to the linker/loader to figure out where to find the library, in the first case the absolute path can can be used to automatically embed rpath information in executables linking against it.

Building and linking a dependency as a shared library

I like your idea of specifying which dependencies to build dynamically in the toolchain file:

Build-Dynamic: ACME/Widgets
Build-Dynamic: ACME/Gizmos

It might also be useful to allow libraries to prescribe how they want to be build in library.json5, for example:

Build-Dynamic: never
Build-Dynamic: always

The result of building a dynamic library might be placed directly in _build or in some sub-directory of _build with a name derived from the name of the package the library comes from, the package namespace or the library name. The package version should be included in the name of the shared object to prevent possible name conflicts between different versions upon installation.

The advantage of placing the shared object in _build or in a sub-directory thereof is to allow easy relocation/installation of the binaries. A sub-directory structure specifically would allow to easily pick up and install the shared dependencies separately from the executables linking against them.

In order for an executable that links against the shared libraries to work with them out of the box and upon relocation one may link with -Wl,-rpath=$ORIGIN/path/to/shared/object on linux, assuming that the shared objects are indeed located relatively to the executable in question.