Support static and dynamic linking on Windows

[compnerd]

Description

Windows does not currently handle linking correctly. When building a library to be linked statically, we must build with -static. The static and shared libraries need a separate build (one for each variant used) including the object files and swiftmodule.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

Swift Package Manager version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

[compnerd]

CC: @abertelrud

[abertelrud]

Probably the most straightforward way of handling this given the current requirements would be modify the build plan so that, when targeting Windows, there it creates two llbuild manifest targets for each library target market as automatic: a static variant and a dynamic variant. Each of them would be configured with the requisite combination of flags, and would in turn specify either the static or dynamic variant of any of its own library dependencies, based on whether it itself is static or dynamic. In other words, the dynamic variant of a library has to link against the dynamic variant of its own dependencies, etc.

Any executables would then have dependencies on the static variants of its libraries, and any dynamic libraries would have dependencies on the dynamic variants. This should allow llbuild to build only the variants that are needed by whatever top-level targets are getting built. There shouldn't be much harm in specifying both the static and dynamic variants of the llbuild targets even when they are not used, except that it makes the llbuild manifests somewhat bigger. The .o files and other intermediates will need to incorporate the variant in the path so that the intermediate files are kept apart.

[al45tair]

A complication here that is probably non-obvious to non-Windows developers is that when you make a Windows dynamic library (a .dll), you also build a corresponding static library (a .lib file) — Windows developers and the Microsoft documentation refers to this as an "import library", and it's the import library that you actually link against when you want to use the dynamic library. The upshot is that it isn't just intermediates that need a separate directory here — you can't build a static library and a dynamic library of the same name into the same directory, because the static library will overwrite the import library (or vice-versa).

[dschaefer2]

In the Windows development I've done, the libraries are usually dynamic located in the same directory as the executable. We've seen issues with too many exported symbols as we bundle a bunch of static libraries into a single product DLL.

I want to make sure we can build SwiftPM with SwiftPM on Windows so I'll play around with different options to see if it helps to build libraries as DLLs by default and not bags of object files like we do today.

[al45tair]

Just noticed I hadn't mentioned above another gotcha that is relevant here, namely that the object files need to be built with different flags on Windows depending on whether you intend them to be part of a DLL or not. SwiftPM's current strategy of building object files and then creating static and/or dynamic library targets from them is therefore a bit broken on Windows, because if you ask for both static and dynamic library output, the code won't be built with the correct flags for both — we'd need to build the objects twice in that case.

That isn't the case on other platforms (for the most part… it is possible on some platforms to build a static library without position independent code, and in some cases that means the objects in that library couldn't be used in a shared library, but PIC is typically just fine in a static library so that's an easy problem to solve).

[compnerd]

The dual build being required was the stumbling block for me. We don't have a good way to model the dual builds because targets don't know how they will be used.

[dschaefer2]

I definitely noticed that. I was stunned to see the --target arg only produced the objects and I had to use --product to produce the lib/exe. We'll probably need some customization of the build graph when the triple is Windows to fix this.

I'll play with this a bit more. Part of my swiftpm contributor training :).

[dschaefer2]

@al45tair do you know off hand what those flags are?

[compnerd]

@dschaefer2 for Swift code, that is just -static for static libraries, nothing for dynamic libraries or executables. For C/C++ code it is a bit more complicated, as in addition to the libc type (/MD, /MDd, /MT, /MTd) each library often has its own macro to indicate if it is being built statically or dynamically (e.g. LIBXML2_STATIC and CURLSTATIC).

[Keithcat1]

How does Rust handle all of this? How about just statically linking everything all the time on Windows? How many things would that break horribly?

[al45tair]

How about just statically linking everything all the time on Windows? How many things would that break horribly?

You can't statically link everything on Windows because the Windows API is a set of functions in DLLs (contrast with Linux where syscalls form part of the ABI; this is not the case on Windows or Apple platforms).

The real pain point though is the C/C++ library. While ucrt has improved the situation markedly, there are still parts of the C/C++ runtimes that are not part of the Windows API. You can statically link those, but if you do so then you may find that the C/C++ library exhibits surprising behaviour across dynamically linked modules in your application (you can read more on Microsoft's website which also has some examples of problems it can cause).

Now, if you're asking about statically linking just Swift things, that's certainly possible — but we definitely want to support dynamic linking as well, not least because without it Swift wouldn't be usable in various contexts on Windows. Even then, statically linking the Swift runtime is fine as long as there's only a single image using Swift within the process… if you have multiple images, statically linking the runtime would then cause similar problems to the ones Microsoft describes in relation to the C/C++ runtimes.

Finally, on Windows, as on Apple's platforms, statically linking things is not the norm. It's not what Windows developers would expect by default, and for certain Windows technologies (COM especially), dynamic linking is really a requirement.

[Keithcat1]

I just want to statically link the Swift runtime and any Swift dependencies, to simplify the build process and make my app easier to distribute (is there a supported way of installing the Swift runtime on Windows?). If it was possible to do LTO on all statically linked Swift code, there would even be some performance / size improvements (although maybe not big ones). It seems Rust statically links Rust code by default, but you can manually modify a Crates cargo.toml to have it be linked dynamically. You'd need to edit your dependencies to dynamically link against them, but you could put them all under one crate as dependencies and link that crate dynamically.

[dschaefer2]

The main issues we're running into in the short term are warnings and errors at link time for duplicate and too many symbols exported from executables. I'm going to try set up the llbuild manifest so that there are static and dynamic versions for every swift object file we create and leave it up to the products to decide which one they want. If there are no dynamic libraries specified, then only the static versions of the object files will get built. Executables where we're having the most problems will use the static versions of the object files. I'd love some feedback on whether this will actually work or not or edge cases I haven't thought of that would break.

There may be bigger issues at play like static linking of the Swift runtime, but we need to get over this short term hurdle first.