CFEP-18: Packaging static libraries

[xhochy]

At the moment, most binary packages that include libraries only ship their shared version. Some packages though also include the static version or only ship the static version. At runtime the static archives are not needed and thus are wasted space to the end user. In addition, our tooling (conda/conda-build/the bots) is also geared towards maintaining a set of dynamically linked packages.

This CFEP should serve as a document to make it clearer how to deal with static libraries.

[CJ-Wright]

What level of support are we planning on providing to the static libs? Should be bot be rebuilding downstreams?

[CJ-Wright]

What is the user experience when a static lib moves and downstreams aren't rebuilt?

[xhochy]

What level of support are we planning on providing to the static libs? Should be bot be rebuilding downstreams?

The lowest possible. One of the goals of this proposal is to make sure developers understand that static libraries are a second-class citizen. Thus I would for now skip any bot automation. I explicitly excluded Go / Rust / Haskell as there it might make sense to have a bot for "world rebuilds" in future but that should be a separate endeavour.

What is the user experience when a static lib moves and downstreams aren't rebuilt?

It's the same as if the lib wasn't updated at all. The used version stays the same.

I think this is generally fine — in principle there are some UX issues to think about (your software build broke because it depended on libblah.a and the libblah package suddenly stopped providing that file) but I am pretty confident that they are rare and only affect power users, and the setup we would be moving to would align well with pretty much every Linux distro out there.

I expect them to happen quite a bit when we remove static libraries. Thus I wrote this CFEP before even broadly removing the static builds from the feedstock I maintain, that there is some place to point people at on what we as a community actually aim for.

My only big question is, what's our Windows policy? I only have a poor understanding of library distribution for Windows, but it seems that depending on a package's heritage and/or build system either static or shared libraries may be preferred. (Along with disagreements about .a vs .lib, etc. I don't believe that we have a very uniform policy about how to package compiled Windows libraries in general, and even if we did it would be hard to apply uniformly? Different build systems seem to strongly prefer different schemes, I think.)

We should be less strict on Windows builds. There are a lot libraries that only work with static linkage as the upstreams didn't take care to define the correct __dllexport lines. If a build works fine with shared linkage, it should stick to only providing those. If a build doesn't yet work really with static linkage on Windows, we shouldn't force the maintainer of the feedstock to write a huge patch to make the linkage work. There is already a lot of hassle and not that much community support for packaging things on Windows.

One additional caveat on Windows is though that there are several types of static libraries. In addition to every shared library (bin\name.dll) there is always accompanying (static) import library (lib\name.dll). Sometimes the static build of a library has the same name as the import library, sometimes it is with a _static suffix, i.e. lib\name_static.lib. The suffix .lib is though only used when you are working with the MSVC-based toolchain. If you use the MinGW-based toolchain, the import library has a .dll.a suffix (though import libraries are necessarily needed for MinGW) and the static archives have solely a .a suffix, omitting the .dll part.

[xhochy]

Incorporated the review comments and added an explicit section about Windows.

[scopatz]

This looks good to me, thanks for pushing it forward @xhochy!

[xhochy]

@conda-forge/core

This PR falls under the CFEP Approval policy, please vote and/or comment on this PR. This PR needs 60% of core to vote yea to pass. To vote please leave Approve (yea) or Request Changes (nay) reviews. If you would like changes to the current language please leave a comment or push to this branch. This vote will end on 2020-07-08.

[mwcraig]

I'd like to abstain on this one -- I don't understand the details well enough to weigh in. [If needed to get us to the voting threshold I'll vote yes]

[scopatz]

Thanks @xhochy - great work here!

[xhochy]

Thanks for the votes; with 15 approvals this is above the 60% percent threshold and thus passes.