Special files we keep or toss in a package on *NIX

[jakirkham]

Which of the following do we keep or toss?

1. *.pc pkg-config files (placed in $PREFIX/lib/pkgconfig/.... useful for compile options to use with a dependency)
2. *.la libtool files (placed in $PREFIX/lib/. useful for figuring out where libraries live).
3. *.m4 m4 files (clear value during build. some are installed in $PREFIX/share/aclocal. Why? 😕)
4. *.1, *.2, etc. man files (placed in $PREFIX/share/man/.... used by man, which we don't use).

My vote is keep 1 and 2, but drop 3 and 4. I know cases where 1 has been helpful ( https://github.com/conda-forge/ncurses-feedstock/pull/11 ). There are cases where 2 is important right now I'm debugging one.

I don't really understand the value of 3. So, drop? Maybe someone with more Linux knowledge (@ocefpaf?) could help us here.

As we can't use 4 now and it can take up some space, my vote is drop.

If we follow this route, we can just always add rm -rf $PREFIX/share/aclocal and rm -rf $PREFIX/share/man. Though honestly that directory more often than not includes junk so we should be able to always do rm -rf $PREFIX/share.

[scopatz]

Hi @jakirkham - I assume that the m4 files that are installed are installed so that the macros are available to other build systems that depend on whatever package installed them. I think these are most often used by autotools.

As per 4, please please please don't drop them. This is one of the most frustrating things about other package managers. If I install a package, I definitely expect to get the docs. Furthermore, just because we don't have a man package, doesn't mean that users can't set the $MANPATH to take advantage of these file. Moreover, fish, and to a lesser but growing extent xonsh, uses man pages to find tab completions. It would be a shame to loose this feature from packaging issue (again). Additionally, it is not true that the share dir contains nothing of value. This is the default place projects are supposed to stick custom data. $XDG_DATA_HOME is likely the most famous example of this. Finally, I don't think that the total space and bandwidth this takes up is significant to worry about.

The advantage of conda and conda-forge to me has always seemed that it has been an opportunity to create and distribute packages that actually contain the full package that the developer intended. The packagers don't go in there and start meddling with what the package should be. If the developer intended for a file to be installed, as an end user I expect to receive that file when I install it. I feel that this is one of the major failings of apt and other systems.

Just my 2 drachmas, but I do feel pretty strongly about this :)

[jakirkham]

Thanks for you feedback, @scopatz. Basically, people have been doing rm -rf $PREFIX/share though not consistently. When its been suggested, people have just done it. This is the first time I am hearing a loud no. So, I think it is good this came up then. Basically, what I was trying to do is either...

1. Reaffirm and justify the best practice
2. Come up with a new best practice and justify the change

What concerns me more is that our best practices are not always written down, which let's things get a little too loose. So, at least getting a show of hands on an issue is a start.

[ocefpaf]

I was going to say 1-3, but @scopatz made a compelling argument to keep 4 as well. So let's try to keep as much as we can and remove only real junk like test remnants, and unused data files when.

[QuLogic]

Do you really need 2? Fedora removes them from its packages.

[jakirkham]

Yes, it appears builds of other things will try to use these files to configure additional compilation flags correctly.

[QuLogic]

OpenSUSE also recommends not packaging .la files. Debian also generally recommends not packaging .la files. The autotools mythbuster suggests they're nearly useless on modern systems. Plus, they include absolute paths that tend to be broken (e.g., ContinuumIO/anaconda-issues#596). All-in-all, if the package includes a .pc file, it's better to include that over the .la file, which is redundant.

[jakirkham]

However, I should note that OpenSUSE says...

Avoid packaging libtool config files (.la files). If you do not package a static library, and if you are also placing the shared library in a standard search directory (i.e. %_lib, /usr/%_lib), they will not be needed.

Unfortunately, neither of these is true. We are packaging static libraries as we view that as a good practice to provide developers linking options. Also, people do use static libraries here. While this is infrequent, there are cases where static linking ends up being the right choice. Further, we don't place them in a standard location as Miniconda could be installed anywhere. So, hoping that things will be figured out correctly like for a Linux package manager using standard paths, is a faulty assumption.

Also, Debian says...

the shared library development package has historically included the .la, it must be retained in the development package (with dependency_libs emptied) until all libraries that depend on it have removed

As currently everything is a dev package, it makes sense to keep them according to this guideline.

As the Autotools Mythbuster link says...

it is not uncommon for distributions to skip packaging of .la files altogether, at least for those packages whose detection is possible through pkg-config.

It is a bit of a pain to determine which things do or do not use these. As it ends up being in some package's dependencies, we will spend an inordinate amount of time trying to determine when this is ok and possibly still being wrong and having to spend more time figuring it out and fixing it.

All that has been demonstrated at this point is that there may be some cases where these files are unused, but it has not been demonstrated that they are a problem.

Plus, they include absolute paths that tend to be broken (e.g., ContinuumIO/anaconda-issues#596).

As for your last point, I download our libffi package and it is correctly fixing the path in the .la. Not sure what the defaults package has wrong, but maybe we can offer some help. Note that some of the defaults packages have build scripts that even precede conda-build; so, it is possible some of them don't get packaged correctly especially if they see infrequent use.

Recently, I was concerned about exactly this issue that we were getting this wrong with .pc and .la files and actually opened this issue ( https://github.com/conda/conda-build/issues/975 ) to make sure we were addressing it correctly. Turns out I was wrong and these are being fixed correctly in all cases that I have seen. If you do find cases at conda-forge that don't do this correctly, please feel free to raise an issue on the feedstock.

Finally, according to Fedora...

Due to bugs in older versions of libtool or bugs in programs that use it, there are times when it is not always possible to remove *.la files without modifying the program.

So, this process of removal can in some cases cause problems too. I would rather not waste people's time on this removal.

TL;DR Removal can cause subtle problems and results in little gain. There does not actually seem to be any problems with .la files currently included. The simplest path remains keeping the .la files.

[pelson]

My 2 cents on this is: unless there is compelling reason not to (they are useless, they break things, they are massive/bloated) then we should keep everything that a package installs. There will always be exceptions to the rule (for instance, I deleted the tests from the matplotlib distribution for size reasons) and wouldn't want to force this on any particular recipe.

[ocefpaf]

@pelson I could not find any reason to keep the .la files (libtool archive). The default channel has a few (libxml2.la, libquadmath.la, and libgfortran.la at least) with bad prefix, without the place holder, and that causes linking problems. See https://github.com/conda-forge/staged-recipes/pull/870#issuecomment-229142477 and https://github.com/conda-forge/gdal-feedstock/pull/87#issuecomment-231277252

Removing them fixes those problems.

More: http://stackoverflow.com/questions/1238035/what-are-libtools-la-file-for

[jakirkham]

Certainly gcc and some other packages from defaults have broken libtool files. Some of them are not built with conda-build actually. So, I'm not surprised to see issues with them. I also have no problem with us cleaning out broken libtool files from those packages in our builds if we need to. Though we definitely should be raising issues about these problems so that we can get them fixed.

However, I have yet to see the same issues with packages here. For instance, here is the libtool file from our libxml2 package ( linux-64_libxml2-2.9.3-8.tar.bz2 ). Note the use of /opt/anaconda1anaconda2anaconda3. This is used as a placeholder for substitution with the PREFIX at install time. Also, note every path in this file begins with this placeholder.

# libxml2.la - a libtool library file
# Generated by libtool (GNU libtool) 2.4.6
#
# Please DO NOT delete this file!
# It is necessary for linking the library.

# The name that we can dlopen(3).
dlname='libxml2.so.2'

# Names of this library.
library_names='libxml2.so.2.9.3 libxml2.so.2 libxml2.so'

# The name of the static archive.
old_library='libxml2.a'

# Linker flags that cannot go in dependency_libs.
inherited_linker_flags=' -pthread'

# Libraries that this one depends upon.
dependency_libs=' -ldl -L/opt/anaconda1anaconda2anaconda3/lib -licui18n -licuuc -licudata -lz /opt/anaconda1anaconda2anaconda3/lib/liblzma.la /opt/anaconda1anaconda2anaconda3/lib/libiconv.la -lm'

# Names of additional weak libraries provided by this library
weak_library_names=''

# Version information for libxml2.
current=11
age=9
revision=3

# Is this an already installed library?
installed=yes

# Should we warn about portability when linking against -modules?
shouldnotlink=no

# Files to dlopen/dlpreopen
dlopen=''
dlpreopen=''

# Directory that this library needs to be installed in:
libdir='/opt/anaconda1anaconda2anaconda3/lib'

Whether or not we fully understand what things do or don't use libtool, is irrelevant as the libtool files at conda-forge appear to be fine. Probably in no small part to our full embrace of conda-build as our sole build tool. If we decide we need to delete them, we need to assess what is affected. Personally I see no pressing reason for either.

[pelson]

Probably in no small part to our full embrace of conda-build as our sole build tool

So I think the problem we were seeing was that /path/to/build/lib/../lib64 was being stored in the .la files. I'm pretty sure this could be achieved with conda-build if the lib64 directory was being used and subsequently renamed by a build script.

If we decide we need to delete them, we need to assess what is affected.

That is always the problem we are going to have. In these kinds of situations, if a Linux distribution has already lead the way and done the research, I'm happy to piggy back on that work.

[jakirkham]

To put it succinctly, I have yet to see a problem with our packages here at conda-forge w.r.t. to libtool files. However, I have seen problems with defaults and their use of libtool files (though in some cases that includes their pkg-config (*.pc) files as well). I'm not sure how coming up with a policy about our libtool files (that seem to be problem free) will fix defaults' libtool files (which have confirmed problems). That time would probably be better spent trying to package those very things from defaults here, which will almost certainly also fix this problem.

[QuLogic]

So I've finally found a reason not to include libtool archives, or at least them modify somewhat: overlinkage.

Take the libxml2 file quoted above:

# Libraries that this one depends upon. dependency_libs=' -ldl -L/opt/anaconda1anaconda2anaconda3/lib -licui18n -licuuc -licudata -lz /opt/anaconda1anaconda2anaconda3/lib/liblzma.la /opt/anaconda1anaconda2anaconda3/lib/libiconv.la -lm'

libxml2 is compiled with icu, so it's got icu in the line above. When libtool links a shared library, it will use that line to add a bunch of extra shared libraries to the link. Now, suddenly your conda package needs to also depend on the same version of icu you built libxml2 against, even if it doesn't actually use icu.

This is noted in the Debian info quoted above:

Also, Debian says...

the shared library development package has historically included the .la, it must be retained in the development package (with dependency_libs emptied) until all libraries that depend on it have removed

The key point being "with dependency_libs emptied". Conda-forge packages don't do this, and it's causing extra linking of packages making for subtle and fragile cross-dependencies.

If you don't want to drop libtool archives entirely, then at the very least start clearing out these lines. You may find that several packages have unnecessary pinning on things they don't really need.

[QuLogic]

For example, matplotlib has an ICU pin, but only because of Qt4, which only has an ICU pin because of fontconfig which only has an ICU pin because of linking with libxml2. Also cross-referenced there is libgd which only has an ICU pin because of libxml2 again.

All of these wouldn't be necessary if libxml2 didn't start causing bogus linking because of these useless libtool archives.

[pkgw]

This issue is has come up again lately. My best interpretation of what's happening is that I had intended to not distribute Libtool .la files with most of the xorg- packages, but due to a mistake the files were included; I recently "fixed" that, but now the disappeared .la files are causing problems when building packages higher in the dependency stack that do include .la files, because ones they depend upon are no longer available.

My best understanding is that .la files are totally unnecessary on modern macOS and Linux and that they can cause the sort of "overlinking" problems mentioned above. And, as mentioned before, some of the ones in defaults have historically been broken. They have definitely caused other aggravating problems for me in the past, but in an automated build/packaging system such as conda-forge's I believe that such problems should be largely avoidable.

My personal inclination is to not package them — again, they're just not needed, especially since pkg-config was invented, and they're basically yet another developer-facing file whose contents (in principle) need to be validated. I've never met a package maintainer who has explicitly wanted to distribute such files; they just come along automatically in autotools-based systems with no easy way to disable them. That's also why .la files often have invalid contents that go unnoticed for a long time.

That being said, if we did include them, I think that would largely be fine once the system equilibrated. To the best of my knowledge, "overlinking" would be the most substantial issue in such a world.

An inconsistent-in-practice policy isn't even the worst thing, but on the other hand I did just break a variety of package builds because I didn't think through the implications of removing the .la files while updating packages.

One other thing: now that we're on conda-build 3 we can in principle start producing "main" and "development" packages, à la what every Linux distro has been doing for decades. I wouldn't be surprised if there's a lot of work to be done to enable that, but, well, IMO there's a good reason that Linux distros all have this separation.

[pkgw]

Discussion in this libxcb PR has converged toward a policy of removing .la files from packages.