seastar.pc: Neither "--static" nor without "--static" does the right thing

When I try to build a Seastar application with

$ c++ getting-started.cc `pkg-config --cflags --libs --static ./build/release/seastar.pc`

The linking fails when trying to find /usr/lib64/libunistring.so, -ltspi and -lidn2. I do have all three libraries installed on my machine, but not the development package (libunistring-devel et al) but just the runtime package. For example, this runtime package includes /usr/lib64/libunistring.so.2, but NOT a symbolic link /usr/lib64/libunistring.so.

That should be enough - Seastar doesn't use any of these packages directly, it just uses GNU TLS, and cmake decied that that needs these libunistring.so, -ltspi and -lidn2. But since GNU TLS is a shared library, it will bring its own dependencies with it and it knows their full path (e.g. /usr/lib64/libunistring.so.2) - see ldd /usr/lib64/libgnutls.so - and does NOT need the symlink from the development packages to be installed.

Note that build without "--static" doesn't work either, it misses, for example, "-lrt".

Furthermore, the latest discussion on the issue you opened, @nyh, seems to indicate that it is indeed a packaging issue in Fedora and perhaps in Debian as well.

On Wed, Jan 16, 2019, 19:25 Jesse Haber-Kucharsky <notifications@github.com wrote:

On Wed, Jan 16, 2019 at 6:23 PM Jesse Haber-Kucharsky < @.***> wrote: This means that when gnutls-devel was packaged, it was linked against libunistring.so.2. Notice the suffix of .2! However, the pkg-config file references libunistring.so without the suffix! First of all, it's indeed a bug that the pkg-config refers to the pathname "libunistring.so", it should refer to "-lunistring" (I opened https://gitlab.com/gnutls/gnutls/issues/675). But that doesn't change anything regarding what you're saying

This doesn't make sense to me.

We agree that there is a bug in the gnutls-devel RPM (or possibly in GnuTLS, though I think it's a Fedora issue), and that a temporary work-around is to add the packages which should be dependencies to install-dependencies.sh.

It is absolutely not worth "throwing the baby out with the bathwater" (all the benefits of a usable seastar.pc in multiple contexts, including installing it to the system, using it from the build directory, supporting shared or static variants, and allowing for the possibility of packaging Seastar into its own RPM or DEB) when the solution is so straightforward.

You now mentioned severeal times that doing what @nyh (basically reverting our seastar.pc to how it was before, including our direct, transient dependencies in Libs) would make our seastar.pc unusable. Why would this be so? Im what regards would it be unusable?

—

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scylladb/seastar/issues/579#issuecomment-454865775, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUy2RyKSx0kDU81Ve0WriNmUypfE4p9ks5vD2B4gaJpZM4Z-Bwg .

You now mentioned severeal times that doing what @nyh (basically reverting our seastar.pc to how it was before, including our direct, transient dependencies in Libs) would make our seastar.pc unusable. Why would this be so? Im what regards would it be unusable?

Both myself and Raphael have explained at length why this is a problem, and if you are not operating in good faith in seeking to understand and learn, then repeating myself again is wasted effort.

@nyh @denesb @hakuch @espindola I think this discussion heated way more than it should. Our goal is to find a solution and not prove each other wrong.

I see here a more profound problem (from the man pages):

pkg-config - Return metainformation about installed libraries

the below could be interpreted in many ways:

--static Output libraries suitable for static linking. That means including any private libraries in the output. This relies on proper tagging in the .pc files, else a too large number of libraries will ordinarily be output.

In any case we are trying to use a tool intended to help link against installed libraries.

But this is what we have and what we are used too, that is why we are over sensitive about it. The third piece of objective information (also from the manual):

Libs.private: This line should list any private libraries in use. Private libraries are libraries which are not exposed through your library, but are needed in the case of static linking.

So as far as I can understand it:

The test to get into private is simple, is the library needed in static linking and is not exposed through our library? (for gnutls i believe the answer is yes).

@nyh is making a valid point about linking:

-l library Search the library named library when linking. (The second alternative with the library as a separate argument is only for POSIX compliance and is not recommended.) The general description below applies to the GNU linker. The linker searches a standard list of directories for the library. The directories searched include several standard system directories plus any that you specify with -L. Static libraries are archives of object files, and have file names like liblibrary.a. Some targets also support shared libraries, which typically have names like liblibrary.so.If both static and shared libraries are found, the linker gives preference to linking with the shared library unless the -static option is used.

If I needed to break it down: Libs - should contain everything that should be present in both static or dynamic linking. Libs.private - should contain everithing that is only relevant for linking against the package in its static form. Does it make sense? I think that since this topic seems to be important to us all we need to structure a solution we all can be comfortable with. But we need to do it soon because right now, nothing works! Any inputs on the above? Moreover, it seems to me that there is indeed a missing feature in pkg-config: consider the following different cases (with seastar being a static library as it doesn't matter): we use: pkg-config without the static flag that produces -lgnutls on 2 systems :

system with gnutls.a
system with gnutls.so The question is will system 1 get the extra -lunistring flag it needs (I believe it will not but it is testable). This essentially means that we have to tell system owner 1 to install gnutls dynamic (that he claims he doesn't really need). In other words either way it will not be perfect solution or a generic one, lets find a solution we all can live with :)

@nyh @denesb @hakuch @espindola I think this discussion heated way more than it should. Our goal is to find a solution and not prove each other wrong.

I see here a more profound problem (from the man pages):

An important document for understanding pkg-config is this one.

pkg-config - Return metainformation about installed libraries

the below could be interpreted in many ways:

--static Output libraries suitable for static linking. That means including any private libraries in the output. This relies on proper tagging in the .pc files, else a too large number of libraries will ordinarily be output.

In any case we are trying to use a tool intended to help link against installed libraries.

But this is what we have and what we are used too, that is why we are over sensitive about it. The third piece of objective information (also from the manual):

Libs.private: This line should list any private libraries in use. Private libraries are libraries which are not exposed through your library, but are needed in the case of static linking.

So as far as I can understand it:

The test to get into private is simple, is the library needed in static linking and is not exposed through our library? (for gnutls i believe the answer is yes).

You've got it backwards.

An easier way to frame this is "is this a private or public dependency of our library?" This is reflected in the name of the field itself : Libs.private.

When we link against a shared library, we don't need to reference the private dependencies because the compiler doesn't need them to link our application against the target.

However, when we link against a static library, we do need to reference the private dependencies because those dependencies have no way of being encoded in a static library file (*.a).

The pkg-config tool names the option --static for linking against a static library, and in doing this it includes the private stuff (because that's what's necessary for static linking.

The document I linked to above can explain more.

@nyh is making a valid point about linking:

Nothing you have said is consistent with what Nadav has said, unless I am misunderstanding you.

-l library Search the library named library when linking. (The second alternative with the library as a separate argument is only for POSIX compliance and is not recommended.) The general description below applies to the GNU linker. The linker searches a standard list of directories for the library. The directories searched include several standard system directories plus any that you specify with -L. Static libraries are archives of object files, and have file names like liblibrary.a. Some targets also support shared libraries, which typically have names like liblibrary.so.If both static and shared libraries are found, the linker gives preference to linking with the shared library unless the -static option is used.

If I needed to break it down: Libs - should contain everything that should be present in both static or dynamic linking. Libs.private - should contain everithing that is only relevant for linking against the package in its static form. Does it make sense? I think that since this topic seems to be important to us all we need to structure a solution we all can be comfortable with. But we need to do it soon because right now, nothing works! Any inputs on the above? Moreover, it seems to me that there is indeed a missing feature in pkg-config: consider the following different cases (with seastar being a static library as it doesn't matter): we use: pkg-config without the static flag that produces -lgnutls on 2 systems :

system with gnutls.a

system with gnutls.so The question is will system 1 get the extra -lunistring flag it needs (I believe it will not but it is testable). This essentially means that we have to tell system owner 1 to install gnutls dynamic (that he claims he doesn't really need). In other words either way it will not be perfect solution or a generic one, lets find a solution we all can live with :)

I'm having trouble understanding what you're saying here, but I explained things quite clearly above. Is there something there that is unclear?

@eliransin:

Libs - should contain everything that should be present in both static or dynamic linking.

This is incorrect. Libs should only contain what is required for dynamic linking, as this is the "default" mode for pkg-config.

Another way of looking at the same thing is "Libs should include the flags for linking against the target and all its public dependencies, unless those dependencies have their own pkg-config support, in which case they should be listed in Requires."

Additionally, from the FAQ of the document I linked to above, which summarizes the issue:

My library z uses libx internally, but does not expose libx data types in its public API. What do I put in my z.pc file?

Again, add the module to Requires.private if it supports pkg-config. In this case, the compiler flags will be emitted unnecessarily, but it ensures that the linker flags will be present when linking statically. If libx does not support pkg-config, add the necessary linker flags to Libs.private.

On Wed, Jan 16, 2019 at 8:46 PM Eliran Sinvani notifications@github.com wrote:

@nyh https://github.com/nyh @denesb https://github.com/denesb @hakuch https://github.com/hakuch @espindola https://github.com/espindola I think this discussion heated way more than it should.

I just sent a patch so that hopefully this discussion will return to less heated arguments and more evaluation of actual code.

Our goal is to find a solution and not prove each other wrong.

In this thread we already had 10 solutions. The debate is which one is more "correct", or less difficult for end-users. Of course I claim that my solution is both correct and easier to use ;-) But that's not agreed yet :-)

I see here a more profound problem (from the man pages):

pkg-config - Return metainformation about installed libraries

the below could be interpreted in many ways:

--static Output libraries suitable for static linking. That means including any private libraries in the output. This relies on proper tagging in the .pc files, else a too large number of libraries will ordinarily be output.

In any case we are trying to use a tool intended to help link against installed libraries.

I'm not sure what many interpretations you're referring to here. I don't think the current debate is whether Seastar or any of its dependencies have been installed (cooking, etc.). That is a different debate, that appears to have died down already :-)

But this is what we have and what we are used too, that is why we are over sensitive about it. The third piece of objective information (also from the manual):

Libs.private: This line should list any private libraries in use. Private libraries are libraries which are not exposed through your library, but are needed in the case of static linking.

So as far as I can understand it:

The test to get into private is simple, is the library needed in static linking and is not exposed through our library? (for gnutls i believe the answer is yes).

@nyh https://github.com/nyh is making a valid point about linking:

-l library Search the library named library when linking. (The second alternative with the library as a separate argument is only for POSIX compliance and is not recommended.) The general description below applies to the GNU linker. The linker searches a standard list of directories for the library. The directories searched include several standard system directories plus any that you specify with -L. Static libraries are archives of object files, and have file names like liblibrary.a. Some targets also support shared libraries, which typically have names like liblibrary.so.If both static and shared libraries are found, the linker gives preference to linking with the shared library unless the -static option is used.

If I needed to break it down: Libs - should contain everything that should be present in both static or dynamic linking.

Libs.private - should contain everithing that is only relevant for linking against the packge in its static form. Does it make sense?

Not exactly. Please see the patch I sent a few minutes ago to Seastar.pc which also explains my view of what "--static" does.

Jesse has made a correct point that pkg-config isn't a perfect tool - it's "--static" does something specific, and we just need to accept that (unless we want to add new modes to pkg-config, which I don't, yet, I'll do it if my patch gets rejected). But we have an argument what it does.

I actually think it's very clear what "pkg-config --static" does. It takes not only the package's direct dependencies but also its transitive dependencies (dependencies of dependencies). This makes sense in one and only one situation - where the entire linking (or at least just Seastar and its dependencies) is done statically, i.e., gcc -static ... pkg-config --libs --static seastar when just Seastar is a static library (and the linker picks up shared libraries for everything else) , "pkg-config --static" simply brings in too many dependencies-of-dependencies which aren't needed.

And since pkg-config --static isn't the right thing to use in default (not-strictly-static) linking, the question is what remains to be done. The answer is, I think, my patch.

Note that had we built both a seastar.a and seastar.so, no fix is needed at all (the current seastar.pc is exactly right). I also explain that in my patch.

I think that since this topic seems to be important to us all we need to structure a solution we all can be comfortable with. But we need to do it soon because right now, nothing works!

Initially I thought that Jesse said that everything works except my pkg-config command line in the demo.

Any inputs on the above? Moreover, it seems to me that there is indeed a missing feature in pkg-config: consider the following different cases (with seastar being a static library as it doesn't matter): we use: pkg-config without the static flag that produces -lgnutls on 2 systems :

system with gnutls.a

On modern systems (basically every system I've seen in the last 25 years) , the default is always the shared library. You always have gnutls.so, and sometimes you have also gnutls.a (chosen with "-static"). Nobody ever has just gnutls.a and not gnutls.so.

This is why pkg-config never bothered proposing a structured solution for this esoteric case.

system with gnutls.so The question is will system 1 get the extra -lunistring flag it needs (I believe it will not but it is testable). This essentially means that we have to tell system owner 1 to install gnutls dynamic (that he claims he doesn't really need). In other words either way it will not be perfect solution or a generic one, lets find a solution we all can live with :)

I didn't understand this.

If I understand correctly, the root cause is that pkgconfig doesn't have a notion of "this library wants to be linked statically but not cause its own dependencies to be linked statically too".

Let's go with the spirit of pkg-config. If it thinks --static means static everything, then that's what we'll do. If it means adding a bunch of stuff to install-dependencies.sh, that's fine.

As to the user, they have several options:

Use static Seastar and not care about the extra -l flags and packages (do they cause any harm?)
Use shared Seastar and not care about anything
Use static Seastar and edit the pkgconfig output if a few extra -l flags cause them to loose sleep
Use a different way of integrating Seastar and the consuming application (like including the .cmake file which we're supposed to export, which one is it?)

If I understand correctly, the root cause is that pkgconfig doesn't have a notion of "this library wants to be linked statically but not cause its own dependencies to be linked statically too".

Correct.

Let's go with the spirit of pkg-config. If it thinks --static means static everything, then that's what we'll do. If it means adding a bunch of stuff to install-dependencies.sh, that's fine.

As to the user, they have several options:
1. Use static Seastar and not care about the extra -l flags and packages (do they cause any harm?)

They show up on DT_NEEDED. If that is harmful for some reason -Wl,--as-needed can be used.

Thanks a lot for summarising.

With a heavy heart, I'm closing this issue. The issue still exists exactly as I formulated when I created this issue, namely, "pkg-config --static seastar" which we recommend to be used is incorrect in the sense that it lists too many libraries that aren't needed. But clearly I've failed to get my point across, as @hakuch and @espindola continue to call the incorrect behavior which I described "correct" and continue to recite theoretic cmake-related explanations which have nothing to do with this issue (which is just about pkg-config). @avikivity also decided against my patch to correct this issue, suggesting, among other things, that since merely installing a few additional RPMs works around this issue, so we don't need to worry about it. So I'm closing this issue.

:-)

If the feature of being able to specify a static lib with shared dependencies is added to pkg-config, I will be more than happy to use it.

For now closing this is the right thing, since there is absolutely nothing wrong in seastar.pc. It provides the correct dependencies with both --static and without.

scylladb / seastar

seastar.pc: Neither "--static" nor without "--static" does the right thing #579