How to get pkg-config to work with transitive dependencies?

sogaiu commented 4 years ago

Here is a situation from a draft of hpkg work for xorg: https://gist.github.com/sogaiu/517347d14469adae71975774afb43c49#file-xorg-hpkg-L340_L347

When built in "mainstream" environments, libX11 depends directly on libxcb, and libxcb in turn depends on libXau. Further, libX11 does not directly depend on libXau:

libX11 -> libxcb -> libXau

When libX11's configure is run, pkg-config is used to determine what libxcb needs. This part works fine because one can arrange for PKG_CONFIG_PATH to be informed of where to look for libxcb's .pc file. That's straight-forward because we can write an expression based on (libxcb :path) (i.e. using libxcb's name).

The information obtained from libxcb's file indicates that libXau is necessary. However, if we don't want to explicitly list transitive dependencies (and I hope we don't), I don't think we can populate PKG_CONFIG_PATH with information about where to look for libXau's .pc file in the same direct manner as we did for libxcb as that would require us to use libXau's name explicitly.

It seems that we need some other way to find libXau's .pc file.

In "mainstream" environments (e.g. Ubuntu, Arch, etc.), it appears that there aren't that many directories that typically house .pc files and these tend to be fairly fixed locations so I don't think this tends to be a problem.

Does this description make sense?

What follows are some related thoughts.

Is there some way for a running build to know the paths of all its build dependencies? If so, it seems like that might give a way to appropriately populate PKG_CONFIG_PATH given that there seems to be a convention that .pc files get stored in directories named pkgconfig.

It seems to me that it might make sense for a running build to be able to determine and at least have read access to its build dependencies. I think something close to that sort of thing is assumed in many build environments. But these are just speculations.

May be there are security implications?

In any case, I'm stuck, please help :)

andrewchambers commented 4 years ago

Hmm yeah, so transitive dependencies in .pc files do sound like an interesting challenge.

You could try calling link-farm to add the dependency pkgconfig files to a packages output. This way a packages pkg-config dir has its own .pc files, and those of the dependencies.

We may be able to design our own .jdn metadata format containing richer information, and write this to a path when a package builds.

sogaiu commented 4 years ago

Thanks for the ideas.

I will investigate the link-farm approach for the moment.

sogaiu commented 4 years ago

If the "create symlink" approach is taken, could any of the following happen?

dangling symlinks
extra runtime dependencies because of symlinks to things that are only needed at build time
conflicts between an existing .pc file and an attempt to create a symlink to another .pc file with the same name

andrewchambers commented 4 years ago

The second two yes, but we need to split lib packages into -rt parts anyway In the future imo.

I think link-farm should try to see if the link target is the same, and if so, do nothing, otherwise we should make it raise error.

sogaiu commented 4 years ago

Ok, thanks.

Should I make an issue for enhancing link-farm then? (I did a couple of quick searches on an Ubuntu machine and an Arch Linux machine and found no duplicate names FWIW).

Out of curiosity, is it technically feasible to have a dynamic variable that gives one access within a builder function to the build-time dependencies?

If so, does it seem like there are some undesirable aspects to this idea?

andrewchambers commented 4 years ago

Should I make an issue for enhancing link-farm then? (I did a couple of quick searches on an Ubuntu machine and an Arch Linux machine and found no duplicate names FWIW).

Sure.

Out of curiosity, is it technically feasible to have a dynamic variable that gives one access within a builder function to the build-time dependencies?

It's seems possible, but you would just get every referenced package. I feel like we may be better off with our own way to write dependencies to disk.

If so, does it seem like there are some undesirable aspects to this idea?

Its more stuff to marshal every time a builder gets run, I'm also not totally sure how to use this information in a meaningful way and filter out the useful packages automatically.

sogaiu commented 4 years ago

Thanks for the elaboration.

I hope the current approach works out -- it does seem like it could be more efficient. Perhaps there won't be too many other situations that are similar (i.e. requiring access to things in other packages at build time).

sogaiu commented 4 years ago

I have tried out an implementation of the idea. So far it seems to be working ok.

I'm using this function:

# XXX: move this out of this file to core or prelude?
(defn link-pc-files
  [target-dir link-dir] # see ln(1) man page
  (when (not (os/lstat link-dir))
    (eprint "non-existent directory: " link-dir)
    (break false))
  (link-farm [target-dir] link-dir
             |(string/has-suffix? ".pc" $)))

Typically after a make install invocation, I have something like:

    (os/mkdir (string (dyn :pkg-out) "/lib"))
    (os/mkdir (string (dyn :pkg-out) "/lib/pkgconfig"))
    (each pkg [libxcb]
      (link-pc-files (string (pkg :path) "/lib/pkgconfig")
                     (string (dyn :pkg-out) "/lib/pkgconfig")))
    (each pkg [util-macros xorgproto xtrans]
      (link-pc-files (string (pkg :path) "/share/pkgconfig")
                     (string (dyn :pkg-out) "/lib/pkgconfig")))

On a side note, these and other instances motivated this issue: https://github.com/andrewchambers/hpkgs/issues/92

sogaiu commented 4 years ago

Ok, I finally got through what I had before of xorg plus related packages to get it and st working.

I followed the suggested symlink idea, but I also removed .la files from a number of packages (not the xorg packages though [yet?]).

It wasn't obvious to me before beginning this work but I guess some of the goals of .la files (libtool) and pkg-config overlap and in some cases this doesn't work out so well. So far I have not figured out how to get libtool to ignore .la files, so I chose the path of eliminating them (except within xorg packages where they seemed to work out ok [1]).

When I looked into it, I learned that at least one distribution appears to eliminates (most / all of?) them -- I think that's Debian. I'm not turning up the link to that ATM, but there is an LFS page that mentions removal as helping at least in one case: http://www.linuxfromscratch.org/blfs/view/svn/introduction/la-files.html

I think this made it possible to remove many (if not all?) explicit mentions of transitive dependencies (that's not across all of community, just the things I worked on).

One notable exception is st. Here I ended up spelling out numerous xorg packages to get the include and library directories along with LDFLAGS to work out. Here are the relevant lines: https://gist.github.com/sogaiu/03be65dd99c0efea5d1fc54bcec4b6e9#file-st-hpkg-L45_L116

I haven't tried to work on reducing the references in this bit, but on first glance it seems somewhat similar to the pkg-config situation. How does it seem to you?

Below are most (all?) of the relevant drafts:

xorg: https://gist.github.com/sogaiu/517347d14469adae71975774afb43c49

st: https://gist.github.com/sogaiu/03be65dd99c0efea5d1fc54bcec4b6e9

fontconfig: https://gist.github.com/sogaiu/25fc2341ede4f3052738da70db1d45e1

freetype2: https://gist.github.com/sogaiu/5eecf178b656210b32b450a308b4393a

libpng: https://gist.github.com/sogaiu/edd0e03570a61e142b2fa3a1dc23666e

expat: https://gist.github.com/sogaiu/04ff0e1ff86ac1a4ba6d3c307ea6052b

[1] IIUC, at least one gentoo package maintainer seemed to think sometimes they might be necessary: https://flameeyes.blog/2008/04/14/what-about-those-la-files/

andrewchambers / hpkgs

How to get pkg-config to work with transitive dependencies? #90