Fill pkg-configPackages using a script

[roberth]

Project description

After #212282, we'll have a set of aliases for pkg-config packages to be used by lang2nix integrations. The list is manually curated, but we have the tools to fill the list with (almost) all pkg-config-supporting packages that we have. I'm not suggesting to regenerate the list from scratch; as the data does not to be polished up, not to select the wrong _version variant, because nix index isn't always complete, and possibly other reasons. Instead, my suggestion is to add a script that any maintainer can run, to see if we have any pkg-config packages in a nix-index index, that haven't been added to the #212282 package set yet.

Starting point

# nix-index ...
nix-locate -r 'lib/pkgconfig/.*.pc$'

This returns an output of the shape:

aravis.dev                                          487 r /nix/store/7xqykk0185wwrsns2kgcc6mvhswld5v3-aravis-0.8.22-dev/lib/pkgconfig/aravis-0.8.pc
apngasm.out                                         469 r /nix/store/034qbq067my5g5xzlh3gx483z6wafq8l-apngasm-3.1.10/lib/pkgconfig/libapngasm.pc
appindicator-sharp.out                              508 r /nix/store/830abjvv1bh60ib1rhs42sxvnl9jka69-appindicator-sharp-5a79cde93da6d68a4b1373f1ce5796c3c5fe1b37/lib/pkgconfig/appindicator3-sharp-0.1.pc
appstream.dev                                       383 r /nix/store/160galxcv9di5s42mr644ad65230i6bf-appstream-0.15.5-dev/lib/pkgconfig/appstream.pc
appstream-glib.dev                                  465 r /nix/store/h9a5db8jqdawkadd0ccjnxhpyzban6aq-appstream-glib-0.8.2-dev/lib/pkgconfig/appstream-glib.pc
libsForQt5.appstream-qt.dev                         389 r /nix/store/9cvz17hlq8r7rkjqlqf5gy21mrgcnmbp-appstream-qt-0.15.5-dev/lib/pkgconfig/appstream.pc
apr.dev                                             441 r /nix/store/kjpf61f0ccnq642bsl2gyih2x5dlv6xw-apr-1.7.0-dev/lib/pkgconfig/apr-1.pc
aprutil.dev                                         512 r /nix/store/2mdphf6yf02g5bd7w23j8kgbmwzaijxf-apr-util-1.6.1-dev/lib/pkgconfig/apr-util-1.pc
apt.out                                             277 r /nix/store/5y6mdv0n5z88fhxaajrksj21hxm9wnnb-apt-2.5.4/lib/pkgconfig/apt-pkg.pc
apk-tools.out                                       367 r /nix/store/q9v7nn2qwrlq82rp8d1r1q97s8gxhqg1-apk-tools-2.12.10/lib/pkgconfig/apk.pc

The file name at the right hand side, before .pc, is should correspond to an attribute name in the pkg-configPackages package set. When it is missing, the script can make a suggestion, such as adding

aravis-0.8 = pkgs.aravis;
apngasm = pkgs.apngasm;

and so on.

This list of suggestions will contain duplicates, in which case they should be reported separately, as these cases seem to require a human to decide which attribute is the best default. (Though perhaps the shortest name may turn out to be a good heuristic)

Regarding the repology label, from what I've heard there's interest from repology to reuse our pkgconfig metadata.

Metadata

• homepage URL:
• source URL:
• license: mit, bsd, gpl2+ , ...
• platforms: unix, linux, darwin, ...

[roberth]

After discussion with @Ericson2314 we'll also want to use a script to fill a complete set of non-unique packages anyway, so the solution has two parts

• generate a database for all packages
• curate the set of default packages from the set of all packages

I'll also note that we may want to switch to a json as the primary data source, which would allow the consumer to do more. Platform-specific exceptions can be overlaid on top of that in Nix, or be represented in the json. Examples: hidapi, wayland. Both are packages that have a different set of pkg-config modules depending on the platform.

[SuperSandro2000]

as the data does not to be polished up

For now not but over time we surely need to move/remove entries because the actual files might change.

nix-index ...

nix-locate -r 'lib/pkgconfig/.*.pc$'

We should searczh for any .pc$ file to find misplaced ones.

I'll also note that we may want to switch to a json as the primary data source

If we would use yaml, we could work with anchors for duplicated data.

[roberth]

For now not but over time we surely need to move/remove entries because the actual files might change.

That's where the tests come in. Also @Ericson2314 suggests using the package meta for this. This allows for a more sustainable architecture where

• Source of truth and best quality data lives in package meta attribute.
• meta attribute is tested in CI. OfBorg should cover this fairly well, and hydra could summarize it.
• We periodically auto-update the json file based on the latest meta attribute information. This serves as an index for lang2nix tools, as they need a function module -> package; not package -> module.
• nix-index is only used for finding new packages that haven't received meta info yet. At least for the initial effort
• In steady state, ofborg could force-apply the meta tester from https://github.com/NixOS/nixpkgs/pull/214304 to the changed/added packages that it builds, which is a check that really any package can and must satisfy anyway. When we have this, nix-index isn't needed anymore.

If we would use yaml, we could work with anchors for duplicated data.

For now a JSON is better, because it can be read by Nix without it having to build anything. If yaml support is added to Nix at some point, this may be a consideration, but as a performance optimization. I doubt that we need it, as git is quite clever with deduplication and general compression. It might only save some space when unpacked, which doesn't seem like much of a problem.

[Ericson2314]

https://github.com/NixOS/nixpkgs/pull/215198 this would prepare for that.