feature request: handle non perl dependencies

[happy-barney]

Provide unified way how to handle non perl dependencies, eg C libraries.

Proposal: extend current: requires => { package => version }

to: requires => { 'ACME::Extra::Dependencies::PkgConfig' => [ { mod => 'glib-2.0', min_version => '2.42.0' }, { mod => 'gtk+3.0' }, ]}

[mohawk2]

I would say the way to do this would be to depend on an Alien package. How would you make this portable to Windows and/or VMS? I would suggest this is a bad idea and should not be done.

[karenetheridge]

I support the general concept of being able to declare other dependencies in META.*, but we have no good ideas for how to implement this sanely.

On Tue, Dec 2, 2014 at 5:43 PM, mohawk2 notifications@github.com wrote:

I would say the way to do this would be to depend on an Alien package. How would you make this portable to Windows and/or VMS? I would suggest this is a bad idea and should not be done.

— Reply to this email directly or view it on GitHub https://github.com/Perl-Toolchain-Gang/CPAN-Meta/issues/79#issuecomment-65339116 .

[happy-barney]

ad mohawk: same way like build process works - platform specific implementation I used PkgConfig only as (hopefully well known) example. Currently there is no common option to tell XS modules to use non system C-library.

ad karenetheridge: i think let's postpone implementation after META.* extension. I think that allowing external modules (plugins) to evaluate dependencies will solve this as well as other dependencies like "I need at least one of these packages, doesn't matter which one".

[haarg]

Writing a spec without considering how it would be implemented is a guaranteed way to create a useless spec.

The main problem with this is correlating names in the spec to something usable. Perl prerequisites are easy because we have a registry of perl modules (CPAN). Linux distributions can handle library and program prereqs because again they have a registry of packages. But CPAN modules are cross platform, so there is no authority for how these prerequisites would be resolved.

[happy-barney]

haarg: correlating names to something usable will be task of platform specific module, like it is now in building process. This logic is already there. And usually if you install library XYZ you will not rename it to ABC just for fun.

[haarg]

The logic can't already be there because there is no authority for what would be used in the meta files. The names used are just things made up by the author, and there may not be any sensible way to list a dependency in a cross platform manner.

As an easy example, a CPAN author might list 'mysql'. What does that actually mean? Is it the mysql libraries? The mysql command line client? The mysql server? Without some authority to decide what that means, there's no way to decide. And if you say "it's part of the platform specific module" then you are essentially including the behavior of any number of random modules as part of the spec. And that isn't a spec at all.

[happy-barney]

ad logic: ok, not said clearly - logic to determine current environment and adjust behavior already exists eg in Module::Build

ad your example: thanks to mention it. Thats why I suggested evaluation modules, so for example; requires => { 'C::Module' => [{ module => 'mysql'}] } requires => { 'External::Tool' => [{ name => 'mysql'}] } requires => { 'External::Tool' => [{ name => 'mysqld'}] } requires => { 'External::Service' => [{ name => 'mysql'}] }

where C::Module will have an implementation modules like C::Module::PkgConfig, C::Module::rpm, C::Module::deb, C::Module::window, etc

of course, you can restrict allowed modules or even allow this syntax only to some kind of placeholders, but point is still same: execute code to evaluate parameterized dependency instead of version only dependency.

[rehsack]

We discussed such a proposal last QA Hackathon but there was still no time to publish the results.

Our basic approach was similar to prereqs - _x_prereqs with same primary categories (configure, build, test, rumtime ...) and dependency spec in there. DS should name pkg-config resolvable names - see http://mail-index.netbsd.org/tech-pkg/2014/03/14/msg012749.html for longer discussion about distribution point of view.

[dagolden]

My view continues to be:

• This is an important issue that affects a lot of people
• This is a really hard issue because of the platform specific nature of it.

If anyone is interested, during the last CFP round, this was discussed as CMSP 16 binary Package Dependencies and it stalled out for the reasons above.

One of the more interesting suggestions was to put information into META that a tool like Devel::CheckLib would consume. That doesn't necessarily help resolve the dependencies, but at least it would let tools detect them.

I tend to think that the Alien::Foo approach will be best, particularly if Alien has tools that support installing dependencies via distro repositories. I think the Alien::Base work get closest to this. (CC: @jberger, @plicease -- and I see @mohawk2 worked on it a fair bit, too.)

[plicease]

Alien::Base based Alien modules won't install system packages for you (RPMs, .debs etc), but it will detect them if they have already been installed.

[sjn]

Since this is still an open issue, I'd like to revive it. :-)

Here's an alternative that doesn't depend on "x_" type keys (which I think goes against the benefits of having a standard way of specifying third-party and system dependencies), and doesn't potentially create a forest of custom solutions like the Alien:: namespace is doing.

How about just allowing for URIs as filenames in the dependencies?

e.g. extending the example in CPAN::Meta::Spec (version 2):

    prereqs => {
      runtime => {
        requires => {
          'perl'   => '5.006',
          'ExtUtils::Install' => '0',
          'File::Basename' => '0',
          'File::Compare'  => '0',
          'IO::File'   => '0',
          'file:///usr/bin/ps' => '0',       # binary executable, but it's enough that the file/path exists
          'executable:netstat' => '0',    # binary executable, but search in standard locations depending on system
        },
        recommends => {
          'Archive::Tar' => '1.00',
          'ExtUtils::Install' => '0.3',
          'ExtUtils::ParseXS' => '2.02',
          'package:gcc' => '9',    # normalized package name (needs to be translated to whatever the local packaging system uses)
          'deb:gcc' => '9',          # debian package version, which can be automatically installed if we're on such a system
        },
      },
   }

I guess this could also be extended to support different resources that a Perl distribution might depend upon, but that's a topic for another ticket/Toolchain Summit :-)

[jberger]

Here's an alternative that doesn't depend on "x_" type keys (which I think goes against the benefits of having a standard way of specifying third-party and system dependencies), and doesn't potentially create a forest of custom solutions like the Alien:: namespace is doing.

I honestly don't see how that solution would meet any of those criteria. Adding dependencies in an incompatible manner would break existing clients which wouldn't know what to do with them, it is the reason you'd want a new key which you'd prototype with x_. And it would necessarily involve a forest of solutions to handle the different uri types, unless it was just supposed to be advisory in which case you're breaking lots of things for little gain. Finally, Alien::Build/Base has a reason that it is the way it is, it will never make global changes to the system, angering sysadmins. It will either use a pre-installed dep or it will build/fetch a copy to be used locally. It took lots and lots of design to make that work, especially given all the various platforms and build systems.

[karenetheridge]

Adding dependencies in an incompatible manner would break existing clients which wouldn't know what to do with them, it is the reason you'd want a new key which you'd prototype with x_.

Presumably we'd be bumping the specification version at the same time. But I'm not sure if we'd ever be able to get a version 3 metaspec to be adopted widely enough to be worthwhile -- a lot of tooling would need to change to make use of it.

[sjn]

I honestly don't see how that solution would meet any of those criteria. Adding dependencies in an incompatible manner would break existing clients which wouldn't know what to do with them, it is the reason you'd want a new key which you'd prototype with x_.

Don't the current clients out there check if they support meta_spec version and then complain or prompt for upgrading if not?

So yes, as @karenetheridge says, a change like this would require bumping the meta_spec version number, assuming clients do the right thing when they see a version that they don't support. Newer clients (that support meta_spec version 3) could do the right thing with any URI prereqs they bump into from then on.

And it would necessarily involve a forest of solutions to handle the different uri types, unless it was just supposed to be advisory in which case you're breaking lots of things for little gain.

Well, a few sensible ones could be supported in the beginning, and if the client bumps into an URI they don't understand, then offer a warning or something like that....

 Finally, Alien::Build/Base has a reason that it is the way it is, it will never make global changes to the system, angering sysadmins. It will either use a pre-installed dep or it will build/fetch a copy to be used locally. It took lots and lots of design to make that work, especially given all the various platforms and build systems.

I think it's fine to look at different deployment scenarios too.

E.g. For generating installation scripts/instructions for installing system dependencies when setting up an application stack in a container.

Or for creating native packages, and using the URI dependencies to generate correct native package dependencies automatically.

Or for using this information to create and query other types of dependencies, like "memory:50GB" or "service:ntpd" and allowing for custom URI handlers to do something useful with this information instead of the default warning...

[Leont]

Don't the current clients out there check if they support meta_spec version and then complain or prompt for upgrading if not?

Complain and fail to install in not generally considered an acceptable outcome. Having a meta 3.x next to meta 2.0 might be acceptable, but for a third file doing almost exactly the same to be worth it, it would have to be pretty damn solid (instead of all the uncertainties I read above)

[sjn]

URI::PackageURL might offer an avenue for this feature, no?

[jberger]

It is interesting to see that someone is considering a universal mechanism of addressability for dependencies like this. However, given that we're talking about system-level dependencies there would still be the problem of not having a single platform/architecture/etc. You'd need to specify likely many different alternatives to install from debian/rpm/windows/mac sources, and even then you'd certainly never get them all.

I think it still would be best to have some standard key, perhaps added in a later 2.x if it must, or x_ prefixed of course, that indicates that there are system dependencies and if possible what they are, perhaps using something like this new format. However, it would still be better if your Makefile.PL (or the like) then knows how to detect its presence and indicate failure. Installation tools that point to system packages will likely never be able to do that.

That said, once again, that was literally the design purpose of Alien::Base. As extended by @plicease you write an alienfile that knows how to detect if a system dependency already exists. The consuming module can then choose if it should be installed if not already present or if it should just bail out. (Also note that A::B never installs system packages because it is not within the principle of least surprise that installing a perl module, possibly locally, will install a system dep, and if it did systems administrators would kill us).

[plicease]

That said, once again, that was literally the design purpose of Alien::Base. As extended by @plicease you write an alienfile that knows how to detect if a system dependency already exists. The consuming module can then choose if it should be installed if not already present or if it should just bail out. (Also note that A::B never installs system packages because it is not within the principle of least surprise that installing a perl module, possibly locally, will install a system dep, and if it did systems administrators would kill us).

I agree with all of this. I personally do not think a cpan client (or Perl modules) should ever by default install system packages, but having an opt-in or plugin system may be appropriate. IMO it probably won't ever be able to support every platform, but supporting the three or four that cover 99% of systems that are used in practice would probably be "good enough" especially if it is designed in a way that it doesn't exclude adding other platforms later.

I can think of a couple of useful ways of handling this.

1. meta provides the appropriate values for discovering the package via Devel::CheckLib. This is really only enough to print an error message that says "you need to install xyz library". It may also not work well on Windows for some subset of Windows Perls or some subset of libraries on Windows (where libraries sometimes have a slightly different name; this can also happen on non-Windows, but in my experience is is more commonly a problem on windows).

2. in addition to the meta that does detection in (1), include the Alien that will provide the library dynamically. Just specifying the Alien as a dependency will functionally do the same thing in the normal case, but if the cpan client can do the library detection it may save some dependencies in the case of a system install (ie the system already provides the needed library). One downside is that older clients that do not recognize the new meta will have no way of knowing to install the Alien, whereas older clients can handle explicit dependencies on Aliens no problem.

3. in addition to the meta that does detection in (1) either the CPAN module provides the native library name and version for platforms known to the developer, or the cpan client can compute the native library name and version somehow. The Spec could also provide for specifying a fallback Alien as described in (2). This way the preferences would be a) try the system lib using Devel::CheckLib b) install the system package if the user has opted into that functionality c) fall back on the specified Alien if that was provided.

4. guess you could also do everything in (3) except the Alien fallback; I personally am against that since I think the Alien concept is useful.

I don't think the first option adds much.

The second I think has some merit in that it does provide some utility with a smaller amount of effort. The second option is more or less what I do with FFI modules now (except substitute FFI::CheckLib and it is handled in the module installer logic instead of the cpan client), and it works pretty well.

The third option I think offers the most flexibility for cpan authors and users of those packages, but it also adds extra complexity.

Most options duplicates at least some of the work that Alien is already doing. This is just my 2¢. FWIW I've been thinking about these options for a while, but I don't think I have ever written them down anywhere.

[sjn]

Good perspectives!

For me, I think it's worth reminding ourselves that CPAN::Meta's purpose is to communicate that different types of dependencies are required (or recommended) for the the affected software to function as expected, and that the question in this ticket is how we can do it in a manner that allows us to get the necessary information to resolve these dependencies (or if that's not possible, as you say, to offer some instructions).

Package URLs are seeming to become a standard way for exactly this, especially in the SBOM space, where security professionals are required by law to ensure that they have a complete picture of where their software components come from, and how/who have modified it on the way from the author/vendor to the user.

For us, this could look as simple as this:

Before:

requires => { package => version },

New:

requires => { package => version },
external => { comment => [ pkgurl_alt1, pkgurl_alt2 ] }

…where pkgurl_altN is a valid PackageURL (as in the spec) possibly implemented by URI::PackageURL?

I guess PackageURLs without a version should be interpreted as "use the latest", but AFAICT the PURL spec doesn't have any examples of how to specify version ranges, etc, so it seems they are mainly intended for reporting what has been installed (e.g. as part of an SBOM).

I still think PURLs could be used for specifying requirements (with a version range) if we make a careful proposal to the spec and the URI::PackageURL author? :grin:.

Then there's the generic PURL type: https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#generic :smile: