API for external dependecies

[melezhik]

Hi Nick!

I am writing this as an issue, though it's just a question. What an API does zef provide for external dependencies?

With regard to this discussion I am thinking about creating of prototype so that Sparrow tool install a module and it's external dependencies where external dependencies data comes from zef API.

Thanks

Aleksei

[niner]

On Montag, 22. Juni 2020 03:54:44 CEST Alexey Melezhik wrote:

I am writing this as an issue, though it's just a question. What an API does zef provide for external dependencies?

With regard to this discussion I am thinking about creating of prototype so that Sparrow tool instal a module and it's external dependencies where external dependencies data comes from zef API.

Why would you need zef for that? External (native) dependencies ought to be part of the dependencies information in META6.json. https://design.raku.org/S22.html#depends

There's an example in the documentation: https://docs.raku.org/language/modules#Preparing_the_module

Use System::Query to collapse the meta info (e.g. the by-distro.name things) down to what the current system needs.

[melezhik]

If I get this information from META directly that will work too.

Let's take a look at the given exmaple:

"depends" : {
        "build": {
            "requires": [
                "Distribution::Builder::MakeFromJSON",
                {
                    "from" : "bin",
                    "name" : {
                        "by-distro.name" : {
                            "macosx" : "python2.7-config",
                            "debian" : "python2.7-config",
                            "" : "python2-config"
                        }
                    }
                }
            ]
        },
        "runtime": {
            "requires": [
                "python2.7:from<native>"
            ]
        }
}

Say I parse the section using System::Query, what do I suppose to do with the data?

• python2.7:from<native>
• python2.7-config

How does that translate to a OS specific ( say Debian ) system packages?

[melezhik]

I mean I can't find such a Debian package named as python2.7-config.

Also for python2.7:from<native> what do I suppose to do with that?

For CentOS there is package named as python27, for Debian - python2.7 , does System::Query make such a mapping for me?

[melezhik]

And what is the difference between foo and foo:from<native> notation in dependencies section?

[niner]

An example for how to map this dependency information to system specific dependencies (i.e. package names): https://github.com/niner/meta2rpm/blob/master/meta2rpm.pl6#L110 That's however an early version that relies on things already being installed. The next step would be to use zypper to query the package database. You should be able to ask tools like zypper or apt or dnf for a package containing /usr/bin/perl or libfoo.24.1.so

For example a "ffcall:from<native>:ver<0.0.2>" can be resolved via:

nine@sphinx:~> zypper -q se --provides /usr/lib64/libffcall.so.0.0.2
S  | Name       | Summary                                        | Type
---+------------+------------------------------------------------+--------
i+ | libffcall0 | Libraries for foreign function call interfaces | package

[ugexe]

foo:from<native> basically checks if sub foo() is native($*VM.platform-library-name("foo".IO).basename) { }() works. If it works then the library must be installed.

foo is short for foo:from<Perl6>

does System::Query make such a mapping for me?

No. This is basically giving you the normalized name that mappings should be made against.

[melezhik]

No. This is basically giving you the normalized name that mappings should be made against.

So does that mean I still need to implement mapping somewhere ? This is what I've always tried to know in all previous discussions ))) And also that means META does not provide OS specific information in that case, only a "normalized" name as you say, to start search with?

What about "python2.7-config"|"python2-config" what do I suppose to do with those chunks?

[ugexe]

Search the PATH for python2.7-config or find a package providing it

[melezhik]

or find a package providing it

Ah ok, that mean in that case it also a "normalized" name, is it?

[melezhik]

So it seems to me now, that META file provides normalized names only, so the mapping of normalized names to OS specific native packages should be done in "client" code?

[ugexe]

Well, its normalized per whatever constraints are in the META. In your example there are two different normalized names:

                        "by-distro.name" : {
                            "macosx" : "python2.7-config",
                            "debian" : "python2.7-config",
                            "" : "python2-config"
                        }

so by "normalized" we also don't mean universally, since python2.7-config is likely doesn't work on e.g. windows (which might be python2-config).

This is all the logic necessary to see if something is installed, which is what zef needs. Now for something like installing native dependencies you need the additional step of mapping that "normalized" name to whatever native package manager understands (probably python2 for above example) and feeding it to e.g. apt-get.

[melezhik]

Now for something like installing native dependencies you need the additional step of mapping that "normalized" name to whatever native package manager understands (probably python2 for above example) and feeding it to e.g. apt-get.

@ugexe - Sure. we are no the same page now.

An example for how to map this dependency information to system specific dependencies (i.e. package names): https://github.com/niner/meta2rpm/blob/master/meta2rpm.pl6#L110

@niner - thanks for that, this is what I'd guessed ...

so by "normalized" we also don't mean universally,

@ugexe btw - this is what I called "meta" packages and suggest here - https://github.com/Raku/ecosystem/issues/505#issuecomment-644446574 😄

[melezhik]

Interesting enough ... , is $*VM.platform-library-name a core Raku function?

[melezhik]

This is all the logic necessary to see if something is installed

@ugexe I am not sure, but looks like it is not. The current version of dependency resolution is limited as there is a variety of paths to search through - https://github.com/niner/meta2rpm/blob/master/meta2rpm.pl6#L117

On my CentOSm for perl:from<native> /usr/lib64/perl5/CORE should have been added:

my $path = </usr/lib64 /lib64 /usr/lib64/perl5/CORE /usr/lib /lib>.first({$_.IO.add($lib).e});

I am not sure if zef client code cares about all SEARCH paths in that respect.

So yeah, it needs to be replaced by OS depended database query probably ....

[ugexe]

Then your code wouldnt work with sub foo() is native($*VM.platform-library-name("perl")) { } and any tests would fail anyway. The entire point is you can list the dependencies the same as you do in your code -- as their library names -- because we test if they are installed by seeing if NativeCall can load the given library name.

[melezhik]

@ugexe I am trying to say this. Say I declare a dependency as a perl:from<native> in META

In my CentOS I have the library installed as a /usr/lib64/perl5/CORE/libperl.so

If zef for example only searches within:

my $path = </usr/lib64 /lib64 /usr/lib /lib>.first({$_.IO.add($lib).e});

as @niner has in his implementation, the test will wrongly fails ( not finding the library ).

So my point is you are going to end up maintaining in zef client code (or any other application implementing such a dependency check ) all the variety of search paths for libraries or you code will produce false negative results. And looks like it's very tedious and hard to achieve task, or am I missing the point ?

[ugexe]

Are you familiar with NativeCall at all? How would NativeCall find some module on a special path? If zef just magically guesses possible library locations and says "oh hey this is installed in some special place!" but NativeCall doesn't see it what good is it?

[melezhik]

well I judged upon the code referenced by @niner it does not work on my system this is what I'm saying , if you'd show how that it works in zef I'd appreciate ... so far I don't see any native calls here ...

[ugexe]

Its pretty easy to find... https://github.com/ugexe/zef/blob/cb15ffc9be8a16b2e84af62ec48a37c1b4f3b564/lib/Zef/Client.pm6#L714-L723

[melezhik]

On my system I have this:

[scheck@sp6 zef]$ cat check.pl6 use NativeCall; sub native_library_is_installed is native("perl") { }; native_library_is_installed();

[scheck@sp6 zef]$ perl6 check.pl6 Cannot locate native library 'libperl.so': libperl.so: cannot open shared object file: No such file or directory in method setup at /opt/rakudo-pkg/share/perl6/core/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 287 in method CALL-ME at /opt/rakudo-pkg/share/perl6/core/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 576 in block at check.pl6 line 3

[scheck@sp6 zef]$ rpm -ql perl-libs /usr/lib64/perl5 /usr/lib64/perl5/CORE/libperl.so /usr/lib64/perl5/vendor_perl /usr/lib64/perl5/vendor_perl/auto

Do I do anything wrong?

[ugexe]

No, but this proves my point. It doesn’t make sense for zef to find your Perl library if NativeCall itself can’t find it (and hence any libraries using it won’t find it). It’s possible NativeCall could improve how it finds libraries (I don’t really know), but that would be an issue for rakudo.

[melezhik]

yeah ... so looks like your (zef's) approach to see if library is installed is to NativeCall on it, conversely @niner is trying to do it yourself by searching among some predefined paths:

my $path = </usr/lib64 /lib64 /usr/lib64/perl5/CORE /usr/lib /lib>.first({$_.IO.add($lib).e});

@niner is it correct?

[melezhik]

For example a "ffcall:from:ver<0.0.2>" can be resolved via:

nine@sphinx:~> zypper -q se --provides /usr/lib64/libffcall.so.0.0.2 S | Name | Summary | Type ---+------------+------------------------------------------------+-------- i+ | libffcall0 | Libraries for foreign function call interfaces | package

@niner in your case you still have calculate path to ffcall *.so library first, which as we've already established could be a problematic.

[niner]

@niner in your case you still have calculate path to ffcall *.so library first, which as we've already established could be a problematic.

Not really. I can just query ldconfig -v to get all the paths that the system is looking for libraries and then simply try them with zypper. That just wastes a few cycles but other than that it has no real down side.

The important part is that there is usually a way to get from the name of a shared library to the package without having to create a huge map of all libraries on all distributions. Instead one just has to find this way once per distribution which makes it quite feasible. Especially if we work together.

[melezhik]

thanks @niner

Not really. I can just query ldconfig -v

Are you suggesting to do the same trick with :from<bin> depedencies?

So if I follow you suggest following:

1. get dependency from META (either as from:<bin> or from:<native> format)
2. resolve os respected absolute path to library/script using ldconfig/$PATH and friends
3. by having absolute path find a package name using native method - https://www.ostechnix.com/find-package-provides-specific-file-linux/
4. check if a package is installed and if not install it

is it correct?

[niner]

Yes, that's pretty much the plan.

[melezhik]

resolve os respected absolute path to library/script using ldconfig/$PATH and friends

the problem could be with that one. chicken and egg. without installing a library we don't know it's system path and visa versa ... we could only make a guess, for example for python2.7-config iterate through all possible options:

• /usr/bin/python2.7-config
• /bin/python2.7-config
• /usr/local/bin/python2.7-config

so on so forth ... seems not effective?

The same picture for finding path for perl:from<:native>, also which is probably even harder without knowing a version ... I don't know ...

[melezhik]

@ugexe btw, if zef does not see that an external dependency is met does it only notify or throw an exception, how do I see such an information in zef log?

[niner]

On Montag, 22. Juni 2020 23:44:25 CEST Alexey Melezhik wrote:

for example for python2.7-config iterate through all possible options:

• /usr/bin/python2.7-config
• /bin/python2.7-config
• /usr/local/bin/python2.7-config

so on so forth ... seems not effective?

It is a bit inefficient, but certainly effective. In other words, it's some additional work that's unfortunate, but it does work and will get us results. Remember that the shell does exactly this every time you run a command: it has to go through $PATH and look at all the places where a command might be. And that's not a problem there, so it shouldn't be one on installation (which is going to take some time anyway).

The same picture for finding path for perl:from<:native>, also which is probably even harder without knowing a version ... I don't know ...

libperl.so is a very special case, as it doesn't get installed into a standard location. That's why I only declared a dependency on the perl binary itself: https://github.com/niner/Inline-Perl5/blob/master/META6.json#L13 and then run perl to find the actual location (in the form of compiler switches) of the shared library: https://github.com/niner/Inline-Perl5/blob/master/META6.json#L58

In the general case, you should always have a version for a native library. Otherwise you will get any version and this will cause compatibility issues. In most cases there won't even be a libfoo.so installed, only the versioned libfoo.so.1. Yes, module authors are still a bit sloppy about this and we have to educate them.

[melezhik]

@niner one more question:

and look at all the places where a command might be

how do you deal with the case when dependency is not yet installed? how do you know a package name, don't you suggest to iterate on those possible paths and make a search? :

my @paths = ( '/usr/bin/', /bin/', '/usr/local/bin/');
for @paths ->$p {
  `/usr/lib/rpm/find-provides $p/python2.7-config`
}

[melezhik]

Another question. Termbox has this:

    "depends"        : {
        "runtime": {
            "requires": [
                "NativeCall",
                "NativeLibs:ver<0.0.7>:auth<github:salortiz>",
                "JSON::Fast",
                "curl:from<native>"
            ]
        },
        "test": {
            "requires": [ "Test", "Test::When" ]
        }
    },

on my CentOS I have this:

[scheck@sp6 raku-native-deps]$ rpm -ql libcurl
/usr/lib64/libcurl.so.4
/usr/lib64/libcurl.so.4.3.0

I have no idea how to map "curl:from<native>" to /usr/lib64/libcurl.so.4 file,

obviously any libcurl.so mapping does not work, because we have libcurl of certain (4?) version:

[scheck@sp6 raku-native-deps]$ rpm -qf /usr/lib64/libcurl.so
error: file /usr/lib64/libcurl.so: No such file or directory

So this would work:

[scheck@sp6 raku-native-deps]$ rpm -qf /usr/lib64/libcurl.so.4
libcurl-7.29.0-51.el7.x86_64

but we don't have a version information in META file, nor it will make any sense because various distros would have different libcurl versions ...

So how do you deal with that?

[melezhik]

yum whatprovides abit better, though I am not sure we would have equivalents on other Linxues ...

[scheck@sp6 raku-native-deps]$ yum whatprovides libcurl

Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * centos-sclo-rh: mirrors.sonic.net
 * epel: sjc.edge.kernel.org
libcurl-7.29.0-57.el7.i686 : A library for getting files from web servers
Repo        : base

libcurl-7.29.0-57.el7.x86_64 : A library for getting files from web servers
Repo        : base

libcurl-7.29.0-51.el7.x86_64 : A library for getting files from web servers
Repo        : @base

[melezhik]

looks like rpm -q --whatprovides $foo works more or less fine for the most of modules listed in another issue.

[melezhik@localhost raku-native-deps]$ rpm -q --whatprovides gpgme curl python perl sqlite libcurl
gpgme-1.3.2-5.el7.x86_64
curl-7.29.0-54.el7_7.2.x86_64
python-2.7.5-86.el7.x86_64
perl-5.16.3-294.el7_6.x86_64
sqlite-3.7.17-8.el7_7.1.x86_64
libcurl-7.29.0-54.el7_7.2.x86_64

Still questions here:

• could we find equivalent command to calculate package names for other Linux distros?
• in case of from<foo:native> should we search as rpm -q --whatprovides $foo or pm -q --whatprovides lib$foo ? looks like sometimes both ways are implied, but should we require a module author always set this explicitly, e.g

{
"libcurl:from<native>"
"curl:from<native>"
}

results in 2 different native CentOS packages on my system:

• libcurl-7.29.0-54.el7_7.2.x86_64
• curl-7.29.0-54.el7_7.2.x86_64

And does this logic makes a sense for a zef? which as far I understand just use different NativeCall semantic trying to see if a foo is installed:

use NativeCall; 
sub native_library_is_installed is native("''$foo', v$version") { !!! }; 
native_library_is_installed(); 

cc @niner @ugexe

[niner]

rpm and find-provides will only search locally. Note that meta2rpm has a slightly different goal: it translates the dependency information from META6.json into dependency information for a .spec file. It is good practice to specify the name of the library instead of a package name for rpm dependencies: Requires: libfcgi.so.0()(64bit) Same is true for binaries: Requires: /usr/bin/raku It's even better practice to have such dependencies be generated automatically via helper scripts in rpmbuild. If your package for example contains a script that starts with #!/usr/bin/raku rpmbuild will automatically add that dependency for you.

An installer like zef would have to use the appropriate tool to query the package repository, i.e. zypper on openSUSE, dnf on Fedora or apt-get (or whatever) on Debian. Or maybe there are more appropriate low level tools.

Your Termbox example shows that our module authors are still mostly using NativeCall wrongly. That's no wonder because until recently, even our documentation gave the wrong impression. The version is really not optional. There should always be a version with very few exceptions. This is true both for the dependency information in the META6.json and the code in the module itself. sub foo is native('curl') is wrong, sub foo is native('curl', v4) would be correct.

Note that v4 is the SO version, not the version of the software or package. That's the same as our :api in module names:

nine@sphinx:~> rpm -qf /usr/lib64/libcurl.so.4
libcurl4-7.70.0-1.2.x86_64

All libcurl.so.4 are compatible with each other, so it doesn't matter which exact curl version a distro ships.

Btw. I don't see that dependency information in https://modules.raku.org/dist/Termbox:cpan:JJATRIA/META6.json did you mean another module?

could we find equivalent command to calculate package names for other Linux distros? Yes. I would be very surprised otherwise. All distros have package repositories. These repositories usually contain databases with the package's contents. They have to because that's just necessary for creating a good package manager.

For NativeCall, when you write sub foo() is native('foo', v1) this means that you want to load libfoo.so.1 on Linux, libfoo.1.dylib on MacOS and foo.dll on Windows. The Windows case is why the "lib" is implicit and we don't spell it out. "curl:from<native>:ver<1>" means you need a libfoo.so.1 which means you need the libcurl package. "curl:from<bin>" means you need the curl command which means you need the curl package.

[melezhik]

Hi @niner , thanks for answers. But I am still confused now. Please clarify, answering these questions

Your Termbox example shows that our module authors are still mostly using NativeCall wrongly.

what is wrong?

An installer like zef would have to use the appropriate tool to query the package repository, i.e. zypper on openSUSE, dnf on Fedora or apt-get (or whatever) on Debian. Or maybe there are more appropriate low level tools.

So you mean, zef should not use nativecall semantic to see that dependencies are meet? Should we use the nativecall method at all?

sub foo is native('curl', v4) would be correct

None of combinations of library names, versions never worked on my system, I wonder if such a nativecall sematic is ever worked, all the time I have a variations of cannot open shared object file: No such file or directory error.

@niner @ugexe please show me a certain example that verify that libcurl/curl is installed on your machine.

Your Termbox example shows that our module authors are still mostly using NativeCall wrongly. That's no wonder because until recently, even our documentation gave the wrong impression. The version is really not optional.

what do you mean by version? I am confused again.

[niner]

On Mittwoch, 24. Juni 2020 15:00:18 CEST Alexey Melezhik wrote:

Your Termbox example shows that our module authors are still mostly using NativeCall wrongly. what is wrong?

I thought I did? It's usually wrong to leave out the version of a native library. Wrong: sub foo() is native('foo') { ... } depends: [ 'foo:from<native>' ] Right: sub foo() is native('foo', v2) { ... } depends: [ 'foo:from<native>:ver<2>' ]

So you mean, zef sha not use nativecall semantic to see that dependencies are meet? Should we use the nativecall method at all?

It's perfectly ok to use NativeCall to check if the library is installed. After all, that is closest to what happens at runtime. It will have to use a distro dependent tool to install a missing dependency.

In peudo code:

unless nativecall_library_loads($library, $version) {
    install_using_distro_tool($library, $version);
}

sub foo is native('curl', v4) would be correct

None of combinations of library names, versions never worked on my system, I wonder if such a nativecall sematic is ever worked, all the time I have a variations of

That depends on what you tried.

cannot open shared object file: No such file or directory

error. @niner  @ugexe  please show me a certain example that verify that
libcurl/curl is installed on your machine

nine@beta:~> rpm -q libcurl4 
libcurl4-7.63.0-247.4.x86_64
nine@beta:~> ll /usr/lib64/libcurl.so.4
lrwxrwxrwx 1 root root 16  9. Jän 2019  /usr/lib64/libcurl.so.4 -> libcurl.so.4.5.0
nine@beta:~> rakudo -e 'use NativeCall; sub curl_easy_init(--> Pointer) is native("curl", v4) { !!! }; dd curl_easy_init;'
NativeCall::Types::Pointer.new(64331248)

what do you mean by version? I am confused again.

https://docs.raku.org/language/nativecall#ABI/API_version

[melezhik]

@niner thanks for quick reply

on my machine, I have this: ( notice that rpm -q libcurl4 does not work for me)

[scheck@sp6 ~]$  rakudo -e 'use NativeCall; sub curl_easy_init(--> Pointer) is native("curl", v4) { !!! }; dd curl_easy_init;'
NativeCall::Types::Pointer.new(63933264)

[scheck@sp6 ~]$ rpm -q libcurl4
package libcurl4 is not installed

[scheck@sp6 ~]$ rpm -q libcurl
libcurl-7.29.0-51.el7.x86_64

[scheck@sp6 ~]$ rpm -ql libcurl
/usr/lib64/libcurl.so.4
/usr/lib64/libcurl.so.4.3.0

Right: sub foo() is native('foo', v2) { ... } depends: [ 'foo:from<native>:ver<2>' ]

So do you mean, these are SO versions and they are not native packages versions, right? Is that approach is portable across different Linuxes?

https://docs.raku.org/language/nativecall#ABI/API_version

Thanks for that

Btw. I don't see that dependency information in https://modules.raku.org/dist/Termbox:cpan:JJATRIA/META6.json did you mean another module?

Yeah, sorry. I meant LibCurl module - https://github.com/CurtTilmes/raku-libcurl/blob/master/META6.json

And the most difficult bit for me:

This SO version, do we need it to only check is a library is installed via nativecall semantic, or do we need it for something else?

As far as understand to map library name taken from META.json to native package name we only need a library name not a SO version, because it has nothing to do with native package version ... Thanks in advance ...

[niner]

On Mittwoch, 24. Juni 2020 15:51:35 CEST Alexey Melezhik wrote:

on my machine, I have this: ( notice that rpm -q libcurl4 does not work for me)

That's precicely why it's better to state dependencies in terms of shared library name instead of some package name.

sub foo() is native('foo', v2) { ... } depends: [ 'foo:from<native>:ver<2>' ]

So do you mean, these are SO versions and they are not native packages versions, right? Is that approach is portable across different Linuxes?

Exactly! And yes, that's true on Linux in general.

Btw. I don't see that dependency information in https://modules.raku.org/dist/Termbox:cpan:JJATRIA/META6.json did you mean another module? Yeah, sorry. I meant LibCurl module - https://github.com/CurtTilmes/raku-libcurl/blob/master/META6.json

Ah, yes. Looking at the module's source code, we see that it actually needs curl:from<native>:ver<4>: https://github.com/CurtTilmes/raku-libcurl/blob/master/lib/LibCurl/ EasyHandle.rakumod#L5

As far as understand to map library name taken from META.json to native package name we only need a library name not a SO version, because it has nothing to do with native package version ... Thanks in advance ...

No, we actually need the version to find the package name! For example on my system there is no libcurl.so, there are only the versioned ones:

nine@sphinx:~> ls /usr/lib/libcurl*
/usr/lib/libcurl.so.4  /usr/lib/libcurl.so.4.6.0

Many modules are doing this wrong currently. This is why e.g. the OpenSSL module requires me to install a package like libssl1.0-dev https://github.com/sergot/openssl/issues/59

So we need the version to:

• check if the library is installed
• install the library if its missing and
• load the libary

That is: we always need the version. There are workarounds like installing some -dev or -devel package if the version is missing, but they are just workarounds. We need to educate module authors about this and we need to send them fixes for these errors. I've had patches for OpenSSL ready for half a year. They just wait for improvements to zef to support alternatives, as OpenSSL actually supports openssl v1 and v1.1

[melezhik]

@niner thanks, you pretty much answered all my questions, makes more sense now.

Still need this bit:

No, we actually need the version to find the package name! For example on my system there is no libcurl.so, there are only the versioned ones:

So do i:

[scheck@sp6 ~]$ rpm -ql libcurl /usr/lib64/libcurl.so.4 /usr/lib64/libcurl.so.4.3.0

Could you please describe formal steps for this certain case ( CentOS/libcurl ) how we calculate a native package name, say in the situation we on fresh machine, and libcurl is not installed here. Let's start from correct META6.json declaration to the very end ...

[niner]

Could you please describe formal steps for this certain case ( CentOS/libcurl ) how we calculate a native package name, say in the situation we on fresh machine, and libcurl is not installed here. Let's start from correct META6.json declaration to the very end ...

Ok, given the dependency stated as: 'depends': [ "curl:from<native>:ver<4>" ]

we can use $*VM.platform-library-name("curl".IO, :version(Version.new(4))) to get the proper library file name libcurl.so.4

On openSUSE I use zypper to search for the package name:

nine@sphinx:~> zypper -q --xmlout se --provides 'libcurl.so.4()(64bit)'
<?xml version='1.0'?>
<stream>

<search-result version="0.0">
<solvable-list>
<solvable status="installed" name="libcurl4" summary="Library for transferring 
data from URLs" kind="package"/>
</solvable-list>
</search-result>
</stream>

On CentOS, RHEL and Fedora, dnf should do the same, but unfortunately I cannot test:

dnf --quiet repoquery --whatprovides 'libcurl.so.4()(64bit)'

That libcurl.so.4()(64bit) part looks strange, but it's pretty straight forward. Just add ()(64bit) on a 64 bit machine and it's OK. I forgot what the first parenthesis means, but it's usually empty. That's rpm syntax, so it should be the same for all distributions based on rpm.

[melezhik]

cool. more or less what I'd expected, I'll try to implement it ...

but how do we handle foo:from<bin> statements, in that case foo is not a library, but script, is not it? and that approach won't work, right?

[melezhik]

@niner another point, dnf search is not fast, at least at my machine:

[melezhik@localhost raku-native-deps]$ time dnf --quiet repoquery --whatprovides 'libcmark.so.0.23.0()(64bit)'
cmark-lib-0:0.23.0-4.el7.x86_64

real    0m8.819s
user    0m3.483s
sys 0m0.249s

[melezhik]

strange enough, after awhile, it's dropped to an order:

[melezhik@localhost raku-native-deps]$ time dnf --quiet repoquery --whatprovides 'libcmark.so.0.23.0()(64bit)'
cmark-lib-0:0.23.0-4.el7.x86_64

real    0m0.563s
user    0m0.458s
sys 0m0.104s

[melezhik]

@niner for some libraries we could have more then one packages:

[melezhik@localhost raku-native-deps]$ dnf --quiet repoquery --whatprovides 'libcurl.so.4()(64bit)'
libcurl-0:7.29.0-57.el7.x86_64
mdatp-0:100.87.66-1.x86_64
mdatp-0:100.90.70-1.x86_64

[melezhik]

@niner @ugexe I've create a simplistic sparrow plugin raku-ext-deps which parses META spec and generates native packages names where it possible, thus producing output for native package managers. It's quite primitive now, only handles from<native> blocks, does not use System::Query to handle os specific section, only works for CentOS, but it'd give your a sense of my Sparrow approach I'd like to present:

For example for this example META6.json

{
    "depends"        : {
        "runtime": {
            "requires": [
                "NativeCall",
                "NativeLibs:ver<0.0.7>:auth<github:salortiz>",
                "JSON::Fast",
                "curl:from<native>",
                "curl:from<native>:ver<4>",
                "cmark:from<native>:ver<0.23.0>"
            ]
        }
    }
}

The plugin will produce a following:

Run as Raku API:

my %state = task-run "get packages", "raku-native-deps", %(
  path => "META6.json"
);

for %state<packages><> {
  say "package: $i<package>"
}

package: libcurl-0:7.29.0-57.el7.i686
package: libcurl-0:7.29.0-57.el7.x86_64
package: libcurl-0:7.29.0-57.el7.x86_64
package: mdatp-0:100.87.66-1.x86_64
package: mdatp-0:100.90.70-1.x86_64

Run as cli ( which makes a sense when one just to want to see generated dependencies ).

$ s6 --plg-run raku-native-deps@path=META6.json
21:37:11 06/24/2020 [get packages] parse META from META6.json ...
21:37:11 06/24/2020 [get packages] parse curl .. 
21:37:11 06/24/2020 [get packages] parse curl .. :ver<4>
21:37:11 06/24/2020 [get packages] parse cmark .. :ver<0.23.0>
21:37:11 06/24/2020 [get packages] libraries found
21:37:11 06/24/2020 [get packages] ===========================
21:37:11 06/24/2020 [get packages] ${:library("curl"), :version("")}
21:37:11 06/24/2020 [get packages] ${:library("curl"), :version("4")}
21:37:11 06/24/2020 [get packages] ${:library("cmark"), :version("0.23.0")}
21:37:11 06/24/2020 [get packages] ===========================
21:37:13 06/24/2020 [get packages] map library: [curl] version: [] to native package, os: [centos] ...
21:37:13 06/24/2020 [get packages] ===>
21:37:13 06/24/2020 [get packages] libcurl-0:7.29.0-57.el7.i686
21:37:13 06/24/2020 [get packages] libcurl-0:7.29.0-57.el7.x86_64
21:37:13 06/24/2020 [get packages] <===
[task check] stdout match (r) <(\S+)> True
[task check] push libcurl-0:7.29.0-57.el7.i686 ...
[task check] push libcurl-0:7.29.0-57.el7.x86_64 ...
21:37:15 06/24/2020 [get packages] map library: [curl] version: [4] to native package, os: [centos] ...
21:37:15 06/24/2020 [get packages] ===>
21:37:15 06/24/2020 [get packages] libcurl-0:7.29.0-57.el7.x86_64
21:37:15 06/24/2020 [get packages] mdatp-0:100.87.66-1.x86_64
21:37:15 06/24/2020 [get packages] mdatp-0:100.90.70-1.x86_64
21:37:15 06/24/2020 [get packages] <===
[task check] stdout match (r) <(\S+)> True
[task check] push libcurl-0:7.29.0-57.el7.x86_64 ...
[task check] push mdatp-0:100.87.66-1.x86_64 ...
[task check] push mdatp-0:100.90.70-1.x86_64 ...
21:37:16 06/24/2020 [get packages] map library: [cmark] version: [0.23.0] to native package, os: [centos] ...
21:37:16 06/24/2020 [get packages] ===>
21:37:16 06/24/2020 [get packages] cmark-lib-0:0.23.0-4.el7.x86_64
21:37:16 06/24/2020 [get packages] <===
[task check] stdout match (r) <(\S+)> True
[task check] push cmark-lib-0:0.23.0-4.el7.x86_64 ...

[melezhik]

@niner @ugexe here is the first successful example of handling native dependencies from META by Sparrow/RakuDist, I had to change META file according @niner recommendations and is going to make a PR to LibCurl module soon.

http://rakudist.raku.org/sparky/report/centos/410

[melezhik]

@niner what do think we could put for sqlite - https://modules.raku.org/dist/DB::SQLite:cpan:CTILMES/META6.json#L16 ?

does not work for me:

19:05:29 06/25/2020 [.] map library: [sqlite3] version: [0.8.6] to native package, os: [centos], arch: [x86_64] ...
19:05:29 06/25/2020 [.] ===>
19:05:29 06/25/2020 [.] stderr: ++ dnf --quiet repoquery --whatprovides 'libsqlite3.so.0.8.6()(64bit)' --arch x86_64
19:05:30 06/25/2020 [.] stderr: ++ set +x
19:05:30 06/25/2020 [.] <===
[task check] stdout match (r) <(\S+)> False

[melezhik]

I can't search sqlitie that way, though it's presented as:

[melezhik@localhost raku-native-deps]$ rpm -ql sqlite-0:3.7.17-8.el7_7.1.x86_64
/usr/bin/sqlite3
/usr/lib64/libsqlite3.so.0
/usr/lib64/libsqlite3.so.0.8.6
/usr/share/doc/sqlite-3.7.17
/usr/share/doc/sqlite-3.7.17/README
/usr/share/man/man1/sqlite3.1.gz

[niner]

On Donnerstag, 25. Juni 2020 21:10:01 CEST Alexey Melezhik wrote:

does not work for me:

19:05:29 06/25/2020 [.] map library: [sqlite3] version: [0.8.6] to native
package, os: [centos], arch: [x86_64] ... 19:05:29 06/25/2020 [.] ===>
19:05:29 06/25/2020 [.] stderr: ++ dnf --quiet repoquery --whatprovides
'libsqlite3.so.0.8.6()(64bit)' --arch x86_64 19:05:30 06/25/2020 [.]
stderr: ++ set +x
19:05:30 06/25/2020 [.] <===
[task check] stdout match (r) <(\S+)> False

I think the SO version is just 0, so dnf repoquery --whatprovides 'libsqlite3.so.0()(64bit)' --arch x86_64 should give something

[melezhik]

@niner thanks, let me give it a try ...