API for external dependecies

[melezhik]

Hi Nick!

I am writing this as an issue, though it's just a question. What an API does zef provide for external dependencies?

With regard to this discussion I am thinking about creating of prototype so that Sparrow tool install a module and it's external dependencies where external dependencies data comes from zef API.

Thanks

Aleksei

[melezhik]

@niner , works for me, thanks!

[task check] stdout match (r) <(\S+)> True
19:28:43 06/25/2020 [.] map library: [sqlite3] version: [0] to native package, os: [centos], arch: [x86_64] ...
19:28:43 06/25/2020 [.] ===>
19:28:43 06/25/2020 [.] stderr: ++ dnf --quiet repoquery --whatprovides 'libsqlite3.so.0()(64bit)' --arch x86_64
19:28:44 06/25/2020 [.] sqlite-0:3.7.17-8.el7_7.1.x86_64
19:28:44 06/25/2020 [.] stderr: ++ set +x
19:28:44 06/25/2020 [.] <===

[melezhik]

@niner some modules require headers files for libraries , for example https://modules.raku.org/dist/GDBM - "In order to install this you will need to have the GDBM development packages installed in order to build the wrapper this library requires on Linux" - how do we calculate native packages for them?

[melezhik]

@ugexe I am thinking about running Sparrow as a zef plugin so that it handle native dependencies from META, what do you think about the idea?

Obviously raku-native-deps and install-package Sparrow plugins needs to be triggered for every module's dependency's META ( not just for module's META ), so it better do it through zef, by running Sparrow as an external plugin ...

[ugexe]

Yes, something like that could work. Installation would probably be fairly easy... adding support for resolving if a given native dependency is available through whatever package manager will be a little tougher since the dependency resolving code isn't setup with adapter pattern yet.

[melezhik]

Sorry, I did not get the last statement "adding support for resolving if a given native dependency is available through whatever package manager will be a little tougher" ? On my view it won't be a responsibility of zef, but of external plugin (Sparrow), this is how I see it:

zef install Foo -> get Raku dependencies ( module A, module B, so on )
for every Raku dependency - download module distribution and unpack it ( say for module A ) 
and run `raku-native-deps` sparrow plugin for META6.json, the plugin generates native packages list, 
install native packages if required using `package-install` plugin

[ugexe]

For a naive implementation yeah that would work. But it doesn't allow zef to ever say a native dependency is missing -- it would have to always assume that if a native dependency is not installed that a later step will find it. That is a problem when handling optional dependencies (which will exist soonish) because we can't build the graph ahead of time. To fix that we pretty much need the dependency resolver to 1) handle :from<native> different and 2) give a way to say 'yes, the plugin / sparrow-resolver thinks it can find this dependency, so we can choose this optional dependency path'. Zef doesn't really need to know anything about that native dependency from this sparrow plugin, just if a plugin thinks it can find it.

[melezhik]

yeah. i see that. I'd prefer having zef first to build a full from:native list for a module and it's dependencies and then have any external tool like Sparrow to handle such a list installing packages if required and then just run zef install again assuming all the native dependencies are set for that moment of time ...

so i still doubt that converting zef into native packages manager is a good idea. the maximum it has to do is to check that dependencies are met.

[ugexe]

Native dependencies might be part of a optional dependency path that also contains Perl 6 code, i.e. "depends" : [ "any" : ["Foo:from<native>", "Bar:from<Perl6>"] ] which cannot know if it should take Bar (a raku dependency) over Foo (a native dependency). So for it to even do what you suggest it needs to be able to resolve ahead of time if whatever plugin is going to find a match.

[melezhik]

so it could be something like that:

zef from-native-list Foo  > list.txt
s6 -plg-run raku-ext-deps@in=list.txt@out=out.txt
s6 -plg-run install-deps@in=out.txt
zef install Foo

[melezhik]

so , yeah if zef is responsible for which version Raku or native to take as "any" it is not a problem in the approach I've mentioned in the previous comment.

[melezhik]

So for it to even do what you suggest it needs to be able to resolve ahead of time if whatever plugin is going to find a match.

So I guess zef from-native-list Foo would solve it?