Add external dependencies checking to ecosystem

[JJ]

Now that @melezhik 's rakudist system is in place, could we maybe add something like that to check for external dependencies? Right now everything with an external library is failing.

[melezhik]

hi @jj. definitely ☺️. this is one if advantage of RakuDist/Sparrow I try to emphasize. But could you please outline some users scenarios for such a checks, so I could give more specific output for that task.

[JJ]

Thanks!

[melezhik]

@JJ I don't know how do you see the whole task, but you might be interested in that post - https://sparrowdo.wordpress.com/2020/01/28/rakudist-update-raku-modules-installation-customization/

the approach of how RakuDist handle external dependencies could be somehow expanded/adjusted to solve the problem.

maybe this could help

[JJ]

Will check it out. The thing is that META6.json already has a way to define external dependencies-ish. So we should try and use it for this, instead of any external ad-hoc way.

[melezhik]

We have already discussed pretty much of that topic in this issue - https://github.com/Raku/problem-solving/issues/144

If people are diligent enough to provide external dependencies info through Meta, then let it be. RakuDist uses Sparrow automation which could parse Meta, or run zef and parse zef output to get external libraries info or even delegate external library installations to zef.

I have already highlighted potential problems here in the mentioned GitHub issue.

At the end of the day Sparrow is just a tool and it is flexible enough and could adapt to any scheme.

I personally believe that Meta is still not the best choice here, but I may be wrong.

cc @ugexe

[melezhik]

As agreed on IRC talk, we could take it for a spin for this project

https://github.com/niner/Inline-Python/blob/master/META6.json

So that Sparrow would pick up dependencies from META file and installs them during RakuDist test.

@ugexe I think zef could probably provide info from META file in programmatic way?

@JJ

• where can I find a META specification documentation?
• where can I find CI pipeline that triggered upon new modules release to eco system, if any?

[AlexDaniel]

Yeah, META6.json provides a way to define external dependencies, but barely anyone is using this feature. Just go through all modules and submit PRs adding the required section.

Here's a list that will keep you busy for at least some time:

• https://github.com/Raku/ecosystem-unbitrot/wiki

After some progress is done we can see where we stand and what can be done using this information.

[AlexDaniel]

where can I find CI pipeline that triggered upon new modules release to eco system, if any?

Unless somebody adds travis to their repo there's no CI pipeline. But we test all modules with Blin regularly.

[melezhik]

Yeah, META6.json provides a way to define external dependencies, but barely anyone is using this feature. Just go through all modules and submit PRs adding the required section.

I would not harry with that. Till we know that user are happy with that approach.

I'd test that on some modules and see how it goes and get some feedback

[AlexDaniel]

Why would somebody be unhappy to receive a PR?

[melezhik]

Unless somebody adds travis to their repo there's no CI pipeline. But we test all modules with Blin regularly.

Let's decide. RakuDist does not need any CI to run tests, it's standalone service. However I understand that many users use travis/whatever CI.

What do we want from this perspective?

a) test eco system modules independently from users ( some cron script would run tests on modules or new tests gets triggered whenever module updated/released in eco system ) - only notifying them when problems occur

b) allow users to set up tests for their modules using custom travis/CI scripts ( and thus running some Sparrow/RakuDist API from those scripts )

[melezhik]

Why would somebody be unhappy to receive a PR?

it's not a PR that might make them unhappy, but the idea they have to declare external dependencies in META file. Is it already "official" approach?

[AlexDaniel]

Is it already "official" approach?

Yeah, pretty much. It's just that almost nobody is using it. You can create a problem-solving ticket arguing that it's wrong, or something, but I don't think it'll go far. If META6.json is used to define raku dependencies then surely native dependencies should be defined in the same file too. We can maybe argue about the structure of the section that describes the native deps, but other than that there's no much room for debate.

What do we want from this perspective?

a) test eco system modules independently from users […] - only notifying them when problems occur

We already do that. We test all modules with Blin, and then sometimes update unbitrot repo using the data from Blin.

b) allow users to set up tests for their modules using custom travis/CI scripts ( and thus running some Sparrow/RakuDist API from those scripts )

People already just use Travis to test their modules. It works fine.

[melezhik]

We already do that. We test all modules with Blin, and then sometimes update unbitrot repo using the data from Blin.

People already just use Travis to test their modules. It works fine.

These two options imply very different implementations and are very different from each other

Also. If we decide to go with Blin, I am now confused how RakuDist could be useful here?

cc @JJ "Now that @melezhik 's rakudist system is in place, could we maybe add something like that to check for external dependencies?"

Imho we either use RakuDist or Blin. I'd agree they looks like similar products but they solve slightly different tasks and with different approaches ...

[melezhik]

Yeah, pretty much. It's just that almost nobody is using it.

That's ok. I am not arguing that. Just wanted to clarify

[nxadm]

I think the discussion touches on 2 very different tasks/setups:

There is the Smoke-testing, like we know from Perl, that pre-dates the popularity of CI frameworks. I think of the continuous testing a matrix of rakudo releases and modules on as many platforms as possible, without a user-action or -defined triggers. This is useful for release managers and for testing on less used architectures. Key is displaying the results in a clear way. Easier said than donem because the many combinations will result in big matrices.

I think Sparrow could provide the backend of a smoke tester, but not the interface with the user (no Sparrow files in the module repo, see below). Adding a CI-like configuration interface, will probably result in just-another-CI-platform, competing with what people already use like Travis and Github Actions. Completing on features will be very hard.

Today, the easiest way to test modules with binary dependencies is using docker (e.g. on Travis). A win would be feeding the META6.json to a container and installing what's needed (the translation could be a webservice). Dependencies should be read from META6.json because if even not perfect it's the standard we have. Adding additional files with their own syntax will add to the confusion and people will need to declare everything twice (e.g. for the smoking infra and for Travis).

Furthermore, I think we need to be careful when adding another DSL to the mix. The devops world is dropping configuration management tooling like puppet very fast with its use is reserved to a very small subset of tasks compared to what they where used just a few years ago.

[melezhik]

(no Sparrow files in the module repo, see below).

Yeah. Even though I suggested this as a part of solution, I am perfectly fine with having all the external dependencies setup through META file in pure declarative way. Like I said Sparrow is just a tool it's relatively easy to change the dependency logic implementation from .rakudist/sparrowfile in module source code:

package-install "perl-devel"

to code executed in smoke test (RakuDist/Docker?) backend:

my @ext-libraries  = parse-meta-file($module-name);
for @exit-libraries -> $p {
  package-install $p
}

So people wan't touch Sparrow at all.

Today, the easiest way to test modules with binary dependencies is using docker (e.g. on Travis). A win would be feeding the META6.json to a container and installing what's needed (the translation could be a webservice).

So, are you talking about following?

Travis CI -> docker -> API ( META -> system commands to install deps, os specific )  

Where API is a layer (Sparrow) to translate $module META to system specific commands. This API could be run remotely ... Is it correct?

[nxadm]

I think the easiest system would be as zero conf as possible. If it's easy pluggable to existing and new Smoke/CI solutions, it would be nice. It would be a good incentive for people to start declaring the dependencies in the META6.json.

Travis CI -> docker -> API ( META -> system commands to install deps, os specific )

The problem with binary dependencies is that Linux distributions have different names for the packages (hence a find the package name service or lib) and other system more often than not don't even have native package managers (e.g. dependencies on Windows, or MacOS with a 3rd party solution).

[melezhik]

well, this is what I've already told here - https://github.com/Raku/problem-solving/issues/144 😃 there is no easy way to solve it using META file, and this is why I came up with a idea of "patches" - os specific scenarios in RakuDist/Sparrow:

https://github.com/melezhik/RakuDist/tree/master/modules/default/files/patches

It's not that I'm saying we have to switch to Sparrow scenarios instead of Meta, but at least this stuff could be useful on backend when translating META to os specific commands ( Linux specific binary dependencies ). And this is where imho Sparrow is really good at.

with a 3rd party solution

See how Sparrow deals with install Cmark from a source ... Just an idea.

https://github.com/melezhik/RakuDist/blob/master/modules/default-github/files/patches/khalidelboray/raku-cmark/sparrowfile

[melezhik]

We could define a high level "meta" packages for external libraries within META6.json, so that Sparrow API could translate those ones into OS specific packages and (maybe) other configuration primitives ( like adding "epel-release" repo for CentOS , adjusting configuration files, and so on ).

Say, we have an OpenSSL Raku module depending on "ssl" package. So dev would define:

"depends" : {
        "build": {
        "runtime": {
            "requires": [
                "meta:ssl"
            ]
        },

Then Travis CI triggers a new build on docker (whatever OS), the build script will call Sparrow API that would do all the underlying job.

So we separate in a clean way external package dependency abstraction from implementation ...

The only challenge here is that Sparrow would have to support all required meta packages abstractions ("ssl", "perl", "python", whatever ) - for Raku modules - but I don't think those are going to be too many and they won't come all at once, as people would add new bindings gradually and we would have enough time to bring a proper support to Sparrow layer ...

[AlexDaniel]

It'd help if we looked at what already exists before reinventing solutions.

• https://github.com/niner/Inline-Perl5/blob/f4473114a296a7ab61e6cfb7e1fe540fe931ec6b/META6.json#L12-L16
• https://github.com/niner/Inline-Python/blob/53a1c2ca1751b5928909d929493e5ed29a721d02/META6.json#L14-L29

[melezhik]

@AlexDaniel I see that. However the idea is to avoid having implemenation details in META6 file. If we have 2 layers ( interface and os specific implimentation ) - it is much easier to maintain. Try to implement for example CMark dependency forCMark module - I pesronaly would not want such a complexity inside META6 file which only needs to declare abstract dependency interface ...

[AlexDaniel]

Try to implement for example CMark dependency for CMark module - I pesronaly would not want such a complexity inside META6 file which only needs to declare abstract dependency interface ...

I don't see the difficulty, just install the required package. If you really want to compile it yourself then you can do it too, but why? Bad example. And it is that easy for most of the modules.

If we have 2 layers ( interface and os specific implimentation ) - it is much easier to maintain.

Let's try solving problems instead of creating new ones. Another layer and needless separation is definitely not going to help our case.

[ugexe]

We’ve settled on a declarative system. Anything that is turing complete and isn’t declarative isn’t something to be specced by a language. Neutering our exiting implementation (which has worked for 2 years) to just have this idea of meta packages that can’t be resolved before downloading code is not the type of system we’ve strived for. After all if someone wanted to use a procedural make-like to handle all their dependencies (instead of the native language packages manager and potentially the language itself) then they can simply Just Do It — no need for speccing anything.

[melezhik]

@ugexe honestly I don't get you. The declarative system you've mentioned works well for Raku dependencies ( as it does for any other exiting langauges), but honestly practically ( few Inline-*modules are not representitive ) it does not work well to handle a complexity and nature of external dependencies. It's naive to think it'd handle all the scenarios (I've already mentioned a few). I don't belive people will try to use zef to install external dependencies that way, neither they do now. Also I don't try to change a decalarative style of META6, what I am trying to do is to separate concerns where it is appropriate and where it could produce better design.

[AlexDaniel]

If anyone is looking for something actionable in this thread, here is my personal opinion: just go and start submitting PRs with META6.json changes to modules that use non-raku dependencies. Here is a good start: https://github.com/Raku/ecosystem-unbitrot/wiki. Everything else is just talk.

[ugexe]

The following modules currently use it in some form:

Termbox
Inline::Perl5
Inline::Python
HTTP::Tinyish
DB::SQLite
GPGME
Audio::OggVorbis
LibCurl
SAT::Solver::MiniSAT

and no, its not meant to allow zef to install native dependencies (although doing so is fairly straight forward). It fixes the native dependency problem for the common case by simply giving a missing dependency error that the developer can act on, all without downloading or running any code.

[melezhik]

I don't see the difficulty, just install the required package. If you really want to compile it yourself then you can do it too, but why? Bad example. And it is that easy for most of the modules.

Ok, it's too early to jump to conclusions, before you try to solve the same task through META6 and then let's compare solutions. And yeah, you simetimes need to compile from source for some dependencies because there is no native packages for them or distibution versions are too old. Of couse it's just an example, I don't agree it's bad because it illustrates potential scenarios ( even though, yeah there is cmark binary package, but agian it's just an example and btw module author chose to source install - https://github.com/khalidelboray/raku-cmark#install , why not ? )

Let's try solving problems instead of creating new ones. Another layer and needless separation is definitely not going to help our case.

There is no current solution, or at least it has (imho) drawbacks even with indenefying required dependencies. I believe I've already explained why on my view separation is needed.

[melezhik]

The following modules currently use it in some form:

Ok, lets work from here. Let me pick the first one.

Termbox - it misses dependencies for other OS ( only CentOS is included ), it misses gcc. Why is that? Not just because an author does not care:

and no, its not meant to allow zef to install native dependencies

This is a clever bit, you never know what dependencies a module need until you try install it, for almost 2 month of RakuDist I've faced with a lot of external dependencies issues installing various modules, most of them are not declared in META6 and could not be easily implemened through META6 declarative way.

Let user only declare that module has A dependency let another layer resolve THE dependency on THAT linux and make a neat report of it. That is it.

[melezhik]

It fixes the native dependency problem for the common case by simply giving a missing dependency error that the developer can act on, all without downloading or running any code.

it does not, because it is imcomplete, it lacks all the information, it only checks something which an author chose to declare ( or does not check if an author chose not to ) and what is the most important it does not try to install a dependency, thus that information is never accurate or practically usefull.

only real CI installation would give that information, you may browse through patches directories I've already mentioned and you'll find a lot of interesting details.

[JJ]

El mar., 16 jun. 2020 a las 4:11, Alexey Melezhik (notifications@github.com) escribió:

It fixes the native dependency problem for the common case by simply giving a missing dependency error that the developer can act on, all without downloading or running any code.

it does not, because it is imcomplete, it lacks all the information, and what is the most important it does not try to install a dependcy, thus that information is never accurate or practically usefull.

It's as accurate as the author wants, it's práctical because it forces you to install missing dependencies in the best possible way. It's also practical in some cases where we could try and pick it up and try and install dependencies before running tests, which is what brought us here. I really don't think there can be any other way of doing that.

[patrickbkr]

Outside of the context of a CI / testing tool I can imagine an interactive, end user tool that helps with installing native dependencies. It should work on a best effort basis, not giving any guarantee of succeeding. If it can not automatically install a dependency it might also just print instructions on what the user needs to do. "Did not find a way to install the caca library. Try to install that library yourself."

On most distributions there are mechanisms that allow locating a package providing a file. So writing a set of distro specific routines to locate matching packages for a needed library / binary is doable. On Windows such a tool could at least try to follow the installation hints a META6.json can give.

I'll put this on my todo list. (Not at the top though.) So if noone beats me to it I might start working on this in a few weeks.

[JJ]

As a matter of fact, there are solutions that don't need yet another language layer: there's guile, and I seem to remember @chloekek was working on something similar to that, using guix We don't really need to reinvent the wheel... What we need is a meta-tool that can defer to... something for external dependencies (guix, Sparrow) and to zef (or whatever tool is used in the future) for Raku deps. It's very likely that we need something additional to META6.json, but, in the case of external dependencies, it can also defer to some external script to install dependencies. In the best case we can maybe just express these dependencies declaratively, in some other cases (for instance, we need to copy libcaca.so by hand to /usr/lib) we will need something additional to that.

[patrickbkr]

@JJ Can you elaborate on the existing solutions? Guile is a programming language and guix is a linux distro. So I don't see how they relate to a generic, external-dependency install-tool.

Note that I am thinking about an end user tool outside of the context of CI / testing. So I imagine a tool that helps the average user to install external deps on her own machine, what ever distro / OS she has installed, not in a container or controlled testing environment.

[JJ]

guix is also a way of declaring dependencies, that uses guile (a lisp dialect) as a language. Check this: https://fosdem.org/2020/schedule/event/guix/ and this: https://fosdem.org/2020/schedule/event/gnuguixpackagemanager/ There was as a matter of fact a guix charm offensive in the last FOSDEM, there are other two talks about it. I first heard about it from @chloekek, as a matter of fact.

[zopsicle]

I worked on Nix infrastructure for Raku, not Guix. Guix is very similar to Nix but I’ve never used it. Note that Nix doesn’t work on Windows.

[JJ]

Well, I wouldn't know about the concept if you hadn't mentioned Nix. I don't know about Guix in Windows; maybe in that case Sparrowdo would be the platform-independent alternative.

[melezhik]

@JJ @patrickbkr these are all valid points, though I could hold different opinions on certain bits. It seems to me (I've already started it from the beginning) we keep talking about 2 different use cases:

• CI/Testing tool, not involving an end user
• End user tool (based on top of zef) to ensure external libraries are installed as a part of a module installation process. Involve an end user.

Let's choose the one we want to focus on and start from here.

[JJ]

Well, the first could use the second, why not.

[melezhik]

Well, the first could use the second, why not.

How do you see this? I guess @patrickbkr has some ideas?

[melezhik]

I believe we've made some good progress with examining of the current state of META/native dependencies approach together with @niner and @ugexe , hopefully I could show some prototype using this approach within few weeks, if I have enough time. However I still want to stick to Sparrow 😄 at least at some bits of implementation ...

[melezhik]

I've added support to RakuDist to install native dependencies defined in META.

Examples:

GPGME - http://rakudist.raku.org/sparky/report/centos/420 LibCurl - http://rakudist.raku.org/sparky/report/centos/410 DB::SQLite - http://rakudist.raku.org/sparky/report/centos/422

The support is limited for CentOS and only handles :from<native> statements, but you could see an idea.

Some PRs to amend META have been created:

• https://github.com/CurtTilmes/raku-gpgme/pull/1
• https://github.com/CurtTilmes/raku-libcurl/pull/15
• https://github.com/CurtTilmes/raku-dbsqlite/pull/10

[melezhik]

if someone is interested in implementation, it is here - https://github.com/melezhik/RakuDist/blob/3b388ae5f78e4972463d61a0ca2fb6fc17f2731d/modules/default/sparrowfile#L72-L81 and based on raku-native-deps Sparrow plugin that does all parsing and converting META spec statements into native packages names