Ecosystem: should we strive for integration with MetaCPAN

[lizmat]

This came up in a discussion between @niner and me at the PTS: @niner thinks it would be a unique selling point for MetaCPAN and Perl in general (both 5 and 6) if Perl 6 modules could be found on MetaCPAN with Inline::Perl6 as a dependency when searching for Perl 5 modules, and Perl 5 modules could be listed with an Inline::Perl5 dependency when searching for Perl 6 modules.

Before taking this to the MetaCPAN team, I think the Perl 6 leadership will have to make up its mind about it.

[AlexDaniel]

(not sure which label that would be with the current system)

Any reason why we can't just go and implement Inline::Perl5 part of it? The general decision doesn't have to be perl5-specific, we can do the same with python and other modules…

[AlexDaniel]

Ah, I'm misunderstanding a bit. I guess the proposal is to make that work on MetaCPAN, and not in the module management tools like zef.

I was thinking more in terms of making zef install DBI:from<Perl5> or zef install matplotlib::pyplot:from<Python> DWIM, but I'm not sure if there's a good way to define the expected behavior of that…

Anyway, ping @ugexe

[ugexe]

I was thinking more in terms of making zef install DBI:from or zef install matplotlib::pyplot:from DWIM, but I'm not sure if there's a good way to define the expected behavior of that…

That is the eventual plan (although mostly for putting those strings inside depends). But it will be a difficult problem solving how to handle the various options that fit Perl6 (installation locations, graph conflicts, etc).

[ugexe]

Also for MetaCPAN to consider this I would expect we need to first implement an actual index/PAUSE. Otherwise they will have to redesign a bunch of things to work with e.g. multiple auths.

[lizmat]

@ugexe: "an actual index/PAUSE" You mean a community-approved recommendation manager that only allows one auth / one api per namespace ?

[ugexe]

Community-approved is irrelevant unless it is MetaCPAN community's approval -- why would they be interested in incorporating an entirely new paradigm / api? Realistically we will need to use PAUSE (+ CPAN).

https://github.com/andk/pause/blob/master/doc/operating-model.md

[lizmat]

On 12 May 2019, at 23:10, Nick Logan notifications@github.com wrote: Community-approved is irrelevant unless it is MetaCPAN community's approval -- why would they be interested in incorporating an entirely new paradigm / api? Realistically we will need to use PAUSE (+ CPAN).

I mentioned “community approved”, because that is one of the current problems of namespace assignment on CPAN: first come, first served. This gives problems when an author decides to become incompatible with a module that is way up river (case in point: JSON::XS). Because CPAN has the “first come, first served”, it had no option to put in another, compatible version in its place.

With a “community approved” recommendation manager, it would have been possible to replace a rogue module by one that has not gone rogue.

And since there can always be a debate on what is ok and what not, did I come up with the more general concept of “recommendation manager”: if some people think such a change was unwarranted, they can set up their own recommendation manager and follow the recommendations of that one.

[patrickbkr]

I'm not sure if it's beyond the scope of this ticket, but I'm mentioning it anyways: Getting the MetaCPAN website (not only API) display Perl 6 modules might also be a thing to consider when contacting the MetaCPAN team. Integration would provide the Perl 6 community with a grown up online module browser and prevent the double maintainance that comes with a fork or separate website. JDV already started the work to get MetaCPAN support Perl6, but the work stalled.

[lizmat]

My point for this issue, was to get consensus on what we want as the Perl 6 community, before we even start thinking about talking to the MetaCPAN people.

[ugexe]

This gives problems when an author decides to become incompatible with a module that is way up river (case in point: JSON::XS). Because CPAN has the “first come, first served”, it had no option to put in another, compatible version in its place.

Authors who value stability over flexibility may do e.g. use Foo:auth<...> and "depends" : ["Foo:auth<...>"]. It really doesn't matter if a specific recommendation manager doesn't allow otherwise identical namespaces, because a consuming client like zef is free to have a recommendation manager that makes a recommendation based on the recommendations from multiple recommendation managers.

If an author really wants to be shown on MetaCPAN then they should use a namespace that fits their permission model. Developers who are installing a distribution are free to ignore e.g. p6c by removing it from zef config (permanent) or passing --/p6c (temporarily). As an author if I really want to influence the recommendation process I could theoretically just use emulates.

With a “community approved” recommendation manager, it would have been possible to replace a rogue module by one that has not gone rogue.

By default zef doesn't prefer CPAN over p6c except when they have the same highest version/api -- if p6c has a higher version then it will get chosen, and the local cache (another recommendation manager) takes the highest priority of all. In production when deploying you probably wouldn't want to use p6c, and instead would have a local DarkPAN (which can be done entirely in zef config file if the DarkPAN just serves a projects.json) where vetted dependencies from p6c would go first.

---

We already have the "anything goes" p6c ecosystem (and the current iteration of our CPAN ecosystem). What we don't have is a strict ecosystem, so I don't see a single owner of a namespace as a drawback in regards to MetaCPAN.

[ugexe]

re: community approved ecosystem

While I've already mentioned this doesn't make much sense for the specific case of MetaCPAN, it could make sense as a separate service that aggregates from MetaCPAN and others (like zef does locally) applying whatever policy the community thinks works best.

[patrickbkr]

I'm not sure plainly listing Perl 6 packages with the added dependency for Perl 5 package searches and vice versa is a good idea. There are tradeoffs when using packages in the other language.

One has to have a Perl of the other kind installed. Not all Perl 5 installations are suitable as a specific set of compile flags are required. One has to account for the additional dependency of an entirely different language and ecosystem. This is a deal breaker in many cases. E.g. when developing a library that's meant to be published to CPAN. Or when the decision whether to add a new language to the stack ends up as "no".

The consequences the use of Inline::* brings with it don't match the effortlessness that is suggested when presenting Perl 6 packages when doing a search for Perl 5 ones. I do think it's a good idea to give the possibilities of Inline::* more prominence though.

Adding support for browsing Perl 6 packages in MetaCPAN web and a banner saying "Have you thought about using Inline::Perl6? Click [here] to search for Perl 6 packages!" might be a better fit.

[niner]

Lots of Perl 5 and Perl 6 distributions have native dependencies. In some cases even those native libraries have to be compiled in a certain way to be usable. Nevertheless we happily list those packages on metacpan and on modules.perl6.org. Why would we treat packages with a transitive dependency on libperl.so differently?

[patrickbkr]

I think what triggers my feeling of this being off are my expectations. I am not surprised to be required to have the mysql client library installed when installing DBD::mysql. Neither am I surprised to need libyaml installed when installing YAML::XS. These are dependencies that are related to the task I want to achieve. Also in most cases there is no real choice, as the native dependency is usually directly related to the modules purpose. Manager: "Why did you add that native dependency?" Me: "Obviously it is necessary to install that MySQL client lib to connect to a MySQL database."

On the other hand having to install Perl 5 when installing Geo::Coder::Google is a surprise to me.

[niner]

In other words: you are simply not used to it. Which is usually true for all changes and something that will pass with time.

You also used the simplest possible examples. Usually there's not just direct dependencies, but dependency trees. And thanks to those, I have been surprised quite often by what I need to install to get some completely different code up and running. Would you have thought that you need Math::Complex to install a web framework like Catalyst?

[patrickbkr]

In other words: you are simply not used to it. Which is usually true for all changes and something that will pass with time.

You might be right about that.

You also used the simplest possible examples. Usually there's not just direct dependencies, but dependency trees. And thanks to those, I have been surprised quite often by what I need to install to get some completely different code up and running. Would you have thought that you need Math::Complex to install a web framework like Catalyst?

There is a difference with the Math::Complex example. In this case I don't need to install software external to CPAN. Catalyst pulls in all the stuff it needs automatically.

I think it never happened to me that a package I tried to install failed to install/run because some external dependency I didn't see coming was needed.

A little sticker on each package that directly or indirectly depends on stuff external to CPAN might help. I guess implementing something like that is difficult though, as there is no reliable way to determine such dependencies.

Anyways, this discussion is mostly about my gut feeling and, as @niner already said, may just be me not being used to stuff working differently.

[Grinnz]

I was asked to comment here, so as I said on IRC, it's important to remember that the MetaCPAN indexes and API are used for more than just website display. Many CPAN-related tools and infrastructure rely on its accurate metadata. Static dependencies are specified by individual distributions and not modified by MetaCPAN when presented via the API. As such introducing any fake dependencies at that level is, in my opinion, a non-starter.

I don't know all the details of the P6 CPAN distribution format, but from what I understand the requirements regarding permissions and structure are very different. I don't think it's worthwhile to attempt to mold the existing indexes and structure and all of their historical baggage to fit these requirements, where a tailor-made structure would better allow searching and display to suit the needs of the P6 ecosystem. Search is hard, and it's harder when you have more than one set of requirements. Of course that raises the question of who will design, implement and maintain that system.

The question of integrations at install time is of course up to the package managers, and not related to MetaCPAN. I think this would be a good level for these integrations, as it can operate on logic general to the entire language rather than introducing logic to individual distributions that only applies under certain circumstances. As for the web search, I think cross-language integrations that similarly don't involve distribution metadata would be more reasonable, technically speaking (I can't promise anything politically).

[ugexe]

No fake dependencies are required.

There is no blessed permission model -- all ecosystems can implement whatever permissions they wish. Structure can be converted to CPAN::Meta; indeed zef used to consume a prototype metacpan that returned Perl6 modules in CPAN::Meta format.

Having worked with both data structures I can say converting META6 to CPAN::Meta is not that difficult, and certainly is easier than designing/implementing/maintaining a custom solution. For instance: PAUSE already modifies meta data for perl6 modules and (setting the source-url), so it could just as easily convert to CPAN::Meta format.