Requires Ref::Util::XS, but this is not noted in the META.yml

[kenneth-olwing]

Hi,

As part of an effort to compute dependencies I use CPAN::FindDependencies which primarily uses the yml file to find deps, I came across the fact that Ref::Util::XS is not listed, but when trying to actually install it, it complains. As far as I understand it, apparently Dist::Zilla adds it to Makefile.PL at some point.

While CPAN::FindDependencies can revert to using Makefile.PL as a source it (currently) only does so if there's no META.yml file. Simple fix to force it, but it also cautions of indiscriminately running Makefile.PL from dists...true, that would happen anyway with a 'cpanm Foo::Bar', but still...

As a very limited thought, would it be possible to add it to META.yml proper? Or would it simply be a better idea to only rely on Makefile.PL in all cases...or is there some other, better idea?

Some background on my usecase may forestall/open up for some relevant questions :-) I'm creating an automated build to make our own inhouse Perl dist (for both Windows and Linux), and outfit it with a large number of modules. I'm also behind a firewall, so at build time I can't get out of the house, and therefore I need to precompute dependencies and dependency order, download all relevant dists and place them locally. At build time it should then be possible to just run the installs for all dists in the computed order, and Bob's your uncle. In principle, it works fine, even if I still need to have some odd workarounds - mostly simple ones such that some modules are simply to be skipped for one or the other of the platforms, or that some tests fail, but are benign. But undeclared dependencies are a little bit of a pain (as well as the fact that some modules exhibit cycle dependencies, e.g. Carp => Exporter => Carp...)

Well, any suggestions on how to this better is obviously also appreciated.

Thanks,

ken1

[karenetheridge]

Ref::Util::XS is a dynamic prereq, which is added when running Makefile.PL after interrogating your environment -- that is, if you have a compiler available, and haven't explicitly requested a pure-perl only installation, the XS module is added (to provide a faster implementation). It would not be appropriate to add it to META.*, which contains static prereqs only, as that would block installation of the module on pure-perl only environments (as well as make the module not fat-packable).

All of this is clued by dynamic_config => 1 in the meta files -- indicating that the static config alone is not sufficient to describe the full set of requirements in all instances. There is documentation on this here: https://metacpan.org/pod/CPAN::Meta::Spec#dynamic_config

In your situation I would suggest either explicitly adding Ref::Util::XS to your list of application requirements, or running cpanm with the --pureperl option. You should probably also check the meta flags in all of your other dependencies to see if there is dynamic configuration in any of them -- as they need to be added to your computed list of dependencies. This computation should be done in as similar an environment to your finall runtime installation as possible, so the dynamic calculations come out the same.

[kenneth-olwing]

Ok, great answer. I think that clues me in to the fact that I really can't do a precomputation the way I've been doing. And I want XS modules. So, my revised plan would be something like this:

• I can, from a list of module names that I know I want, precompute which dists I need, using the static deps. This list might be correct, but more likely not quite (as like with Ref::Util::XS). This is fine.
• I download the lot of them
• During the build I create a minicpan with just the dists I have,
• I cycle through the dists and let a 'cpanm --showdeps -M myminicpan' worry about the dependencies
• At this point, I might come up short with things not captured by the static check - so I just download them and manually add them then - rinse and repeat.
• After collecting all deps, I toposort the whole thing and thus know the order of dists to install
• Cycle this list and let cpanm do the actual installs (at this point I occasionally have workarounds to do for specific dists that are known not to pass inconsequential tests, for instance).

Whew. A bit of a drag, but doable, and as automated as possible so I can version control it, but fairly easily repeat it for newer modules and later Perls.

Thanks for your help.

[karenetheridge]

Yes, unfortunately you need to do a full installation to find out the full list of prereqs you need. You might find https://metacpan.org/release/Carton of use though -- you can specify your top level list of application dependencies in the cpanfile file, then carton install will generate a cpanfile.snapshot that contains all of the dependencies. You can then use that file to pull down all the tarballs from a cpan mirror to store locally, and on your production machine you can just carton install --deployment to install from that local store. (Some of the configs are fiddly, but there is good support available.)

[kenneth-olwing]

True - I'll consider this!