clone-modules: gets confused when CPAN distribution is named differently from module

[jkeenan]

perldoc clone-modules appears to assemble a list of modules installed against one perlbrew-installed perl before trying to install them against a different perlbrew-installed perl. However, when it goes out to CPAN to look for the latest versions, it is searching for distributions. It fails when a CPAN distribution contains no *.pm file corresponding to the name of the distribution.

Excerpts from the output of perlbrew clone-modules 5.32.0 5.34.0 attempted today:

! Finding Role::Identifiable on cpanmetadb failed.
! Finding Role::Identifiable () on mirror http://www.cpan.org failed.
! Couldn't find module or a distribution Role::Identifiable

There's a Role-Identifiable distribution, but no Role::Identifiable module. (See https://metacpan.org/pod/Role::Identifiable::HasTags)

...
! Finding TAP::Harness::Multiple on cpanmetadb failed.
! Finding TAP::Harness::Multiple () on mirror http://www.cpan.org failed.
! Couldn't find module or a distribution TAP::Harness::Multiple

There is a TAP-Harness-Multiple distribution, but inside it the modules are TAP::Harness::Archive::MultipleHarnesses and TAP::Harness::ReportByDescription.

...
! Finding libwww::perl on cpanmetadb failed.
! Finding libwww::perl () on mirror http://www.cpan.org failed.
! Couldn't find module or a distribution libwww::perl

The distribution is libwww-perl, but no libwww::perl module. The most prominent module in the distribution is LWP.

I believe there are ways to map modules to distributions and vice versa. Their use should be explored here.

[gugod]

Both clone-modules and list-modules command are just using ExtUtils::Install, which is looking up information based on the exstence of .packlist files that does not presever the original distribiton name.

We can probably look up the "main module name" from install.json instead.

[gugod]

Looks like @skaji has already did some work in this part: https://metacpan.org/pod/Distribution::Metadata

[gugod]

This will be fun... the "receipt" of LWP looks is this:

# cat  darwin-2level/.meta/libwww-perl-6.52/install.json | jq
{
  "provides": {
    "LWP::Protocol::nogo": {
      "version": "6.52",
      "file": "lib/LWP/Protocol/nogo.pm"
    },
    "LWP::Protocol::data": {
      "file": "lib/LWP/Protocol/data.pm",
      "version": "6.52"
    },
    "LWP::Protocol": {
      "file": "lib/LWP/Protocol.pm",
      "version": "6.52"
    },
    "LWP::RobotUA": {
      "version": "6.52",
      "file": "lib/LWP/RobotUA.pm"
    },
    "LWP::DebugFile": {
      "version": "6.52",
      "file": "lib/LWP/DebugFile.pm"
    },
    "LWP::Protocol::loopback": {
      "file": "lib/LWP/Protocol/loopback.pm",
      "version": "6.52"
    },
    "LWP::Debug::TraceHTTP": {
      "file": "lib/LWP/Debug/TraceHTTP.pm",
      "version": "6.52"
    },
    "LWP::ConnCache": {
      "version": "6.52",
      "file": "lib/LWP/ConnCache.pm"
    },
    "LWP::UserAgent": {
      "version": "6.52",
      "file": "lib/LWP/UserAgent.pm"
    },
    "LWP::Protocol::cpan": {
      "version": "6.52",
      "file": "lib/LWP/Protocol/cpan.pm"
    },
    "LWP::Protocol::mailto": {
      "version": "6.52",
      "file": "lib/LWP/Protocol/mailto.pm"
    },
    "LWP::Protocol::file": {
      "version": "6.52",
      "file": "lib/LWP/Protocol/file.pm"
    },
    "LWP::Authen::Digest": {
      "version": "6.52",
      "file": "lib/LWP/Authen/Digest.pm"
    },
    "LWP::MemberMixin": {
      "version": "6.52",
      "file": "lib/LWP/MemberMixin.pm"
    },
    "LWP::Protocol::nntp": {
      "version": "6.52",
      "file": "lib/LWP/Protocol/nntp.pm"
    },
    "LWP::Protocol::http": {
      "version": "6.52",
      "file": "lib/LWP/Protocol/http.pm"
    },
    "LWP::Simple": {
      "file": "lib/LWP/Simple.pm",
      "version": "6.52"
    },
    "LWP::Debug": {
      "x_deprecated": 1,
      "version": "6.52",
      "file": "lib/LWP/Debug.pm"
    },
    "LWP::Protocol::ftp": {
      "version": "6.52",
      "file": "lib/LWP/Protocol/ftp.pm"
    },
    "LWP::Protocol::gopher": {
      "version": "6.52",
      "file": "lib/LWP/Protocol/gopher.pm"
    },
    "LWP::Authen::Basic": {
      "file": "lib/LWP/Authen/Basic.pm",
      "version": "6.52"
    },
    "LWP": {
      "file": "lib/LWP.pm",
      "version": "6.52"
    },
    "LWP::Authen::Ntlm": {
      "file": "lib/LWP/Authen/Ntlm.pm",
      "version": "6.52"
    }
  },
  "version": "6.52",
  "dist": "libwww-perl-6.52",
  "name": "libwww::perl",
  "target": "O/OA/OALDERS/libwww-perl-6.52.tar.gz",
  "pathname": "O/OA/OALDERS/libwww-perl-6.52.tar.gz"
}

The actual module names are listed under "provides", we could let list-modules print one of them as the representitive, so long as the it can be mapped back to the correct dist by cpan / cpanm.

OTOH, the top-level keys give us this

{
  "provides": {
    ...
  },
  "version": "6.52",
  "dist": "libwww-perl-6.52",
  "name": "libwww::perl",
  "target": "O/OA/OALDERS/libwww-perl-6.52.tar.gz",
  "pathname": "O/OA/OALDERS/libwww-perl-6.52.tar.gz"
}

Which does not mention "LWP" at all.

For the purpose of clone-modules.... I think we could take "pathname" in these files and feed it to cpanm. cpanm recognize this format and runs correctly.

[gugod]

I give this a bit of thoughts and it now seems to me that, for the scope of perlbrew the sensible solution is to make list-modules and clone-modules do some internal mapping so that modules with unconventional names such as libwww::perl are mapped to LWP. Which might be a bit tedious but so long as a good amount of popular "troublemaker" are dealt with, I'm happy.

[karenetheridge]

Why go from module name -> distribution name -> module name at all? We know what modules are installed, so just reinstall those, without converting to dist name.

[gugod]

@karenetheridge Because that is how ExtUtils::Installed reported... currently the list of "modules" obtained by doing this:

perl -MExtUtils::Installed -E 'say for ExtUtils::Installed->modules'

I guess the definiton of "modules" might be different in different places.

[karenetheridge]

Because that is how ExtUtils::Installed reported

Ok so this is an ExtUtils::Installed bug then.

$ perl -MExtUtils::Installed -E 'say for ExtUtils::Installed->modules'
...
libintl-perl
libwww::perl
...

..is clearly wrong.

[Grinnz]

Related: https://github.com/Perl-Toolchain-Gang/ExtUtils-Install/issues/33