thaljef / Pinto

Curate your own repository of Perl modules
https://metacpan.org/module/Pinto::Manual
66 stars 49 forks source link

Unable to replicate module list on newer Pinto #225

Closed realyst2k closed 8 years ago

realyst2k commented 8 years ago

I'm attempting to upgrade our local Pinto repository from a very old version (0.054: old enough that migrate won't work) to a more recent one(0.11). However, in attempting to replicate the modules from the old one into the new one and am running into issues, specifically where Pinto is seeing a conflict in MakeUtils between various IO::Compress libs and is subsequently removing one package and replacing with another, in the process removing some of the modules the previous one included and the new one doesn't.

I'm aware much of the IO::Compress tools are in fact core now, but this is for an older version of Perl with some proprietary tweaks, so I need to include them manually. I'm using the old pinto server as a pull source.

An example of this would be the following:

$ pinto pull IO::Compress::Base -M --stack=test-build $ pinto list --stack=test-build [rl-] Compress::Raw::Bzip2 2.060 PMQS/Compress-Raw-Bzip2-2.060.tar.gz [rl-] Compress::Raw::Zlib 2.060 PMQS/Compress-Raw-Zlib-2.060.tar.gz [rf-] File::GlobMapper 1.000 PMQS/IO-Compress-Base-2.015.tar.gz [rf-] IO::Compress::Base 2.015 PMQS/IO-Compress-Base-2.015.tar.gz [rf-] IO::Compress::Base::Common 2.015 PMQS/IO-Compress-Base-2.015.tar.gz [rf-] IO::Uncompress::AnyUncompress 2.015 PMQS/IO-Compress-Base-2.015.tar.gz [rf-] IO::Uncompress::Base 2.015 PMQS/IO-Compress-Base-2.015.tar.gz [rf-] MakeUtil 0 PMQS/IO-Compress-Base-2.015.tar.gz

$ pinto pull IO::Compress::Zlib::Extra --stack=test-build

the commit message shows nothing is really changing here

$ pinto list --stack=test-build [rl-] Compress::Raw::Bzip2 2.060 PMQS/Compress-Raw-Bzip2-2.060.tar.gz [rl-] Compress::Raw::Zlib 2.060 PMQS/Compress-Raw-Zlib-2.060.tar.gz [rf-] File::GlobMapper 1.000 PMQS/IO-Compress-Base-2.015.tar.gz [rf-] IO::Compress::Base 2.015 PMQS/IO-Compress-Base-2.015.tar.gz [rf-] IO::Compress::Base::Common 2.015 PMQS/IO-Compress-Base-2.015.tar.gz [rf-] IO::Uncompress::AnyUncompress 2.015 PMQS/IO-Compress-Base-2.015.tar.gz [rf-] IO::Uncompress::Base 2.015 PMQS/IO-Compress-Base-2.015.tar.gz [rf-] MakeUtil 0 PMQS/IO-Compress-Base-2.015.tar.gz

so I add the package into the repository manually and try to register it into the stack

$ pinto add /tmp/repository/.pinto/authors/id/P/PM/PMQS/IO-Compress-Zlib-2.015.tar.gz --no-fail --norecurse --stack=test-build

It wants to commit these changes to the stack, which erases IO-Compress-Base

-[rf-] PMQS/IO-Compress-Base-2.015.tar.gz +[rl-] PMQS/IO-Compress-Zlib-2.015.tar.gz Downgrading package PMQS/IO-Compress-Base-2.015/MakeUtil~0 to PMQS/IO-Compress-Zlib-2.015/MakeUtil~0 on stack test-build

$ pinto list --stack=test-build [rl-] IO::Compress::Adapter::Deflate 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Compress::Adapter::Identity 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Compress::Deflate 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Compress::Gzip 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Compress::Gzip::Constants 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Compress::RawDeflate 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Compress::Zip 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Compress::Zip::Constants 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Compress::Zlib::Constants 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Compress::Zlib::Extra 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Uncompress::Adapter::Identity 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Uncompress::Adapter::Inflate 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Uncompress::AnyInflate 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Uncompress::Gunzip 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Uncompress::Inflate 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Uncompress::RawInflate 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz [rl-] IO::Uncompress::Unzip 2.015 PMQS/IO-Compress-Zlib-2.015.tar.gz`

Now IO::Compress::Base is no longer in the list, but the ZLib stuff is there. And if I pull in the Bzip stuff, everything else gets wiped:

$ pinto list --stack=test-build [rl-] Compress::Raw::Bzip2 2.060 PMQS/Compress-Raw-Bzip2-2.060.tar.gz [rl-] Compress::Raw::Zlib 2.060 PMQS/Compress-Raw-Zlib-2.060.tar.gz [rl-] IO::Compress::Adapter::Bzip2 2.015 PMQS/IO-Compress-Bzip2-2.015.tar.gz [rl-] IO::Compress::Bzip2 2.015 PMQS/IO-Compress-Bzip2-2.015.tar.gz [rl-] IO::Uncompress::Adapter::Bunzip2 2.015 PMQS/IO-Compress-Bzip2-2.015.tar.gz [rl-] IO::Uncompress::Bunzip2 2.015 PMQS/IO-Compress-Bzip2-2.015.tar.gz [rl-] MakeUtil 0 PMQS/IO-Compress-Bzip2-2.015.tar.gz

Attempting to pin a package runs into this issue:

$ pinto add /tmp/repository/.pinto/authors/id/P/PM/PMQS/IO-Compress-Zlib-2.015.tar.gz --no-fail --norecurse --stack=test-build /tmp/repository/.pinto/authors/id/P/PM/PMQS/IO-Compress-Zlib-2.015.tar.gz is the same as PMQS/IO-Compress-Zlib-2.015.tar.gz -- using PMQS/IO-Compress-Zlib-2.015.tar.gz instead Unable to register distribution PMQS/IO-Compress-Zlib-2.015.tar.gz: MakeUtil is pinned to PMQS/IO-Compress-Bzip2-2.015/MakeUtil~0

Trace begun at /home/pinto2/pinto-server/lib/perl5/Pinto/Util.pm line 88 Pinto::Util::throw('Unable to register distribution PMQS/IO-Compress-Zlib-2.015.tar.gz: MakeUtil is pinned to PMQS/IO-Compress-Bzip2-2.015/MakeUtil~0') called at /home/pinto2/pinto-server/lib/perl5/Pinto/Schema/Result/Distribution.pm line 276

And doing it in reverse does the same, in reverse.

Is there a way to shoehorn this into place? This exact configuration of modules were present in our old pinto repository and is what we draw on to create our install packages.

Any help is much appreciated. Cheers, Réal

thaljef commented 8 years ago

It sounds like the problem is Pinto sees a MakeUtil package in both distributions. But the default behavior of Pinto is to evict all the packages in an incumbent distribution if an incoming distribution contains any (but not necessarily all) of those same packages. This ensures that the stack will provide you with exactly one version of Foo::Bar, no matter what you asked to install. This wasn't always the default, so your old repo may have worked differently.

However, that behavior isn't always desirable. Sometimes, you need to allow the same package to be in two distributions at the same time. I think the most common case for this are internal utility packages. For example, a helper package that is used during testing or building, but not actually installed. Most of the time, these packages are probably not supposed to be indexed, but the author may have forgotten to exclude them in the metadata, or Pinto may be indexing incorrectly.

I can think of two possible solutions. The first is to use the --no-index option with the add command. This allows you to tell Pinto to ignore certain packages when indexing the distribution. Unfortunately, you can't do this after-the-fact, so you'll either have to start again with a clean repo, or use the delete command to remove one of the offending distributions. So I think the process would look something like this:

$>  pinto pull --stack=test-build -M IO::Compress::Base
$> wget http://www.cpan.org/authors/id/P/PM/PMQS/IO-Compress-Zlib-2.015.tar.gz
$> pinto add --stack=test-build -M --author=PMQS --no-index=MakeUtil ./IO-Compress-Zlib-2.015.tar.gz

The other solution is to change the Pinto configuration to allow it to keep multiple distributions in the index, even if they share one or more package names. This makes Pinto behave more like PAUSE does. The index will still only contain one entry for MakeUtil (the last one wins), but it won't kick out the entire distribution when it sees another MakeUtil package. The setting for that is called intermingle and the config file is at .pinto/config/pinto.ini within the repository. Uncomment the line and set the value to 1, like this:

; Allow stacks to intermingle distributions
; Defaults to 0
intermingle = 1

Once that is changed, use the pull command on both distributions and it should do the right thing. You don't need to start over with a clean repository for this one, but you can never change it back to intermingle = 0 and this option affects the entire repository (not just one stack). Depending on your world view and level of paranoia, this might be the right setting for you anyway. And it would probably match the behavior of your old repo better. But make sure you understand it before going down this road.

Let me know if that helps.

realyst2k commented 8 years ago

Thanks! That's exactly the behaviour I needed. Retrying the migration, so far it's definitely matching the old one much better.

Once we update our Perl versions, I'll see if I can get away with a intermingle-less instance.