Upgrade pinto repository

[miquelruiz]

Hi,

I have installed the last version of pinto, and then copied a repository from an older installation to be used in the new server.

When using 'pinto' command with the copied repository, I get an error:

$ pinto -r /path/to/repo list Your repository is too old for this version of Pinto

Is there any way to upgrade a repo, so it can be used with the new version? The versions I've been working with are the following:

pinto@new-server:~$ pinto version App::Pinto 0.054 Pinto 0.065

pinto@old-server:~$ pinto version App::Pinto 0.053 Pinto 0.060

I'll be glad if you point me to the piece of documentation which shows how to do it, if it exists (I've been unable to find it). If any further information is needed, please request it. Thank you very much in advance.

Regards, Miquel

[thaljef]

On Dec 27, 2012, at 8:32 AM, Miquel Ruiz wrote:

I'll be glad if you point me to the piece of documentation which shows how to do it, if it exists (I've been unable to find it). If any further information is needed, please request it. Thank you very much in advance.

There is a migration process, but it is manual and does not preserve your revision history. The next version of Pinto (0.066) will have a migration tool so that repositories >= 0.066 can be automatically migrated to subsequent versions of Pinto. So if you wait until 0.066 (later this month), then you'll only have to do the manual migration once.

Basically, the manual migration process is something like this (bash):

with old Pinto installed

export PINTO_REPOSITORY_ROOT=path/to/old_repo pinto stacks --format '%k' > stacks.txt for stack in cat stacks.txt; do pinto list --format '%A/%f' $stack | sort | uniq > $stack.txt done;

with new Pinto installed

export PINTO_REPOSITORY_ROOT=path/to/new_repo pinto init --bare --source file:///path/to/old_repo for stack in cat stacks.txt; do pinto new $stack pinto pull --stack $stack --norecurse --message 'Migrated from old repository' < $stack.txt done;

In other words, make text files listing the distributions belonging to each stack in the old repository. Then create a new repository that uses the old one as the --source. Finally, create and populate each stack in the new repository by pulling the distributions named in the corresponding file. When finished, you'll probably want to edit the config of the new repository to point to http://cpan.perl.org

I'll let you know when the next Pinto release is out.

-Jeff

[schwern]

Will the manual process preserve the history and logs?

[thaljef]

Will the manual process preserve the history and logs?

Not the way I described it above. To preserve history, I think you'd have to parse the history (log --detailed) and construct a sequence of commands to feed into the new repository. But that still wouldn't produce an exact replica of history (e.g. timestamps would not match).

I should have worked on the migration tools sooner -- lesson learned. For now, I'm telling people to wait for the next release (with automatic migration to future versions), and use the current Pinto for proof-of-concept work only.

[pplu]

Hi,

Do you have a rough estimate of when 0.066 will be released? Maybe we can pull the migration tool from the trunk to try a migration? Would that help in some way to get the migration tool ready for releasing?

Best Regards,

Jose Luis

[thaljef]

I'll be making a dev release next Tuesday. But I'm afraid it won't solve your migration problem (at least not by itself).

The next release has hooks to automatically migrate new repositories to future versions. But it can't migrate repositories that already exist right now.

To migrate the repository you have now, we'll have to do something manual. How many stacks does your repo have? How many commits?

[schwern]

I'm working for that client again this week who makes heavy use of Pinto. I was going to look into migrating their Pinto database to the new version, hopefully back port the existing migrator to their version, but found that Pinto::Migrator in 0.66 appears to be a stub.

What's the state of the art in Pinto migration?

[thaljef]

Pinto::Migrator will migrate repositories created with 0.066 or later. Prior to that, repositories weren't versioned at all, so I had no real way to migrate them.

In your current repository...

• Do you have multiple stacks that you care about?
• How much do you care about the revision history?
• Do you have any pins in place?

Those questions will dictate how much work a migration will be.

[thaljef]

I know this sucks. I should have addressed this sooner. But bear with me and I'll get you through it.

[schwern]

Don't believe I have any stacks or pins I care about. It's the content and the revision history I care about. I want to make sure the reasons for choosing and patching each distribution are there.

No worries, I knew I was getting in the door early and this is still WAAAY less work than building something myself. :)

[schwern]

I hacked together a variation on your list & pull script so I can preserve why things are in the repository.

I hacked the list action to also spit out the log message so I could supply it to the pull. I did this by adding a new %g format to Registration

     g => sub {
         my @changes = $self->package->registration_changes;
         my $message = $changes[-1]->revision->to_string("%g");
         $message =~ s{\n}{\\n}g;
         return $message;
     },

Then pinto list --format '%A/%f\t%g'. A little script can split that back into revision and message.

use perl5i::2;
use Path::Tiny;

my $list_file = shift;

my $fh = path($list_file)->openr;
while(my $line = <$fh>) {
    chomp $line;
    my($release, $msg) = split /\t/, $line, 2;
    $msg =~ s{\\n}{\n}g;
    my @cmd = ("pinto", "pull", "--no-recurse", "--message", $msg, $release);
    @cmd->join(" ")->say;
    system @cmd;
}

One problem I found is pinto list on a repository which has had versions pulled on top of other revisions will tend to have multiple versions of the same distribution in the index, especially if you're doing a lot of downgrading. This confuses pinto, I got an error from pull like Unable to determine ordering: DROLSKY/DateTime-Locale-0.42/DateTime::Locale::Base~0 <=> DROLSKY/DateTime-Locale-0.45/DateTime::Locale::Base~0.

To solve that I found all the duplicate distributions

perl -nwle '($f)=m{^.*?/(\S+)}; $f=~m{(.*)-[\d\.]}; print $1' stack.txt | sort | uniq -d

And used pinto log -d in the old repository to figure out which version was the correct one.

[thaljef]

I hacked together a variation on your list & pull script so I can preserve why things are in the repository.

That is pretty clever! Much better than what I was thinking of. You'll only get the message for the last revision that touched each package, but that's probably good enough.

multiple versions of the same distribution in the index ... This confuses pinto, I got an error from pull like...

That's not supposed to ever happen. If you construct a reproducible test case for me (e.g. just a series of pinto commands) then I'll figure out what is going on.

[thaljef]

One other suggestion is to stash a listing from your old repository in a text file, and then diff(1) it with one from your new repository. That way you'll know when they match (and you're done).

The default format for the listing changed slightly in 0.066. So you'll want to use the --format option to produce comparable listings from both repositories.

[schwern]

I think I only care about why the latest revision is there. I also have a git repository for all my patched releases, much like patchpan, and that contains an even more concise history. I'm just glad I can upgrade and preserve history.

I posted the multiple versions problem in #48.