Right now, to update Gitpan you iterate through all 35,000 distributions again. This is, obviously, a bit wasteful. It would be better to update releases after a date.
Write a timestamp in the gitpan directory when a full run or update run completes.
Use that timestamp to select new releases.
Create Gitpan::Dist objects, pre-populated with the selected releases.
Run Gitpan::Dist->import_releases as normal.
The timestamp is the newest release in the BackPAN::Index at the start of the run. It's better to have a bit of an overlap than to miss some.
Because the BackPAN index creator was still broken, the latest full import will be since 2014-10-28T22:08:49 aka 1414534129. Keep that in mind when you run the update.
Right now, to update Gitpan you iterate through all 35,000 distributions again. This is, obviously, a bit wasteful. It would be better to update releases after a date.
Gitpan::Dist->import_releases
as normal.The timestamp is the newest release in the BackPAN::Index at the start of the run. It's better to have a bit of an overlap than to miss some.