search

cpan-testers / cpantesters-backend

Backend data processing for CPAN Testers
Other
0 stars 4 forks source link

Scan local CPAN and BackPAN mirrors to update database #13

Open preaction opened 6 years ago

preaction commented 6 years ago

We need to know what distributions and versions are released on CPAN. We need to know who released them so we can show author dashboards. Since we have a local CPAN mirror, we can just scan the directory and write that data into the database.

This process should read the existing database, read the local CPAN mirror, and insert any new data from the CPAN mirror into the database.

This is a replacement of the current uploads process, located in https://github.com/cpan-testers/cpantesters-backend/tree/master/old/uploads. The bulk of the code is in CPAN::Testers::Data::Uploads.

Note: The current uploads process has two other functions:

  1. generate which deletes the current uploads table and rebuilds it. I do not think this is safe: It would break all the foreign keys in the entire database. Until we can do this safely, we should not be able to do it at all.
  2. reindex, which deletes the ixlatest table and rebuilds it. This is perfectly safe, but I do not think this belongs here. This will get its own ticket and process.
zmughal commented 1 year ago

I'm not sure if this is the right place to note this (posted this previously in the #cpantesters-discuss IRC channel):

Bit of an oddity, I've had my Renard-Curie module up for 2 years without tests. I was playing around with the options at the cpantesters.org website and noticed that for some reason it thinks version 0.005 (the latest) is "BACKPAN ONLY"

It should not be "BACKPAN Only" as far as I am aware.

Link.