Improve how Wiki Education Dashboard counts references added

[gabina]

This issue is the main issue for the "Improve how Wiki Education Dashboard counts references added" Outreachy (round 27) project.

Timeline project can be found here.

Current Behavior:

The Dashboard currently gets statistics on references added by fetching the features data supplied by Wikimedia's article quality machine learning models, and comparing the values of one revision with values for the previous revision. This reference counting method is constrained by the availability of article quality machine learning models, which are not accessible for the majority of Wikipedia language editions, including the Spanish language version. That means that not every language has references added metric available.

Desired Behavior:

References added metric should be available for every language version of Wikipedia. One high priority is to enable reference counting for Spanish Wikipedia, in support of the Spanish #1Lib1Ref campaign.

Main tasks:

Note: the following tasks are high-level tasks just to keep a general idea of the project progress. More details can be added on the fly.

• [x] Research the current method for calculating references added.
• [x] Evaluate which API we will use to get the references added number for all the languages (misalignment.wmcloud.org/api/v1/quality-revid-features API is one option, building our own API for this particular case is another one).
• [x] Build our own API to get references and host it in Toolforge (as a conclusion of the previous step).
• [x] Determine where to store data from the new API. Handle database migration if necessary. Note: pull request was closed because we decided to use the existing features and features_previous fields. (PR #5567)
• [x] Develop a streamlined method for making requests to the new API (similar to LiftWingApi class). (PR #5565)
• [x] Implement a way to use the newly created class to import and store data in the database (similar to RevisionScoreImporter).For this, we did the following things:
  • Refactor the existing RevisionScoreImporter (PR #5578).
  • Create a new intermediate class to handle API requests (PR #5594).
  • Update the Revision::references_count method to use the new data populated in features (or features_previous) from the reference-counter API (PR #5615)
• [x] Make a performance report for the new score import process.
• [x] Make a quality report for the new score import process.
• [x] Integrate all changes.
• [x] Clean up code (PR #5675).

[akshayw1]

Can I work on this issue?

[gabina]

@akshayw1 this is the issue for an Outreachy project. I created it because I'm currently working on the project.

There are a lot of other issues you can work on. I suggest you to use labels (such as newcomer friendly or react) to filter issues you're interested in.