Integrate WP1 selection tools

openzim / wp1

Wikipedia 1.0 engine & selection tools

https://wp1.openzim.org

GNU General Public License v2.0

24 stars 17 forks source link

Integrate WP1 selection tools #171

Open kelson42 opened 4 years ago

kelson42 commented 4 years ago

Available at https://github.com/openzim/wp1_selection_tools Using the Python framework.

kelson42 commented 4 years ago

@audiodude One of a first step would be maybe to integrate its source with the same repository and put it in the docker-compose.yml ?

audiodude commented 3 months ago

We'd like to start this task in earnest.

[x] Design doc written (link redacted)
[x] We have requested a Trove database in Wikimedia Cloud VPS, to allow for the growth of our article tables
[x] Some foundational work has already been explored with getting the pageviews database ingested into the WP1 db

Data ingestion of views, links, and lang links can happen on a monthly cadence to keep parity with download.openzim.org. During this process, we should produce tsv "dump" files in the same format as the current wp1_selection_tools, and preferably upload them to that website. As a side effect, we can also write the data to the page_scores table described in the design doc, since we will already have the calculated values.

kelson42 commented 3 months ago

@audiodude I see nothing I disagree in your comment. I try to phrase it differently to ensure we are aligned:

External sources (databases mostly) need to scraped/scanned to extract the information we need for publishing TSV at https://download.openzim.org/wp1 and save them in our DB
One this done, we will generate our custom selections and scoring: all.tsv.zip custom/*tsv * `tops/tsv`
Then finally we will put in our schedulers, TSV export routines to keep publishing straight: all.tsv.zip langlinks.tsv.zip pagelinks.tsv.zip pages.tsv.zip pageviews.tsv.zip projects.zip ** redirects.tsv.zip