Closed dylanmccall closed 3 years ago
@dylanmccall Sorry for the late feedback. There is no problem with this request but why not using the 50.000 tsv list we use to generate like before? What is so special with this list?
@dylanmccall Sorry for the late feedback. There is no problem with this request but why not using the 50.000 tsv list we use to generate like before? What is so special with this list?
The only reason for that is our custom index page has some articles which aren't in that list, so this list is that existing tsv plus some articles in the custom index page. (I'm unsure if there's a mechanism to add articles linked from the index page automatically, so I added them to the list to be sure).
@dylanmccall This is definitly the way to go with what we have for the moment, but believe just having a whitelist (like we have a blacklist) with surnumerous/complementary articles would be a better idea. Will try to implement it that way.
Actually the principle of the whitelist was already implemented. We have one for example for the selection in indonesian. I have updated everything and a new ZIM should be ready in a week (unfortunately I can not do quicker as I'm in vacation).
This is fixed now and works fine. Unfortunately the scrape is impacted by an other bug https://github.com/openzim/mwoffliner/issues/1464
https://farm.openzim.org/recipes/wikipedia_en_endless/config
In addition, please replace
articleList
(http://download.openzim.org/wp1/enwiki/customs/endless.tsv) with the included (zipped) file: endless.tsv.zip