search

openzim / wikihow

WikiHow scraper
https://download.kiwix.org/zim/wikihow/
GNU General Public License v3.0
16 stars 2 forks source link

WikiHow pt 30% smaller between Feb and May #126

Closed kelson42 closed 2 years ago

kelson42 commented 2 years ago

Look at https://download.kiwix.org/zim/wikihow/

Wikihow PT for example https://files.slack.com/files-pri/T41CNGNKF-F03GJ1UV7DX/image.png

rgaudin commented 2 years ago

The February one was before the 1.1.0 release which completely changed the way we discover articles.

Looking at the log, I see many 404 on Articles with encoded characters so I'd advise you re-run it using #124's fix. Should it be the reason for the change, we should then re-run them all.

holta commented 2 years ago

Thanks for posting this for investigation. Possibly the failure of all top-level links in the March wikiHow below...might be related?

Every link is broken within the 19 category pages under: http://iiab.me/kiwix/wikihow_fr_maxi_2022-03/ (8.3G)

Whereas these links all worked fine a month prior: http://iiab.me/kiwix/wikihow_fr_maxi_2022-02/ (15G)

rgaudin commented 2 years ago

Not sure to understand what you mean by top-level links. Please share complete URLs. I could easily find missing articles in there though ; all of which included encoded chars in their paths so this may be related.

holta commented 2 years ago

Every link is broken within the 19 category pages under: http://iiab.me/kiwix/wikihow_fr_maxi_2022-03/ (8.3G)

Whereas all such links worked in the prior month below:

Whereas these links all worked fine a month prior: http://iiab.me/kiwix/wikihow_fr_maxi_2022-02/ (15G)

holta commented 2 years ago

Every link is broken within the 19 category pages under: http://iiab.me/kiwix/wikihow_fr_maxi_2022-03/ (8.3G)

  • Click on any of the 19 category pages above
  • Click any link under Articles dans la catégorie : (every such link, on all pages, appears broken!)

CORRECTION/CLARIFICATION:

I clicked on ~20 such links and the majority were broken links.

(Click randomly on the 19 category pages, and you will see!)

rgaudin commented 2 years ago

OK, that matches what I encountered. Thanks for confirming, I was afraid we were not looking at the same thing.

holta commented 2 years ago

The 18G ZIM file published yesterday appears perfect:

http://iiab.me/kiwix/wikihow_fr_maxi_2022-05/

Thanks for the fix(es), which hopefully also help wikiHow in other languages too !

rgaudin commented 2 years ago

Yes, it was a generic fix. Thank you for reporting it