Open kelson42 opened 5 years ago
The recipe didn't succeed
Reopening, ZIM is still not published.
Last run (https://farm.openzim.org/pipeline/fda86c72-527b-480c-963d-5160336068c5) was quite successfully processing the website but I had to stop it, it was at 29% (338277 / 1150670) - yes, more than 1Million pages - after already 20 days in the pipe.
This does not seem reasonable, one has to identify why so many pages are present / wether this is really needed to scrape all these pages.
This million pages thing reminds me of #650 (though this other one as microsoft is slightly more believable). Could it be a scraper issue where it sees pages that do not really exist?
All these pages do really exist. They might be a bit "virtual" in the sense that they are URLs to pages which are dynamically rendered on the real server, or even a bit fake due to a bug in upstream server, but the crawler (except very rare bug) cannot invent new URLs.
From @amirh123123 on September 14, 2018 10:52
ganjoor.net/
Copied from original issue: openzim/mwoffliner#376