openzim / ted

Provide the best of TED.com for offline usage!
https://download.kiwix.org/zim/ted/
GNU General Public License v3.0
13 stars 9 forks source link

Detect scraping blockage #138

Open rgaudin opened 2 years ago

rgaudin commented 2 years ago

In some cases #136 the scraper doesn't get results from the API, apparently due to some blockage on TED servers' end. We should be able to detect when this happens and report it properly.

Overcoming the blockage would mainly start with limiting concurrent scraping at zimfarm level