search

hochschule-darmstadt / openartbrowser

Exploring the world of arts using open data
http://openartbrowser.org/
MIT License
40 stars 10 forks source link

#527 Implements retries while fetching the wikipedia extracts. #545

Closed DavidReyer closed 3 years ago

DavidReyer commented 3 years ago

527 Each time a language and type extract fails the script tries again with a smaller batch size. This also changes request_utils.py as this script handled the error caused by the chunk size being too large by calling exit(-1), completely stopping the crawl which is less than ideal.

This has been tested in branch 524 and is working as intended.

DavidReyer commented 3 years ago

I pushed an increase to the chunk size after a successful chunk as long as it is <20.