Closed dbrower closed 3 years ago
The carrel scrapers, especially for the public carrels, get a lot of sqlite errors that the database is locked.
??? Don, let's discuss. --Eric
Implemented batching of updates. Because of a sqlite limitation on "INSERT OR REPLACE" each batch can have no more than 500 rows.
The carrel scrapers, especially for the public carrels, get a lot of sqlite errors that the database is locked. This might be pointing to a limitation of using sqlite. But before calling it done, try to optimize the scraping to only save everything into sqlite once at the very end instead of doing it once for each carrel.