marsara9
commented
1 year ago Looks like the issue might have been because of the original way posts were being added to the database, with single query for EACH post, EACH author and EACH community, etc... Now that all of those are batched, the issue seems to have gone away.
Every time the crawler starts, it makes it through about 3000-4000 posts before encountering an error.
Need to find a way to gracefully handle these but at the same time they can't be skipped. Unless we also want to find a way to restart the indexing progress periodically. But as it stands the number of new posts being created per day is exceeding the number of posts being crawled.