I realize that I introduced a race condition and while it may have been very unlikely to have occurred in practice, I think it's good to remove it anyway.
To reproduce the race condition error, I added a sleep(10) into the for loop (for url in urls_to_crawl:) and indeed it gives:
I realize that I introduced a race condition and while it may have been very unlikely to have occurred in practice, I think it's good to remove it anyway.
To reproduce the race condition error, I added a
sleep(10)into the for loop (for url in urls_to_crawl:) and indeed it gives: