Closed bidoubiwa closed 1 year ago
While, by default the storage should be purged after each crawling process, the enqueued url's were still cached on the server session.
Might be linked to this issue.
The solution I found was to set the list of starting URLS in my own instance of requestQueue, and drop it after the crawling process. See docs
While, by default the storage should be purged after each crawling process, the enqueued url's were still cached on the server session.
Might be linked to this issue.
The solution I found was to set the list of starting URLS in my own instance of requestQueue, and drop it after the crawling process. See docs