Closed Prometheus3375 closed 4 years ago
Here is the project archive. Now db and strategy workers share the same database. Even if they do not share, the issue is still present.
The body is None
because STORE_CONTENT
is false by default.
I think this should be mentioned in Discovery strategy description.
I am using discovery strategy. This is the database of DB worker after adding seeds, other tables are empty.
After the moment when strategy worker, spider and db worker are started, crawling begins. Since robots.txt are scheduled, they are processed first. Unfortunately, response of robots.txt has body equals to
None
for some reason.Such errors are raised for all 3 seeds. What is the reason of such behaviour?