Working with requests and responses in scrapy

[mbuthiajn]

Proposed title of article

Working with requests and responses in scrapy

Proposed article introduction

Scrapy uses Request and Response objects for crawling web sites. Typically, Request objects are generated in the spiders and pass across the system until they reach the Downloader, which executes the request and returns a Response object which travels back to the spider that issued the request.

Both Request and Response classes have subclasses which add functionality not required in the base classes. These are going to be described in Request subclasses and Response subclasses during the course of this article.

Key takeaways

1.Request objects 2.Passing additional data to callback functions 3.Accessing additional data in errback functions 4.Request.meta special keys 5.Stopping the download of a Response 6.Request subclasses

7. Conclusion

Article quality

I plan to use examples to illustrate requests and responses in scrapy and its application in Python. I will also use a language that a beginner will understand and still remain relevant to seasoned programmers.

References

N/A

[louisefindlay23]

Hi, @mbuthiajn. There are many similar articles on this topic. Can you tell me what unique content you will cover or unique perspective you will add.

[WanjaMIKE]

I think this topic does not add much value than what is already covered in the official documentation. Feel free to suggest another topic. Thanks