search

scrapinghub / frontera

A scalable frontier for web crawlers
BSD 3-Clause "New" or "Revised" License
1.29k stars 216 forks source link

Add support for “post requests and data” when integrating scrapy. #317

Open CN-hanyi opened 6 years ago

CN-hanyi commented 6 years ago

Add support for “post requests and data” when integrating scrapy. Resolve the bug for “wrong request data in metadata” when response updates metadata.

sibiryakov commented 6 years ago

Hi @CN-hanyi thank you for your PR! Would be great to have a tests passing and to propagate your changes to all backends: Memory, HBase, MessageBus. Probably it's also worth to check if it needs to be mentioned in docs.

CN-hanyi commented 6 years ago

Thanks for your reply. I have added the test for the submitted code. Add test for “post request and data support when integrating scrapy ” , propagate this change to all backends: memory, hbase,redis . propagate this change to json codecs。 Add MessageBusBackend test for new codecs(json and msgpack)。 Fix some bugs after the test.