Is there a way to make 'POST' request?

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

http://scrapy-cluster.readthedocs.io/

MIT License

1.18k stars 323 forks source link

Is there a way to make 'POST' request? #166

Closed YanzhongSu closed 6 years ago

YanzhongSu commented 6 years ago

I yield a POST request

yield Request(item["url"], method = 'POST', body = item['attrs']['query'], callback=self.parse)

But it is treated as a GET request. Did I do something wrong or it will get overwritten as 'GET' request?

madisonb commented 6 years ago

This should be fixed in the dev branch - it uses the scrapy built in request serializer vs a custom one. Can you check that and close this ticket if that is the case?

madisonb commented 6 years ago

Closing due to inactivity. The improved serializer should take care of any scrapy request emanating from the spiders.