istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.17k stars 323 forks source link

A mistake in parameter deliver through two method! #225

Closed kevin-ZZZ closed 4 years ago

kevin-ZZZ commented 5 years ago

Hey,madisonb: I found a problem in parameter delivering! I instantiate an item object in method A,and then deliver it through meta.but I got a list object in method B,not the same item object !

def parse(self,response):
    item = item_object()
    item["xx"] = "xxx"
    yield scrapy.Request(url, meta={"item": item}, callback=self.parse_images)

def parse_image(self,response):
    item = response.meta["item"]

The type of item will be list,not a item object!

However ,when I used a dict object as parameter to deliver through different method,I could get the same dict like this:

def parse(self,response):
    item = item_object()
    item["xx"] = "xxx"
    yield scrapy.Request(url, meta={"item": dict(item)}, callback=self.parse_images)

def parse_image(self,response):
    item = dict(response.meta["item"][0])

And I can get a dict object! Is this a bug or just a mistake? Thanks for help!

madisonb commented 5 years ago

What branch are you on? Can you replicate this behavior in the dev branch too? We changed how the request serialization works and now use the internal Scrapy serializer, so I would be curious if you can replicate it there.

kevin-ZZZ commented 5 years ago

@madisonb I was on dev branch,I used python3.6, but the code was cloned half year ago,So I'm curious when you solve the problem ,All I need do is just need update the code? And it will be nice to show me the git version about this problem?

madisonb commented 4 years ago

Did you ever figure this out? Otherwise I am going to close the issue due to inactivity

madisonb commented 4 years ago

Closing.