rmax / scrapy-redis

Redis-based components for Scrapy.
http://scrapy-redis.readthedocs.io
MIT License
5.49k stars 1.59k forks source link

Push json will failed #238

Closed songhao8080 closed 2 years ago

songhao8080 commented 2 years ago

push json will failed

image

class RedisMixin(object):
    .....
    def make_request_from_data(self, data):
        # url = bytes_to_str(data, self.redis_encoding)
        formatted_data = bytes_to_str(data, self.redis_encoding)

        # change to json array
        parameter = {}
        if type(formatted_data) == dict:
            parameter = json.loads(formatted_data)
        else:
            return FormRequest(formatted_data, dont_filter=True)

        url = parameter['url']
        del parameter['url']
        metadata = {}
        try:
            metadata = parameter['meta']
            del parameter['meta']
        except Exception:
            pass

        return FormRequest(url, dont_filter=True, formdata=parameter, meta=metadata)

formatted_data from bytes_to_str will be alwalys str , not be the dict , so will not go to parameter = json.loads(formatted_data)

KashifAhmed commented 1 year ago

How you solve this issue, still i am getting this error [ERROR]: Error downloading <GET %7B'url':%20'https://www. .....