search

rmax / scrapy-redis

Redis-based components for Scrapy.
http://scrapy-redis.readthedocs.io
MIT License
5.52k stars 1.59k forks source link

There is a compatibility issue with the latest scrapy-redis package of the new version of scrapy #289

Closed georgeJzzz closed 5 months ago

georgeJzzz commented 7 months ago

I encountered a new problem。 Program error interrupt, I am using scrapy==2.11.1, scrapy-redis==0.7.3

image

In addition, since the latest package of scrapy-redis that is compatible with the new version of scrapy has not been released, need to package it

I tried to package it myself and it works fine, is it possible to update the latest scrapy-redis package? Very necessary

thanks

Ehsan-U commented 7 months ago

I'm also encountering the same problem, using scrapy==2.11.1, scrapy-redis==0.7.3

georgeJzzz commented 7 months ago

I'm also encountering the same problem, using scrapy==2.11.1, scrapy-redis==0.7.3

Just need an official upgrade to the compatibility package. I upgraded locally, but it's not convenient for future use。

Ehsan-U commented 7 months ago

i'm not able to upgrade locally, how you did that? image

i'm using Ubuntu 22

georgeJzzz commented 7 months ago

git clone https://github.com/darkrho/scrapy-redis.git cd scrapy-redis

After executing make clean && make dist, pypi will be generated in the dist directory, just install it.

Ehsan-U commented 7 months ago

image

there is no dist directory

georgeJzzz commented 7 months ago

My mistake,

you try python setup.py install

Ehsan-U commented 7 months ago

Yes python setup.py install

image

i already installed zope-interface 6.2 installed on my system.

shengchenyang commented 7 months ago

i already installed zope-interface 6.2 installed on my system.

@Ehsan-U It's not a zope-interface problem, it's Twisted. You need to pip install Twisted==22.10.0 and then python setup.py install again.

Twisted version needs to be less than or equal to 22.10.0.

This project can consider optimizing dependency management.

Ehsan-U commented 7 months ago

@shengchenyang Thanks, its upgraded successfully.

VQHieu1012 commented 5 months ago

Hello @shengchenyang , how did you know the problem is Twisted? Can you share how you found out this?

shengchenyang commented 5 months ago

@VQHieu1012 Compatibility issues can be located through the install_requires parameter in setup.py, which is requirements.txt.Then you need to check step by step.

But I still recommend running pip install wheel && make dist in your python virtual environment. The binary package will be generated in the dist folder.Instead of using python setup.py install.

VQHieu1012 commented 5 months ago

@shengchenyang thanks for sharing.

sify21 commented 4 months ago

I did this in requirements.txt, which seems ok

scrapy-redis @ git+https://github.com/rmax/scrapy-redis@48a7a8921ae064fe7b4202b130f1054ede9103d6