istresearch scrapy-cluster issues

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

http://scrapy-cluster.readthedocs.io/

MIT License

1.17k stars 324 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

ERROR: Unable to connect to Kafka in Pipeline due to attempt to connect already-connected SSLSocket!, raising exit flag.

#269 BeamoINT opened 10 months ago
1
runspider: error: Unable to load 'link_spider.py': attempted relative import with no known parent package

#268 BeamoINT closed 8 months ago
4
Upgrade the project to python 3.10

#267 borisjota opened 10 months ago
4
Is this project no longer supported?

#266 flyingdev closed 10 months ago
2
Update Dockerfile

#265 shivaabed1994 opened 2 years ago
0
fix missing package python-lxml

#264 itisrasoul closed 2 years ago
0
lxml dependency is broken for python3.6 base image

#263 rafaharo opened 2 years ago
0
Updated Names and statement for localsettings.py

#262 abhalawat opened 2 years ago
0
Crawler image - Python dependency fix

#261 igortelheiro closed 2 years ago
0
ui exception No connection adapters were found

#260 new-wxw opened 2 years ago
0
Crawler unhandled exceptions not logged

#259 getorca closed 3 years ago
1
scrapy-cluster don't scroll the entire pages

#258 RochdiBoudokhane closed 3 years ago
7
Fix online dmoz test

#257 madisonb closed 3 years ago
0
Update quickstart.rst

#256 oprog closed 3 years ago
0
Stats keys flooding redis + Crawl resilience + retry strategy

#255 benjaminelkrieff closed 3 years ago
5
Incremented fail stats

#254 mingxuan1 closed 3 years ago
0
Upgrading the ELK stack

#253 4OH4 opened 3 years ago
2
TypeError: can't pickle thread.lock objects

#252 benjaminelkrieff opened 3 years ago
2
Jobs disappearing: is there a way to monitor crawls ?

#251 benjaminelkrieff closed 3 years ago
4
Scrape delay

#250 tluyben closed 3 years ago
2
docs: fix grammatical errors in docker.rst

#249 mslowikowski closed 3 years ago
1
Bump jinja2 from 2.10 to 2.10.1 in /rest

#248 dependabot[bot] closed 3 years ago
1
Bump requests from 2.18.4 to 2.20.0 in /rest

#247 dependabot[bot] closed 3 years ago
1
Bump werkzeug from 0.14.1 to 0.15.3 in /rest

#246 dependabot[bot] closed 3 years ago
1
Bump requests from 2.18.4 to 2.20.0 in /crawler

#245 dependabot[bot] closed 3 years ago
1
Bump requests from 2.18.4 to 2.20.0 in /kafka-monitor

#244 dependabot[bot] closed 3 years ago
1
Bump twisted from 18.4.0 to 20.3.0 in /crawler

#243 dependabot[bot] closed 3 years ago
1
Dependency Resolution Fixes

#242 madisonb closed 3 years ago
0
maxdepth can not large than 2

#241 anthony9981 closed 3 years ago
5
Bump cryptography from 1.8.1 to 3.2

#240 dependabot[bot] closed 3 years ago
1
Bump cryptography from 1.8.1 to 3.2 in /crawler

#239 dependabot[bot] closed 3 years ago
1
Update Releases, Docker compose Issues

#238 MrMoon closed 3 years ago
2
Ansible deprecated sudo

#237 j3k00 closed 3 years ago
1
Add missing ELK restart instructions to docker-compose file

#236 Shique closed 3 years ago
1
Future of the project

#235 demisx opened 4 years ago
9
Fixed an error with MarkupSafe installation

#234 Shique closed 4 years ago
2
Fixed an error with MarkupSafe installation

#233 Shique closed 4 years ago
0
Any suggestion on the best way to get the crawled data out to a file?

#232 ahoquegh closed 4 years ago
0
How to submit a POST request through crawler REST API

#231 YanzhongSu closed 4 years ago
2
Python 3

#230 deed02392 opened 4 years ago
6
Why encoding byted body

#229 YanzhongSu closed 4 years ago
2
Bump werkzeug from 0.12.1 to 0.15.3

#228 dependabot[bot] closed 4 years ago
1
Can't up cluster from master branch

#227 MasterSergius closed 4 years ago
1
supervisord install error while vagrant up

#226 fatemeeeeeee closed 4 years ago
2
A mistake in parameter deliver through two method!

#225 kevin-ZZZ closed 4 years ago
4
online integration test error

#224 rawiafray closed 4 years ago
2
Cannot turn off DEBUG level log

#223 YanzhongSu closed 5 years ago
2
Coveralls fix

#222 madisonb closed 5 years ago
1
`Demo.incoming` topic keeps piling up

#221 YanzhongSu closed 5 years ago
6
Docker ERROR: Could not ping Zookeeper

#220 Shique closed 5 years ago
6