issues
search
istresearch
/
scrapy-cluster
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k
stars
323
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
ERROR: Could not reach out to get public ip
#169
mohit0749
closed
6 years ago
8
exceptions.KeyError: 'crawlid'
#168
bipabo1l
closed
6 years ago
1
Hotfix 1.2.1
#167
madisonb
closed
6 years ago
1
Is there a way to make 'POST' request?
#166
YanzhongSu
closed
6 years ago
2
Add crawler redis pagecount filter
#165
devspyrosv
closed
6 years ago
7
Is there a way to get dynamic url or other tag links?
#164
bipabo1l
closed
6 years ago
2
it should be "producer" connection in __docs__.
#163
mohit0749
closed
6 years ago
1
Docker Quickstart Error for Crawler container
#162
edolix
closed
6 years ago
6
Fix spider name in crawler's online tests
#161
devspyrosv
closed
6 years ago
1
Update Crawler, Kafka-Monitor, Redis-Monitor requirements
#160
devspyrosv
closed
6 years ago
8
Add auto encoding resolution for crawling items.
#159
devspyrosv
closed
6 years ago
7
ImportError: No module named online
#158
mohit0749
closed
6 years ago
9
Support for Tor + Privoxy
#157
Paw2Pom
closed
6 years ago
1
Redis server failed,what should I do?
#156
crisfan
closed
6 years ago
7
Unclear documentation regarding outbound crawled data integration
#155
synergiator
closed
6 years ago
2
expires in request meta not considered when pop from RedisThrottledQueue
#154
softwarevamp
closed
6 years ago
0
python 3 support
#153
codinguncut
closed
6 years ago
0
Ansible kafka zookeeper host list.
#152
ghost
closed
6 years ago
2
vm.max_map_count tip with link to elastic documentation
#151
ghost
closed
6 years ago
1
Docker Elasticsearch not running
#150
ghost
closed
6 years ago
4
duplicate logging confusing
#149
softwarevamp
closed
5 years ago
3
there is a typer error
#148
softwarevamp
closed
6 years ago
0
Update dmoz.org to dmoztools.net since dmoz.org now redirects. (#145)
#147
topiaruss
closed
6 years ago
1
Trivial tense/typos
#146
topiaruss
closed
6 years ago
5
dmoz.org is used in demos, and now redirects to dmoztools.net
#145
topiaruss
closed
6 years ago
1
trivial verb tense
#144
topiaruss
closed
6 years ago
3
Fix assert deprecations
#143
topiaruss
closed
6 years ago
1
Enable docker-compose build
#142
topiaruss
closed
6 years ago
3
TypeError: can't pickle thread.lock objects
#141
aafeher
closed
6 years ago
2
exposing kafka from docker
#140
KiaraGrouwstra
closed
6 years ago
5
Does anybody get 'Connection pool is full' error when start the docker-compose?
#139
qianlifu12345
closed
6 years ago
2
Utilizing scrapy-cluster to handle high volumes of input URLs
#138
steveshi7
closed
6 years ago
3
past.utils import old_div error
#137
ahlusar1989
closed
6 years ago
1
Fix REST-Kafka topic variable name in docker settings.py
#136
jaturken
closed
6 years ago
2
How to deploy on amazon ECS?
#135
hustshawn
closed
5 years ago
4
Update design.rst
#134
mohit0749
closed
7 years ago
1
redis.exceptions.ResponseError: unknown command 'PFCOUNT'
#133
mohit0749
closed
7 years ago
2
Improve request serialization handling in scheduler
#132
gas1121
closed
7 years ago
7
log failure exception
#131
softwarevamp
closed
5 years ago
3
lstat crawler/requirements.txt: no such file or directory
#130
jin10086
closed
7 years ago
7
Scrapy-Cluster UI
#129
damienkilgannon
closed
7 years ago
17
Zookeeper watcher's behavior is different on python 3
#128
gas1121
closed
7 years ago
6
Travis build sometimes fails
#127
gas1121
closed
3 years ago
4
Add python 3 support.
#126
gas1121
closed
7 years ago
11
Hotfix stats collector deactivate
#125
licenseplated
closed
7 years ago
2
add deactivate method to ThreadedCounter
#124
licenseplated
closed
7 years ago
2
Redis queue takes a lot of memory.
#123
arheys
closed
7 years ago
4
scutils py3 support
#122
damienkilgannon
closed
7 years ago
4
Scutils LogFactory update
#121
madisonb
closed
7 years ago
1
Handle URLs with unicode characters in logging statement.
#120
py-in-the-sky
closed
7 years ago
3
Previous
Next