issues
search
istresearch
/
scrapy-cluster
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k
stars
324
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Made ansible script CentOS 7 compatible
#66
knirbhay
closed
8 years ago
0
Made ansible script CentOS 7 compatible
#65
knirbhay
closed
8 years ago
9
Slow Scheduler Memory Build Up
#64
madisonb
closed
8 years ago
3
applied suggested changes
#63
saq7
closed
8 years ago
1
refactored KafkaMonitor._process_messages for readability
#62
saq7
closed
8 years ago
3
Concurrent requests from a spider to a domain
#61
esrk
closed
8 years ago
9
Multiple instances of "kafka_monitor" and "redis_monitor"
#60
SHSE
closed
8 years ago
3
Zookeeper Domain blacklist
#59
madisonb
closed
8 years ago
1
Upgrade Unit Testing
#58
madisonb
closed
8 years ago
1
Improve Unit Tests
#57
madisonb
closed
7 years ago
1
Virtualenv fix
#56
madisonb
closed
8 years ago
1
Vagrant test environment missing cryptography method
#55
anderfjord
closed
8 years ago
6
LogFactory rolling log doesnt actually roll
#54
madisonb
closed
8 years ago
1
_get_bin takes hours with queue size 1M.
#53
yrik
closed
8 years ago
11
API
#52
yrik
closed
8 years ago
1
How to get amount of crawled pages for specific crawl request?
#51
yrik
closed
8 years ago
2
Improve documentation
#50
madisonb
closed
7 years ago
1
Scrapy Cluster 1.1 Merge
#49
madisonb
closed
8 years ago
0
Dockerization
#48
madisonb
closed
8 years ago
1
Elastic Moderated Throttled Queue
#47
madisonb
closed
7 years ago
2
Python 3 Support
#46
madisonb
closed
7 years ago
11
Offload virtual machine deployment
#45
madisonb
closed
8 years ago
1
Reduce potential Redis key collisions
#44
madisonb
closed
8 years ago
1
Redis Monitor Locks for processing
#43
madisonb
closed
8 years ago
2
Switch from Kafka-Python to PyKafka
#42
madisonb
closed
8 years ago
4
Add Examples Folder in Utils
#41
madisonb
closed
7 years ago
0
Plugin for Queue Statistics API
#40
madisonb
closed
8 years ago
1
Plugin for Zookeeper Crawler Control
#39
madisonb
closed
8 years ago
1
Can the spiders not exit after all the job done? #question
#38
rocdeng
closed
8 years ago
2
Zookeeper dependency?
#37
sibiryakov
closed
8 years ago
2
Add a Gitter chat badge to README.md
#36
gitter-badger
closed
8 years ago
0
Pass "spiderid" param to feed function and got "invalid json received" error
#35
rocdeng
closed
8 years ago
2
Upgrade to Scrapy 1.0.4
#34
madisonb
closed
8 years ago
1
Feeding speed is slow, how to speed up?
#33
rocdeng
closed
8 years ago
7
Scutils pkging
#32
jasonrhaas
closed
8 years ago
3
1.1 Troubles
#31
quasiben
closed
8 years ago
17
Feature/add travis ci
#30
quasiben
closed
8 years ago
1
Feature/add travis ci
#29
quasiben
closed
8 years ago
0
Feature/add travis ci
#28
quasiben
closed
8 years ago
0
Status
#27
quasiben
closed
8 years ago
2
Discussion: Docker vs Pip vs Virtual Machine
#26
madisonb
closed
8 years ago
2
UI for displaying information about Cluster
#25
madisonb
opened
8 years ago
22
Rest Services for API requests
#24
madisonb
closed
7 years ago
8
Sc pep8
#23
jasonrhaas
closed
8 years ago
0
Scrapy Cluster Test Enviornment
#22
jasonrhaas
closed
9 years ago
4
Sc utils
#21
jasonrhaas
closed
9 years ago
0
tests_offline.py path dependency issue
#20
jasonrhaas
closed
9 years ago
1
Integrate Travis CI for offline tests
#19
jasonrhaas
closed
8 years ago
4
Multiple Spiders in Single Process
#18
madisonb
closed
8 years ago
2
Sample Kibana and Logstash configs
#17
madisonb
closed
8 years ago
1
Previous
Next