issues
search
istresearch
/
scrapy-cluster
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.17k
stars
323
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
how to process different type of item in Processor?
#219
ChengkaiYang2022
closed
5 years ago
3
what's the purpose of item_copy in pipelines.py?
#218
ChengkaiYang2022
closed
5 years ago
2
Plans to support latest version of Kafka, ELK & Scrapy?
#217
kumaresra
closed
5 years ago
1
add timeout to urlopen. For some reason urlopen may block for a while.
#216
mooosu
closed
3 years ago
4
Replace Kafka with another pub/sub service?
#215
jrmlhermitte
closed
5 years ago
3
Website did't response for a long time,how to solve the problem
#214
Johnson0016
closed
5 years ago
1
cluster mode online test hangs
#213
danmsf
closed
5 years ago
3
How does it work with CrawlSpider?
#212
kevin-ZZZ
closed
5 years ago
1
Does it support python3?
#211
xingzhicn
closed
5 years ago
1
Fill a single item from many requests / "can't pickle thread.lock objects"
#210
llermaly
closed
5 years ago
3
Adding support of shared cookies for requests having the same crawlid
#209
MohamedMb
opened
5 years ago
3
Adding support of shared cookies for requests having the same crawlid
#208
MohamedMb
closed
5 years ago
1
adding connection timeouts for redis clients
#207
ronniesworld
closed
5 years ago
2
Docker compose : kafka and crawler containers restarting forever
#206
llermaly
closed
5 years ago
6
Pass cookies through the distributed scheduler
#205
devspyrosv
closed
5 years ago
1
Rest - signal only works in main thread
#204
Shique
closed
5 years ago
2
Lost config from Zookeeper makes spider down
#203
jamesliu668
closed
5 years ago
3
Add Kafka Request Size Param to Crawler Producer
#202
madisonb
closed
5 years ago
1
Removed method timer wrapper in non-main thread
#201
madisonb
closed
5 years ago
1
Improve global page per domain filter documentation.
#200
devspyrosv
closed
5 years ago
1
cryptography library update to 2.3
#199
damienkilgannon
closed
5 years ago
0
remove unused code.
#198
Anderson-Liu
closed
6 years ago
1
Rebuilding crawler container fails
#197
mrvnklm
closed
6 years ago
1
scrapy-cluster stops crawling of some websites after a while
#196
alotfitakami
closed
6 years ago
2
add redis password
#195
aafeher
closed
5 years ago
3
Missing KAFKA_PRODUCER_TOPIC in docker rest settings
#194
aafeher
closed
6 years ago
1
Crawl complete signal
#193
wysiwygism
closed
6 years ago
5
Docker vs Vagrant
#192
mshahriarinia
closed
6 years ago
1
SC-Threaded-counter:
#191
devspyrosv
closed
5 years ago
1
Adding support for issue #182
#190
knirbhay
opened
6 years ago
3
Getting twisted.web._newclient.ResponseNeverReceived exception
#189
spidysenses
closed
6 years ago
2
Add login in LinkSpider
#188
josselinlbe
closed
6 years ago
2
Help processing pages returned by crawler
#187
hellsingnorevy
closed
6 years ago
2
Kafka-Monitor stats can potentially accumulate indefinitely if Redis restarts/fails.
#186
devspyrosv
closed
5 years ago
2
Initial package updates
#185
madisonb
closed
6 years ago
1
:pencil2: Typo on the document of kafka-monitor API section
#184
chihkaiyu
closed
6 years ago
1
Typo on document of Kafka-monitor API section
#183
chihkaiyu
closed
6 years ago
2
Support for custom header and cookies for the initial request from kafka_monitor.py feed
#182
knirbhay
opened
6 years ago
5
Centos systemd only
#181
devspyrosv
closed
6 years ago
2
lxmlhtml fix relative html link joins
#180
devspyrosv
closed
6 years ago
1
Start supervisord in systemd debian systems
#179
devspyrosv
closed
6 years ago
2
Inter-spider communication
#178
shenbakeshkishore
closed
6 years ago
1
DUPEFILTER_CLASS Doesn't Work
#177
jamesliu668
closed
6 years ago
1
No output when dumping incoming or outbound_firehose
#176
cliff-km
closed
6 years ago
14
Fix Travis build job
#175
madisonb
closed
6 years ago
2
Uitest
#174
backtrack-5
opened
6 years ago
21
Ui-UI test
#173
backtrack-5
closed
6 years ago
1
What is Kafka's role in scrapy-cluster?
#172
crisfan
closed
6 years ago
2
Is it necessary to override the spider's start_requests method
#171
newthis
closed
6 years ago
3
Can you upgrade scrapy to version 1.5? There is an error in the upgrade
#170
zusrsoft
closed
6 years ago
4
Previous
Next