istresearch scrapy-cluster issues

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

http://scrapy-cluster.readthedocs.io/

MIT License

1.18k stars 324 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Should "Found url to crawl" logging string be unicode?

#119 py-in-the-sky closed 7 years ago
2
Make scuitls Logger callbacks Python 2 and Python 3 compatible

#118 joequery closed 7 years ago
2
TypeError: can't pickle thread.lock objects

#117 wenxzhen closed 7 years ago
3
UI Integration Checklist

#116 madisonb opened 7 years ago
4
Why Couldn't initialize kafka consumer for topic

#115 sangli00 closed 7 years ago
1
Does redis-monitor can connect redis with password?

#114 movingheart closed 5 years ago
2
Kafka startup fails - Docker compose

#112 thebennos closed 7 years ago
3
Release 1.2

#110 madisonb closed 7 years ago
1
Updated dmoz link

#109 JayWhittaker closed 7 years ago
1
Scrapy-Cluster doesn't respect the Request type

#108 wenxzhen closed 5 years ago
5
Limit number of crawled pages per domain

#107 kazuar closed 6 years ago
1
Bloom Filter based on Redis

#105 leopeng1995 closed 7 years ago
4
Can Scrapy Cluster work with Scrapy 1.3.3?

#104 wenxzhen closed 7 years ago
2
Limit number of crawled pages per domain

#103 kazuar closed 6 years ago
4
Error in installing requirements.txt

#102 chenweisomebody126 closed 7 years ago
7
Zookeeper and kafka source urls do not work in vagrant quickstart

#101 kazuar closed 7 years ago
4
Release 1.2 Checklist

#100 madisonb closed 7 years ago
2
Error running offline tests

#99 vkchan closed 7 years ago
9
ELK Docker Integration - Elasticsearch Unreachable

#98 DreadfulDeveloper closed 7 years ago
2
The scrapy-cluster support redis cluster?

#97 gongshaojie12 closed 7 years ago
3
Simple spider that sends keyword counts to some database.

#96 py-in-the-sky closed 7 years ago
2
UI service MVP

#95 damienkilgannon closed 7 years ago
20
[Middleware Settings] Integrate scrapy-splash into scrapy-cluster

#94 hustshawn closed 7 years ago
7
Migrate to using kafka-python futures callback

#93 madisonb closed 7 years ago
1
Scutils log callbacks

#92 joequery closed 7 years ago
12
LogFactory Callbacks

#91 madisonb closed 7 years ago
2
subdomain based throttling

#90 softwarevamp closed 7 years ago
3
moderate throttling not works as expected

#89 softwarevamp closed 7 years ago
4
callback Issues

#88 rksaxena closed 7 years ago
6
New Rest Component

#87 madisonb closed 7 years ago
2
Merge pull request #1 from istresearch/master

#86 carlesm closed 7 years ago
2
Circuit breaker design patterns

#85 madisonb opened 8 years ago
0
crawlid KeyError when using the default spider with maxdepth>=1

#84 arthurpham closed 8 years ago
6
change the word error of design.rst

#83 shangliuyan closed 8 years ago
1
Merge pull request #1 from istresearch/dev

#82 skymaze closed 8 years ago
1
Cloned it. Did vagrant up, but couldn't get past step 3.

#81 everyoneknowsadave closed 8 years ago
1
Two new middlewares

#80 madisonb closed 8 years ago
1
How to solve this?Tried to stop a LoopingCall that was not running.

#79 keimhaqi closed 8 years ago
7
Trouble when adding a new spider

#78 casertap closed 8 years ago
6
dockerized sc : "python kafkadump.py dump -t demo.crawled_<appid>" does not dump out the scraped site

#77 Paw2Pom closed 8 years ago
2
PEP8 compliance and some other cleanups

#76 tarhan closed 7 years ago
5
docker settings.py : env variable naming + incorrect [] array bracket for KAFKA_HOST

#75 Paw2Pom closed 8 years ago
5
ELK - export.json for Kibana contains incorrect field (if log is updated via filebeat)

#74 Paw2Pom closed 8 years ago
4
First part of Docker images optimizations

#73 tarhan closed 8 years ago
12
Add Docker Support

#72 madisonb closed 8 years ago
3
Remove Ansible requirement from Host OS in case of Vargant deployment.

#71 tarhan closed 8 years ago
3
kafka/zookeeper install failed

#70 z333d closed 8 years ago
3
Logging non-json to include content of `extras` dict

#69 andrewkcarter closed 7 years ago
2
Production ready ?

#68 PradeepJaiswar closed 8 years ago
2
CentOS 7 Compatibility. Merged with dev branch.

#67 knirbhay closed 8 years ago
22

Previous Next