issues
search
scrapinghub
/
frontera
A scalable frontier for web crawlers
BSD 3-Clause "New" or "Revised" License
1.29k
stars
215
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
seed urls not loading
#363
arun477
opened
5 years ago
7
Failed install frontera in a cluster mode on Python 3.7
#362
phongtnit
opened
5 years ago
5
Update after redis update
#361
khellan
closed
5 years ago
1
cluster kafka db worker doesnt recognize partitions
#360
danmsf
closed
5 years ago
16
Got error "ERROR:messagebus.stats:TClient instance has no attribute 'get_stats'"
#359
kevin-ZZZ
closed
5 years ago
8
Increase session.timeout.ms for KafkaConsumer
#358
jpbalarini
opened
5 years ago
1
use 2.1.0 version of cachetools
#357
vezunch1k
closed
5 years ago
1
update cachetools to version 3.0.0
#356
vezunch1k
closed
5 years ago
1
Docs: BaseCrawlingStrategy._refresh_states() also takes a non iterab…
#355
IAlwaysBeCoding
closed
5 years ago
2
Addition: Docs need to say that _refresh_states() also takes a non iterable
#354
IAlwaysBeCoding
opened
5 years ago
1
Fix sqlalchemy queue component partition_id
#353
jpbalarini
opened
5 years ago
3
Fix bug when ordering elements on queue when get next batch of requests
#352
jpbalarini
opened
5 years ago
6
Update cluster-setup docs
#351
guillermoap
closed
5 years ago
2
I have done some workaround across implementing ActiveMq to the messagebus so would like to contribute
#350
amitsing89
opened
5 years ago
6
ImportError: cannot import name 'FrontierManager' from 'frontera'
#349
samueljklee
closed
5 years ago
0
idle_spider signal not fired when LOCAL_MODE = False
#348
MottiniMauro
closed
4 years ago
0
Crash on closing redis backend, missing 'force_clear' argument
#347
vfcosta
closed
5 years ago
1
Requirements for Graphs extra incompatible with Python3
#346
krudflinger
opened
6 years ago
1
Remove invalid keyword argument 'sleep' from kafka poll
#345
icapurro
closed
5 years ago
5
How to configure with Scrapy CrawlSpider
#344
villeristi
closed
1 year ago
6
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 140553838536448 and this is thread id 140554174347072
#343
nmweizi
opened
6 years ago
4
Mention local_mode option in docs
#342
sibiryakov
closed
6 years ago
0
Fix general spider example
#341
sibiryakov
closed
6 years ago
0
ModuleNotFoundError: No module named 'frontier.workersetting
#340
songproducer
closed
6 years ago
3
NameError: Module 'frontera.contrib.backends.sqlalchemy' doesn't define any object named 'SQLAlchemyBackend'
#339
nmweizi
closed
6 years ago
2
MessageBusBackend returning null states, breaking SchedulerSpiderMiddleware
#338
anjackson
closed
6 years ago
2
KeyError: b'frontier_request'
#337
nmweizi
closed
6 years ago
3
Python 3.7 support
#336
gastlich
closed
4 years ago
3
don't canonicalize URLs before passing them to URL_FINGERPRINT_FUNCTION
#335
kmike
opened
6 years ago
1
it is not documented how HBaseBackend prioritizes requests
#334
kmike
closed
6 years ago
2
How to make scrapy-splash work with frontera?
#333
JohnMTrimbleIII
opened
6 years ago
2
Test fixes 3
#332
sibiryakov
closed
6 years ago
0
Crawling strategy is moved to FrontierManager
#331
sibiryakov
closed
6 years ago
2
Tests fixing session
#330
sibiryakov
closed
6 years ago
0
Framed transport for Kafka message bus
#329
sibiryakov
opened
6 years ago
1
Black listing in DBW batch-gen
#328
sibiryakov
opened
6 years ago
2
Create_request added to Manager, API changes
#327
sibiryakov
closed
6 years ago
0
No need to flush scoring log on every batch
#326
sibiryakov
closed
6 years ago
0
Skip links-extracted when reached limit for domain
#325
sibiryakov
closed
6 years ago
0
HBase state cache statistics
#324
sibiryakov
closed
6 years ago
0
Non-exhausted generators must be closed manually
#323
sibiryakov
closed
6 years ago
0
Improved logging of batch-gen and SW
#322
sibiryakov
closed
6 years ago
0
Partition split between batch-gen instances
#321
sibiryakov
closed
6 years ago
0
string formatting corrected in the connection information being logged
#320
akshayphilar
closed
6 years ago
0
Can't start zmq broker
#319
tamt
closed
6 years ago
1
Keyword BACKEND Meaning Inconsistent Between Spider and Workers
#318
grammy-jiang
opened
6 years ago
2
Add support for “post requests and data” when integrating scrapy.
#317
CN-hanyi
opened
6 years ago
2
more precise scoring, encoding from scrapy, hash for request
#316
sibiryakov
closed
6 years ago
0
Kafka-python update and params change
#315
sibiryakov
closed
6 years ago
0
DBW components teardown
#314
sibiryakov
closed
6 years ago
0
Previous
Next