issues
search
shriphani
/
pegasus
:racehorse:✈️ Pegasus is a scalable, modular, polite web-crawler for Clojure
http://getpegasus.io
Eclipse Public License 1.0
262
stars
17
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
updates to core dependencies, harden default agent writing behavior, set cookie policies for clj-http
#43
akacase
closed
6 years ago
0
bump version and clojure to 1.9 to alleviate spec error
#42
akacase
closed
6 years ago
0
option for non-friendly crawl.
#41
akacase
closed
6 years ago
4
Upgrade to latest core.async for Clojure 1.9
#40
tirkarthi
closed
6 years ago
1
Clojure 1.9 async problem
#39
collinalexbell
closed
6 years ago
1
Dsl for crawling
#38
shriphani
closed
8 years ago
0
fix foo.clj and allow custom states for users
#37
shriphani
closed
8 years ago
0
Crawler failing
#36
fabianmurariu
closed
8 years ago
3
Relax state assumption?
#35
ejschoen
closed
8 years ago
6
Timeouts part of config
#34
shriphani
closed
8 years ago
0
warc writer integration
#33
shriphani
opened
8 years ago
0
make a new host pipeline component
#32
shriphani
opened
8 years ago
0
added insecure-true
#31
shriphani
closed
8 years ago
0
Spalakod/pipeline protocol
#30
shriphani
closed
8 years ago
0
https ?
#29
shriphani
closed
8 years ago
1
bumped this up to two tb
#28
shriphani
closed
8 years ago
0
MDB overflows - set up data-structures based on data-size
#27
shriphani
closed
8 years ago
2
DSL for crawls.
#26
shriphani
closed
8 years ago
2
Restarts and incremental crawls
#25
shriphani
opened
8 years ago
3
enqueue log littering
#24
shriphani
closed
8 years ago
1
timeouts for the frontier should be a parameter.
#23
shriphani
closed
8 years ago
1
Nondeterministic hanging on enqueue-url
#22
dhruvbhatia
closed
8 years ago
4
Merge pull request #1 from shriphani/master
#21
dhruvbhatia
closed
8 years ago
1
Writer bugfix
#20
shriphani
closed
8 years ago
0
A better pipeline
#19
shriphani
closed
8 years ago
0
switch to fort-knox
#18
shriphani
closed
8 years ago
0
Move to lmdb
#17
shriphani
closed
8 years ago
0
Update deps.
#16
dhruvbhatia
closed
8 years ago
0
switch to clj-lmdb
#15
shriphani
closed
8 years ago
1
Multiple files - every so often switch files
#14
shriphani
opened
8 years ago
0
Gzip writer
#13
shriphani
closed
8 years ago
0
S3 writer
#12
shriphani
opened
8 years ago
0
default writer improvements
#11
shriphani
closed
8 years ago
1
discard jcs dependency
#10
shriphani
closed
8 years ago
1
Fixes 7
#9
shriphani
closed
8 years ago
0
exponential backoff
#8
shriphani
opened
8 years ago
3
Error in test-stop-unique
#7
djg123
closed
8 years ago
7
A few minor enhancements
#6
dhruvbhatia
closed
8 years ago
0
tests only work on *nix.
#5
shriphani
closed
8 years ago
1
restart support
#4
shriphani
opened
8 years ago
0
Init looks ugly and all over the place - unify it.
#3
shriphani
closed
8 years ago
1
with-config macro
#2
shriphani
closed
8 years ago
1
Support for crawlers other than clj-http?
#1
dhruvbhatia
opened
8 years ago
12