issues
search
stewartmckee
/
cobweb
Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
226
stars
45
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Bump sinatra from 1.4.8 to 2.2.3
#69
dependabot[bot]
opened
1 year ago
0
Bump addressable from 2.3.8 to 2.8.1
#68
dependabot[bot]
opened
1 year ago
0
Bump sinatra from 1.4.8 to 2.2.0
#67
dependabot[bot]
closed
1 year ago
1
Bump sidekiq from 3.5.3 to 6.4.0
#66
dependabot[bot]
opened
2 years ago
0
Bump sidekiq from 3.5.3 to 6.2.2
#65
dependabot[bot]
closed
2 years ago
1
Bump nokogiri from 1.8.5 to 1.12.5
#64
dependabot[bot]
opened
2 years ago
0
Bump addressable from 2.3.8 to 2.8.0
#63
dependabot[bot]
closed
1 year ago
1
Bump nokogiri from 1.8.5 to 1.11.4
#62
dependabot[bot]
closed
2 years ago
1
Error raised when there's a valid <base> tag in <head>
#61
svenaas
opened
3 years ago
3
Bump rack from 1.6.11 to 2.2.3
#60
dependabot[bot]
opened
4 years ago
0
Bump rake from 12.3.0 to 13.0.1
#59
dependabot[bot]
opened
4 years ago
0
Bump nokogiri from 1.8.5 to 1.10.8
#58
dependabot[bot]
closed
3 years ago
1
Bump rack from 1.6.11 to 2.0.8
#57
dependabot[bot]
closed
4 years ago
1
Bump haml from 4.0.7 to 5.1.2
#56
dependabot[bot]
opened
4 years ago
0
Bump nokogiri from 1.8.5 to 1.10.4
#55
dependabot[bot]
closed
4 years ago
1
Feature request: Stop crawl at time
#54
samnissen
opened
7 years ago
3
Falling into Crawl Traps
#53
fuzzygroup
opened
7 years ago
10
undefined method `banner' for main:Object (NoMethodError) on calling from command line
#52
gushonorato
opened
7 years ago
1
documenting response time for page data returned
#51
gushonorato
closed
7 years ago
3
documenting :queue_system option
#50
gushonorato
closed
8 years ago
1
Proxy authentication while parsing url with Net::HTTP
#49
suvarnarajkumar
opened
8 years ago
2
Error on first run
#48
rap2hpoutre
closed
8 years ago
2
Should depend on slop 3.6.0 or update cobweb bin
#47
mhmaguire
opened
8 years ago
1
How can I start stop crawling website
#46
qiun
opened
8 years ago
1
Relative urls are handled proper
#45
peric
closed
8 years ago
1
Cobweb gem causes Rails app to run 10x slower
#44
SimonBirrell
closed
8 years ago
9
3309 wf fix riffle on crawl
#43
wflanagan
closed
8 years ago
1
Tidy up README
#42
maxehmookau
closed
8 years ago
1
error while installing cobweb-1.0.28.gem: Invalid argument @ rb_sysopen
#41
illtellyoulater
closed
8 years ago
5
remove reference to awesome print
#40
pisaacs
closed
9 years ago
1
LoadError with version 1.0.26
#39
pisaacs
closed
9 years ago
2
Aj meta refreshes
#38
andrejj
closed
9 years ago
0
Aj meta refreshes
#37
andrejj
closed
9 years ago
0
Redirect Limit causing crawl to stop
#36
stewartmckee
opened
9 years ago
0
Binary not installed
#35
fabn
closed
9 years ago
1
Code organization
#34
andrejj
opened
9 years ago
1
Change options checking order to skip redis if cache is nil
#33
adrianouno
closed
9 years ago
1
Require namespaced_redis fails. require redis-namespace works.
#32
etagwerker
closed
9 years ago
1
Removed puts which generates a lot of noise.
#31
etagwerker
closed
9 years ago
1
Issue 29: Normalize URLs when storing inbound links
#30
gh2k
closed
9 years ago
1
Inbound links are not normalized when stored
#29
gh2k
closed
9 years ago
0
Should it be possible to add "depth" in the data hash ?
#28
ABrisset
opened
10 years ago
3
Should be possible to use Ruby's URI implementation instead of Addressable::URI
#27
gh2k
opened
10 years ago
3
`require': cannot load such file -- resque (LoadError)
#26
NielsKSchjoedt
closed
8 years ago
3
Standalone Crawler gives error for redis
#25
Shehbaz
closed
9 years ago
6
Improve connection handling
#24
stewartmckee
opened
10 years ago
0
Update ruby to latest patch level
#23
markprovan
closed
10 years ago
0
Forced UTF-8 codification of request.body so it can be serialised correc...
#22
wuiscmc
closed
3 years ago
0
Encoding problems
#21
wuiscmc
closed
10 years ago
5
Add missing config options, fix default values and layout
#20
fqxp
closed
10 years ago
0
Next