issues
search
stewartmckee
/
cobweb
Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
227
stars
45
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Thin web-server works slowly
#19
sunloverz
closed
9 years ago
4
License missing from gemspec
#18
bf4
closed
10 years ago
2
Suggestion: Compatibility with Sidekiq
#17
NebJ
closed
10 years ago
4
external_urls not treated as external
#16
stewartmckee
opened
11 years ago
0
Use base url
#15
rojotek
closed
11 years ago
0
I think the gem need to require json when using it standalone
#14
ghost
closed
11 years ago
1
Two improvements for you to look at here - inprogress + updating of setting the queued state
#13
rojotek
closed
11 years ago
1
only try and crawl links on a page 1 time.
#12
rojotek
closed
11 years ago
1
Wrote a fix for default tag options
#11
thomasdavis
closed
11 years ago
1
Changes to redirect and crawl finished
#10
rojotek
closed
11 years ago
0
Improved handling of redirects
#9
rojotek
closed
11 years ago
0
More crawl finished reliability work
#8
rojotek
closed
11 years ago
0
Feature/direct call process job
#7
rojotek
closed
11 years ago
0
Fixed bugs -- please merge.
#6
rojotek
closed
12 years ago
1
Change to ensure that utf8 content can be crawled.
#5
rojotek
closed
12 years ago
0
bug fix on head -- was seeing errors on a redirect
#4
rojotek
closed
12 years ago
1
cobweb http requests should not include the fragment.
#3
rojotek
closed
12 years ago
0
Fixed bug where url('xxx') directives in styles were not handled correctly.
#2
rojotek
closed
12 years ago
0
Running locally and gzip support - and replacing the hash monkey patch with a static helper.
#1
rojotek
closed
12 years ago
0
Previous