stewartmckee cobweb issues

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.

MIT License

227 stars 45 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Bump sinatra from 1.4.8 to 2.2.3

#69 dependabot[bot] opened 1 year ago
0
Bump addressable from 2.3.8 to 2.8.1

#68 dependabot[bot] opened 1 year ago
0
Bump sinatra from 1.4.8 to 2.2.0

#67 dependabot[bot] closed 1 year ago
1
Bump sidekiq from 3.5.3 to 6.4.0

#66 dependabot[bot] opened 2 years ago
0
Bump sidekiq from 3.5.3 to 6.2.2

#65 dependabot[bot] closed 2 years ago
1
Bump nokogiri from 1.8.5 to 1.12.5

#64 dependabot[bot] opened 2 years ago
0
Bump addressable from 2.3.8 to 2.8.0

#63 dependabot[bot] closed 1 year ago
1
Bump nokogiri from 1.8.5 to 1.11.4

#62 dependabot[bot] closed 2 years ago
1
Error raised when there's a valid <base> tag in <head>

#61 svenaas opened 3 years ago
3
Bump rack from 1.6.11 to 2.2.3

#60 dependabot[bot] opened 4 years ago
0
Bump rake from 12.3.0 to 13.0.1

#59 dependabot[bot] opened 4 years ago
0
Bump nokogiri from 1.8.5 to 1.10.8

#58 dependabot[bot] closed 3 years ago
1
Bump rack from 1.6.11 to 2.0.8

#57 dependabot[bot] closed 4 years ago
1
Bump haml from 4.0.7 to 5.1.2

#56 dependabot[bot] opened 4 years ago
0
Bump nokogiri from 1.8.5 to 1.10.4

#55 dependabot[bot] closed 4 years ago
1
Feature request: Stop crawl at time

#54 samnissen opened 7 years ago
3
Falling into Crawl Traps

#53 fuzzygroup opened 7 years ago
10
undefined method `banner' for main:Object (NoMethodError) on calling from command line

#52 gushonorato opened 7 years ago
1
documenting response time for page data returned

#51 gushonorato closed 7 years ago
3
documenting :queue_system option

#50 gushonorato closed 8 years ago
1
Proxy authentication while parsing url with Net::HTTP

#49 suvarnarajkumar opened 8 years ago
2
Error on first run

#48 rap2hpoutre closed 8 years ago
2
Should depend on slop 3.6.0 or update cobweb bin

#47 mhmaguire opened 8 years ago
1
How can I start stop crawling website

#46 qiun opened 8 years ago
1
Relative urls are handled proper

#45 peric closed 8 years ago
1
Cobweb gem causes Rails app to run 10x slower

#44 SimonBirrell closed 8 years ago
9
3309 wf fix riffle on crawl

#43 wflanagan closed 8 years ago
1
Tidy up README

#42 maxehmookau closed 8 years ago
1
error while installing cobweb-1.0.28.gem: Invalid argument @ rb_sysopen

#41 illtellyoulater closed 8 years ago
5
remove reference to awesome print

#40 pisaacs closed 9 years ago
1
LoadError with version 1.0.26

#39 pisaacs closed 9 years ago
2
Aj meta refreshes

#38 andrejj closed 9 years ago
0
Aj meta refreshes

#37 andrejj closed 9 years ago
0
Redirect Limit causing crawl to stop

#36 stewartmckee opened 9 years ago
0
Binary not installed

#35 fabn closed 9 years ago
1
Code organization

#34 andrejj opened 9 years ago
1
Change options checking order to skip redis if cache is nil

#33 adrianouno closed 9 years ago
1
Require namespaced_redis fails. require redis-namespace works.

#32 etagwerker closed 9 years ago
1
Removed puts which generates a lot of noise.

#31 etagwerker closed 9 years ago
1
Issue 29: Normalize URLs when storing inbound links

#30 gh2k closed 9 years ago
1
Inbound links are not normalized when stored

#29 gh2k closed 9 years ago
0
Should it be possible to add "depth" in the data hash ?

#28 ABrisset opened 10 years ago
3
Should be possible to use Ruby's URI implementation instead of Addressable::URI

#27 gh2k opened 10 years ago
3
`require': cannot load such file -- resque (LoadError)

#26 NielsKSchjoedt closed 8 years ago
3
Standalone Crawler gives error for redis

#25 Shehbaz closed 9 years ago
6
Improve connection handling

#24 stewartmckee opened 10 years ago
0
Update ruby to latest patch level

#23 markprovan closed 10 years ago
0
Forced UTF-8 codification of request.body so it can be serialised correc...

#22 wuiscmc closed 3 years ago
0
Encoding problems

#21 wuiscmc closed 10 years ago
5
Add missing config options, fix default values and layout

#20 fqxp closed 10 years ago
0