issues
search
postmodern
/
spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
MIT License
800
stars
109
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
fetch_titles not following 301
#37
audy
closed
2 years ago
2
Added possibility to pass cookies as options
#36
spk
closed
9 years ago
5
Documentation fix about http response class
#35
spk
closed
9 years ago
0
HTTP Basic auth problem
#34
tit
closed
11 years ago
4
Rewrite spec/helpers/wsoc.rb as a shared_example with 100% less eval
#33
postmodern
closed
2 years ago
1
Spidering pages with no content-type header
#32
bcobb
closed
12 years ago
4
Infinite path loop
#31
ethicalhack3r
opened
12 years ago
2
SSL session reuse may fail
#30
nirvdrum
opened
12 years ago
1
handle NoMethodError exception when @sessions[url]=null
#29
fakechris
closed
12 years ago
4
catching SSLErrors
#28
lucasluitjes
closed
12 years ago
2
#<NoMethodError: undefined method `closed?' for nil:NilClass>
#27
ethicalhack3r
closed
2 years ago
1
Multithreading
#26
ethicalhack3r
opened
13 years ago
13
uninitialized constant Spidr::Headers::Set
#25
scottsampson
closed
13 years ago
3
application/xml should be handled as xml
#24
rfletcher
closed
13 years ago
1
Link depth?
#23
ethicalhack3r
closed
13 years ago
6
fix issue #14 "bad component(expected absolute path component)"
#22
emseabrown
closed
13 years ago
2
99% of cpu usage while crawling bigger websites...
#21
nu7hatch
closed
13 years ago
1
Improve network connection to HTTPS server via HTTPSProxy.
#20
falaise
closed
13 years ago
2
Automatically detect and parse sitemap.xml
#19
postmodern
opened
13 years ago
7
Store history queue in Hash of host:port and paths.
#18
postmodern
opened
13 years ago
0
every_html_page shouldn't process javascripts
#17
nu7hatch
closed
13 years ago
7
trailing slashes
#16
nu7hatch
closed
13 years ago
1
The :depth option support
#15
nu7hatch
closed
13 years ago
3
expected absolute path component: sites/ftp.apache.org/
#14
ethicalhack3r
closed
13 years ago
4
Redirect fails silently - need debug messages
#13
perplexes
opened
14 years ago
4
Malformed mailto error
#12
ethicalhack3r
closed
14 years ago
2
Any way to automatically obey robots.txt?
#11
cbmeeks
closed
8 years ago
3
Multiple cookies from same page
#10
ethicalhack3r
closed
13 years ago
1
undefined method `attr' for #<Nokogiri::XML::Element:0xb77423c4> (NoMethodError)
#9
ethicalhack3r
closed
14 years ago
2
handle css @import spidering
#8
zapnap
opened
14 years ago
0
Odd Net::HTTP error when requesting the last page.
#7
postmodern
closed
14 years ago
1
Spidr behaves oddly on bad host-names under Ruby 1.8.7-p249
#6
postmodern
closed
13 years ago
4
Passing Headers to Net::HTTP
#5
evoltech
closed
14 years ago
4
Problem building gem
#4
evoltech
closed
14 years ago
1
static course specs
#3
zapnap
closed
14 years ago
0
SSL and network, in general
#2
justfalter
closed
14 years ago
4
URL Normalization.
#1
justfalter
closed
14 years ago
3
Previous