search

kwilczynski / ruby-magic

Simple interface to libmagic for Ruby Programming Language
Apache License 2.0
27 stars 8 forks source link

Fix 412 when downloading file from fossies #7

Closed dosuken123 closed 3 years ago

dosuken123 commented 3 years ago

This commit fixes the OpenURI::HTTPError when the gem install ruby-magic runs.

It looks like User-Agent must be specified when the system downloads a file from fossies.

Confirmed in the following test script:

[21] pry(main)> File.open(filename, 'wb') do |target_file|
[21] pry(main)*   URI.open("https://fossies.org/linux/misc/file-#{LIBMAGIC_TAG}.tar.gz", 'rb') do |read_file|  
[21] pry(main)*     target_file.write(read_file.read)    
[21] pry(main)*   end    
[21] pry(main)* end  
/home/shinya/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/pry-0.12.2/lib/pry/exceptions.rb:28: warning: $SAFE will become a normal global variable in Ruby 3.0
OpenURI::HTTPError: 412 Precondition Failed
from /home/shinya/.rbenv/versions/2.7.2/lib/ruby/2.7.0/open-uri.rb:387:in `open_http'
[37] pry(main)> File.open(filename, 'wb') do |target_file|
[37] pry(main)*   URI.open("https://fossies.org/linux/misc/file-#{LIBMAGIC_TAG}.tar.gz", "User-Agent" => "RubyMagic") do |read_file|  
[37] pry(main)*     target_file.write(read_file.read)    
[37] pry(main)*   end    
[37] pry(main)* end  
=> 954266
dosuken123 commented 3 years ago

cc @stanhu @kwilczynski

kwilczynski commented 3 years ago

Hello everyone!

There is an on-going effort from @stanhu to move the implementation of a custom download to use the mini_portile Ruby Gem, similarly to what Nokogiri uses at the moment, and thus I wonder moving to it would solve the problem with our request lacking things such as User-Agent, etc. I assume that mini_portile might be a little be more behaved (although, I haven't verified yet whether it does set all the things as required).

More at https://github.com/kwilczynski/ruby-magic/pull/5.

Krzysztof

stanhu commented 3 years ago

@kwilczynski I'm now seeing this 412 with mini_portile:

Using mini_portile version 2.5.0
2 retrie(s) left for file-5.39.tar.gz
1 retrie(s) left for file-5.39.tar.gz
0 retrie(s) left for file-5.39.tar.gz
412 Precondition Failed
*** extconf.rb failed ***
brodock commented 3 years ago

I've tried this PR on my local machine and it seems to have worked... but I'm not sure if we just didn't reached the "rate limit" for the new User-Agent or not.

stanhu commented 3 years ago

Note that URI.open uses Ruby as the User-Agent. I suspect there might be rate-limiting by User-Agent.