grosser / parallel

Ruby: parallel processing made simple and fast
MIT License
4.16k stars 254 forks source link

Net::HTTP open_timeout doesn't work with `in_threads` but does work with `in_processes` #276

Closed owst closed 4 years ago

owst commented 4 years ago

It appears that setting open_timeout doesn't work if I use Parallel.map with in_threads vs in_processes with a Net::HTTP request. This came about from me wanting to check the health of various services in parallel (on their private IP addresses), but I'd forgotten to connect my VPN, causing the connection time outs.

The following is a standalone reproduction - my default gateway (where 10.0.0.1 is routed) seems to just drop the packets:

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'parallel', '= 1.19.1'
end

require 'net/http'

def go(type, i)
  start = Time.now

  net = Net::HTTP.new("www.google.com", 80 + i)
  net.open_timeout = 1
  net.get('/')
rescue => e
  puts "#{e.class.name}: #{type} for #{net.inspect} in #{Time.now - start}"
end

Parallel.map([1], in_processes: 2) do |i|
  go("in_processes", i)
end

Parallel.map([1], in_threads: 2) do |i|
  go("in_threads", i)
end

When I run this, I see:

Net::OpenTimeout: in_processes for #<Net::HTTP 10.0.0.1:3000 open=false> in 1.007244
Errno::ETIMEDOUT: in_threads for #<Net::HTTP 10.0.0.1:3000 open=false> in 75.128787

What causes the difference in Exception raised, and why does the in_threads version timeout in ~75 seconds vs ~1 second as was set as the open-timeout?

grosser commented 4 years ago

Yeah can confirm when using www.google.com:81 also it works correctly when doing:

Thread.new do
  go("in_threads", 1)
end.join

which should be the same as the parallel call. ... also tried with .each to simplify further but no luck

reduced it to this:

require 'timeout'

Parallel.each([1], in_threads: 1) do |i|
  begin
    Timeout.timeout(0.1) { sleep 0.2 }
  rescue Timeout::Error
    puts "OK"
  else
    puts "BROKEN"
  end
end

problem was introduced here https://github.com/grosser/parallel/pull/251

grosser commented 4 years ago

1.19.2