search

socketry / async-http

MIT License
298 stars 45 forks source link

Use with an http proxy? #25

Closed rdubya closed 4 years ago

rdubya commented 4 years ago

We currently use this gem in an environment where we have outgoing http requests blocked unless they go through a proxy. Does this gem support proxies? It looks like it probably doesn't since it is handling things at the tcp level, but am hoping that maybe I missed something. Would it be possible to make it work with an http proxy?

ioquatix commented 4 years ago

Can you explain what you mean by work with proxies? You mean, work with HTTP proxies using CONNECT?

rdubya commented 4 years ago

Thanks for the quick response.

I don't know enough about lower level connections to be able to know the answer to that question unfortunately. In other gems that we use, we can set proxy information. For example, we use this for HTTParty: http_proxy(Settings.http_proxy.address, Settings.http_proxy.port). In the HTTPClient gem you create the new client with the proxy setting: 

  # HTTPClient.new takes optional arguments as a Hash.
  #  * :proxy - proxy url string
  #  * :agent_name - User-Agent String
  #  * :from - from header String
  #  * :base_url - base URL of resources
  #  * :default_header - header Hash all HTTP requests should have
  #  * :force_basic_auth - flag for sending Authorization header w/o gettin 401 first
  # User-Agent and From are embedded in HTTP request Header if given.
  # From header is not set without setting it explicitly.

I haven't found a way to do something similar with this gem. Unfortunately, the cloudflare gem uses this gem so it means we can't use the cloudflare gem unless we can figure out a way to get these to go through the proxy.

ioquatix commented 4 years ago

Ah I see. I am happy to implement this feature.

In order to prioritise this work, would you be wiling to sponsor the project to support this development? I'm actually still working out all the details of this sponsorship process, but let me know if this is something you be willing to explore.

rdubya commented 4 years ago

I can run it by my boss. Our alternative at this point is to build a micro-service outside our locked environment that wraps this gem and send all our calls through that. We'd need the feature in the next week or so to be able to avoid doing that.

ioquatix commented 4 years ago

Based on my current schedule, I could deliver this by the 4th of August.

ioquatix commented 4 years ago

@rdubya I have a working implementation but I am just sorting out some bugs. Hopefully I can give you access to an experimental branch today.

ioquatix commented 4 years ago

The way it works is also completely generic - I know that's probably less important to you, but basically:

# The proxy HTTP server:
client = Async::HTTP::Client.new(Async::HTTP::Endpoint.parse("https://proxy.local"))
# Represents an endpoint that can connect using TCP to the given host/port using CONNECT:
proxy = Async::HTTP::Proxy.new(client, "www.google.com:443")
# An HTTP endpoint that will connect via the proxy:
endpoint = proxy.endpoint("https://www.google.com")

# An HTTP client that will connect to the given endpoint via the proxy:
proxy_client = Async::HTTP::Client.new(endpoint)

# Make a normal request, it will go through the proxy:
response = proxy_client.get("/search?q=kittens")
puts response.read
ioquatix commented 4 years ago

(Of course the final interface would be much simpler).

rdubya commented 4 years ago

Cool, sounds good. Will we also need to make changes to the cloudflare gem to support it or will it be done in a way that we can specify the proxy server's info and this gem will be able to pick it up without any changes to the cloudflare gem?

ioquatix commented 4 years ago

It should be possible to use these changes without changing the cloud flare gem, but we can check that.

ioquatix commented 4 years ago

There was a bug in the proxy implementation, but I found it. It took a few days. Hopefully have something early this week.

ioquatix commented 4 years ago

@rdubya Can you please tell me what your proxy server software is?

rdubya commented 4 years ago

Glad to see you got through it. We use squid as our proxy.

ioquatix commented 4 years ago

Okay, all specs are passing:

Async::HTTP::Protocol::HTTP10
  behaves like Async::HTTP::Proxy
    echo server
      can connect to remote system using block
      can connect to remote system
    proxied client
      can get website

Async::HTTP::Protocol::HTTP11
  behaves like Async::HTTP::Proxy
    echo server
      can connect to remote system using block
      can connect to remote system
    proxied client
      can get website

Async::HTTP::Protocol::HTTP2
  behaves like Async::HTTP::Proxy
    echo server
      can connect to remote system using block
      can connect to remote system
    proxied client
      can get website

Finished in 1.7 seconds (files took 0.36588 seconds to load)
9 examples, 0 failures

This is a test that covers client/server HTTP proxy for HTTP/1.0, HTTP/1.1 and HTTP/2.0

I'm assuming squid only supports HTTP/1?

I'd like to test it on travis with a real proxy.

ioquatix commented 4 years ago

Also I should test Cloudflare integration. It should be straight forward.

rdubya commented 4 years ago

I think you are correct, based on this: https://wiki.squid-cache.org/Features/HTTP2 I don't think it supports it yet. Glad to see you are making progress!

ioquatix commented 4 years ago

I have started releasing all the bits required for this. The last part required is some kind of spec for Cloudflare, and that requires some adjustments to async-rest.

ioquatix commented 4 years ago

Everything seems to be working, I've just run the first set of tests. I modified the Cloudflare test suite to use a proxy and ran a local instance of squid. I want to do a few more checks but I'll try to get that all released today.

ioquatix commented 4 years ago

All green: https://travis-ci.org/socketry/cloudflare/builds/571600394

Please check the implementation here:

https://github.com/socketry/cloudflare/blob/54514461cf13b30f5a6726a0ac22e1e02f6b08da/lib/cloudflare/rspec/connection.rb#L41-L44

Make sure you close the connection after you are done with it:

https://github.com/socketry/cloudflare/blob/54514461cf13b30f5a6726a0ac22e1e02f6b08da/lib/cloudflare/rspec/connection.rb#L52-L53

Please open new issue if you run into problems.

rdubya commented 4 years ago

Great. I'll take a look at it. We're unfortunately battling datacenter issues right now, but I'll get back to this as soon as I can!

rdubya commented 4 years ago

Hey @ioquatix, Sorry for the delay, I finally got back to digging into this. I'm not sure if this is jruby specific or if it is something else, but I get these errors when I try to use the cloudflare gem with a proxy:

40.67s    error: <Async::Task:0x80e pipe writer failed> [pid=36119] [2019-10-04 08:59:40 -0400]
               |   ArgumentError: mode not supported for this object: r
               |   → org/nio4r/Selector.java:124 in `register'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/reactor.rb:129 in `register'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/wrapper.rb:222 in `wait_for'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/wrapper.rb:122 in `wait_readable'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/generic.rb:218 in `async_send'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/generic.rb:67 in `block in read_nonblock'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/stream.rb:242 in `fill_read_buffer'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/stream.rb:87 in `read_partial'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-http-0.48.2/lib/async/http/body/pipe.rb:88 in `writer'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/task.rb:248 in `block in make_fiber'
40.81s    error: <Async::Task:0x810 pipe writer failed> [pid=36119] [2019-10-04 08:59:40 -0400]
               |   ArgumentError: mode not supported for this object: r
               |   → org/nio4r/Selector.java:124 in `register'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/reactor.rb:129 in `register'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/wrapper.rb:222 in `wait_for'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/wrapper.rb:122 in `wait_readable'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/generic.rb:218 in `async_send'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/generic.rb:67 in `block in read_nonblock'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/stream.rb:242 in `fill_read_buffer'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/stream.rb:87 in `read_partial'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-http-0.48.2/lib/async/http/body/pipe.rb:88 in `writer'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/task.rb:248 in `block in make_fiber'
40.87s    error: <Async::Task:0x812 pipe writer failed> [pid=36119] [2019-10-04 08:59:40 -0400]
               |   ArgumentError: mode not supported for this object: r
               |   → org/nio4r/Selector.java:124 in `register'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/reactor.rb:129 in `register'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/wrapper.rb:222 in `wait_for'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/wrapper.rb:122 in `wait_readable'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/generic.rb:218 in `async_send'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/generic.rb:67 in `block in read_nonblock'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/stream.rb:242 in `fill_read_buffer'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-io-1.25.0/lib/async/io/stream.rb:87 in `read_partial'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-http-0.48.2/lib/async/http/body/pipe.rb:88 in `writer'
               |     /Users/robert.widmer/.rvm/gems/jruby-9.2.8.0/gems/async-1.21.0/lib/async/task.rb:248 in `block in make_fiber'

It appears that it is making a connection to the proxy (from the squid access logs):

1570193980.519    305 127.0.0.1 TCP_TUNNEL/200 39 CONNECT api.cloudflare.com:443 - HIER_DIRECT/2606:4700::6810:1f5a -
1570193980.581     53 127.0.0.1 TCP_TUNNEL/200 39 CONNECT api.cloudflare.com:443 - HIER_DIRECT/2606:4700::6810:1f5a -
1570193980.657     68 127.0.0.1 TCP_TUNNEL/200 39 CONNECT api.cloudflare.com:443 - HIER_DIRECT/2606:4700::6810:1f5a -

This is how I am configuring the cloudflare connection (simplified version):

endpoint = Async::HTTP::Endpoint.new(URI::HTTP.build(Settings.http_proxy.to_h))
client = Async::HTTP::Client.new(endpoint).proxied_endpoint(::Cloudflare::DEFAULT_ENDPOINT)
Cloudflare.connect(endpoint, Settings.cloudflare.credentials, &block)

Any thoughts on things that could potentially be misconfigured?

ioquatix commented 4 years ago

Just a heads up:

The Protocol::HTTP::Header::Authorization header has changed the interface slightly.

It's now

headers.add('authorization', Protocol::HTTP::Header::Authorization.basic(username, password))

@rdubya please open a new issue if you are having specific problems. JRuby has interface compatibility issues, which I believe they are working on. But if you have a short repro, we can pull the JRuby developers in and try to get a fix.