Problem using an HTTP proxy

[dmrlawson]

Hi, I'm trying to use a proxy with gun (2.0.0-rc.2) but hitting a problem.

I can make a basic request without a proxy:

1> application:ensure_all_started(gun).
{ok,[cowlib,gun]}
2> {ok, ConnPid} = gun:open("api.dmrlawson.co.uk", 80).
{ok,<0.19294.0>}
3> StreamRef = gun:get(ConnPid, "/").
#Ref<0.2469824543.2384199681.110771>
4> Headers = gun:await(ConnPid, StreamRef).
{response,nofin,200,
          [{<<"server">>,<<"nginx/1.18.0 (Ubuntu)">>},
           {<<"date">>,<<"Wed, 07 Sep 2022 14:50:47 GMT">>},
           {<<"content-type">>,<<"application/json">>},
           {<<"content-length">>,<<"16">>},
           {<<"connection">>,<<"keep-alive">>}]}
5> Body = gun:await(ConnPid, StreamRef).
{data,fin,<<"{\"status\":\"ok\"}\n">>}

When I use gun:connect to tunnel through a proxy, I get this (changed to example-http-proxy.com):

1> application:ensure_all_started(gun).
{ok,[cowlib,gun]}
2> {ok, ConnPid} = gun:open("example-http-proxy.com", 3128).
{ok,<0.19294.0>}
3> {ok, http} = gun:await_up(ConnPid).
{ok,http}
4> TunnelStreamRef = gun:connect(ConnPid, #{
4>     host => "api.dmrlawson.co.uk",
4>     port => 80
4> }).
#Ref<0.2583085341.2115764225.99078>
5> {response, fin, 200, _} = gun:await(ConnPid, TunnelStreamRef).
{response,fin,200,[]}
6> StreamRef = gun:get(ConnPid, "/").
#Ref<0.2583085341.2115764225.99096>
7> =ERROR REPORT==== 7-Sep-2022::14:53:43.490370 ===
** State machine <0.19294.0> terminating
** Last event = {cast,{request,<0.19270.0>,
                               #Ref<0.2583085341.2115764225.99096>,<<"GET">>,
                               "/",[],<<>>,infinity}}
** When server state  = {connected,
                            {state,<0.19270.0>,
                                {up,#Ref<0.2583085341.2115764227.54865>},
                                "example-http-proxy.com",3128,<<"http">>,
                                "api.dmrlawson.co.uk",80,
                                [#{host => "example-http-proxy.com",port => 3128,
                                   protocol => http,transport => tcp,
                                   type => connect}],
                                #{},undefined,#Port<0.154>,gun_tcp,true,
                                {tcp,tcp_closed,tcp_error},
                                gun_http,
                                {http_state,#Port<0.154>,gun_tcp,
                                    #{stream_ref =>
                                          #Ref<0.2583085341.2115764225.99078>,
                                      tunnel_transport => tcp},
                                    'HTTP/1.1',keepalive,<<>>,
                                    #Ref<0.2583085341.2115764225.99078>,[],
                                    head,
                                    {0,0},
                                    head},
                                undefined,gun_default_event_h,undefined}}
** Reason for termination = error:badarg
** Callback modules = [gun]
** Callback mode = state_functions
** Stacktrace =
**  [{lists,split,
            [1,#Ref<0.2583085341.2115764225.99096>],
            [{file,"lists.erl"},{line,1571}]},
     {gun,dereference_stream_ref,2,
          [{file,"/home/david/gun_test/_build/default/lib/gun/src/gun.erl"},
           {line,1319}]},
     {gun,connected,3,
          [{file,"/home/david/gun_test/_build/default/lib/gun/src/gun.erl"},
           {line,1263}]},
     {gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1419}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]

=CRASH REPORT==== 7-Sep-2022::14:53:43.490704 ===
  crasher:
    initial call: gun:init/1
    pid: <0.19294.0>
    registered_name: []
    exception error: bad argument
      in function  lists:split/2
         called as lists:split(1,#Ref<0.2583085341.2115764225.99096>)
      in call from gun:dereference_stream_ref/2 (/home/david/gun_test/_build/default/lib/gun/src/gun.erl, line 1319)
      in call from gun:connected/3 (/home/david/gun_test/_build/default/lib/gun/src/gun.erl, line 1263)
      in call from gen_statem:loop_state_callback/11 (gen_statem.erl, line 1419)
    ancestors: [gun_conns_sup,gun_sup,<0.19289.0>]
    message_queue_len: 0
    messages: []
    links: [<0.19291.0>,#Port<0.154>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 6772
    stack_size: 28
    reductions: 14984
  neighbours:

=SUPERVISOR REPORT==== 7-Sep-2022::14:53:43.503955 ===
    supervisor: {local,gun_conns_sup}
    errorContext: child_terminated
    reason: {badarg,[{lists,split,
                            [1,#Ref<0.2583085341.2115764225.99096>],
                            [{file,"lists.erl"},{line,1571}]},
                     {gun,dereference_stream_ref,2,
                          [{file,"/home/david/gun_test/_build/default/lib/gun/src/gun.erl"},
                           {line,1319}]},
                     {gun,connected,3,
                          [{file,"/home/david/gun_test/_build/default/lib/gun/src/gun.erl"},
                           {line,1263}]},
                     {gen_statem,loop_state_callback,11,
                                 [{file,"gen_statem.erl"},{line,1419}]},
                     {proc_lib,init_p_do_apply,3,
                               [{file,"proc_lib.erl"},{line,240}]}]}
    offender: [{pid,<0.19294.0>},
               {id,gun},
               {mfargs,{gun,start_link,undefined}},
               {restart_type,temporary},
               {significant,false},
               {shutdown,5000},
               {child_type,worker}]

I'm unsure as to whether this is a bug or if I'm just using the api incorrectly.

In wireshark I can see that the tunnel gets opened but the GET request is never sent.

[essen]

You need to use the tunnel option in the gun:get call, otherwise Gun can't know this is for a tunnel and will try to send the request to the proxy server itself. This is a change required following the addition of tunneling over HTTP 2 proxies. Try:

StreamRef = gun:get(ConnPid, "/", [], #{tunnel => TunnelStreamRef}).

[dmrlawson]

You need to use the tunnel option in the gun:get call, otherwise Gun can't know this is for a tunnel and will try to send the request to the proxy server itself. This is a change required following the addition of tunneling over HTTP 2 proxies. Try:

StreamRef = gun:get(ConnPid, "/", [], #{tunnel => TunnelStreamRef}).

Ah right, thank you! I can confirm that this works.