WS + Proxy + SSL

[vboikotriller]

Hello What is the correct use of gun:connect ? The following code raises connection timeout error

            case gun:open(ProxyHost, ProxyPort, #{transport => tls, protocols => [http]}) of
                {ok, ConnPid} ->
                    case gun:await_up(ConnPid) of
                        {ok, http} ->
                            _StreamRef = gun:connect(ConnPid, #{
                                host => Host,
                                port => Port
                            }),
                            StreamRef = gun:ws_upgrade(ConnPid, ?API_ENDPOINT, [
                                {?SESSION_HEADER, SecretToken},
                                {?DEVICE_ID_HEADER, DeviceId},
                                {?NAMESPACE, Namespace}]),
                            case wait_for_ws_upgrade(ConnPid, StreamRef, 5000) of
                                {error, Reason} ->
                                    gun:close(ConnPid),
                                    {error, Reason};
                                _ -> {ok, ConnPid, StreamRef}
                            end;
                        {error, Reason} ->
                            gun:close(ConnPid),
                            {error, Reason}
                    end;
                {error, Reason} -> {error, Reason}
            end;

[essen]

Lots of examples here: https://github.com/ninenines/gun/blob/master/test/tunnel_SUITE.erl

[vboikotriller]

I checked examples and I found they can't work with any proxy. For example, I tried tinyproxy and it returns HTTP/1.0, which gun do not seem to support. I tried squid, but I receive reply to CONNECT command instead of reply to URL request.

I have not found a working configuration yet. Any hints ? Current code:

wait_for_ws_upgrade(ConnPid, StreamRef, Timeout) ->
    io:format("waiting for WS upgrade~n"),
    receive
        Res1 ->
            io:format("received message: ~p~n", [Res1]),
            case Res1 of
                {gun_upgrade, ConnPid, StreamRef, [<<"websocket">>], _} ->
                    ResponseHeader = binary:encode_unsigned(?PT_INFO_REQUEST),
                    Packet = << 0, ResponseHeader/binary, <<0:(19 * 8)>>/binary, 1, 0, 0 >>,
                    gun:ws_send(ConnPid, StreamRef, {binary, Packet});
                {gun_response, ConnPid, _, _, Status, _Headers} ->
                    {error, io_lib:format("Status: ~p", [Status])};
                {gun_error, ConnPid, _StreamRef, Reason} ->
                    {error, io_lib:format("gun_error: ~p", [Reason])}
            end
    after Timeout -> {error, timeout}
    end.

tst() ->
    application:ensure_all_started(gun),
    Host = "something.something",
    Port = 443,
    ProxyHost = "127.0.0.1",
    ProxyPort = 3128,

    case inet:getaddr(Host, inet) of
        {ok, _IPv4} ->
            %% Initiate SSL connection
            case gun:open(ProxyHost, ProxyPort, #{transport => tcp, protocols => [http]}) of
                {ok, ConnPid} ->
                    io:format("connected to proxy~n"),
                    %% now creating connection through the proxy to origin host
                    StreamRef1 = gun:connect(ConnPid, #{
                                                        host => Host,
                                                        port => Port,
                                                        transport => tls,
                                                        protocols => [http]
                                                       }),
                    io:format("connected to host~n"),
                    case gun:await_up(ConnPid) of
                        {ok, http} ->
                            io:format("up!~n"),
                            timer:sleep(300), %% attempt to receive reply to CONNECT statement
                            StreamRef2 = gun:ws_upgrade(ConnPid, ?API_ENDPOINT,
                                                        [{?SESSION_HEADER, SecretToken}]),
                            io:format("WS upgrade sent~n"),
                            case wait_for_ws_upgrade(ConnPid, StreamRef2, 5000) of
                                {error, Reason} ->
                                    timer:sleep(10000),
                                    gun:close(ConnPid),
                                    {error, Reason};
                                _ -> {ok, ConnPid, StreamRef2}
                            end;
                        {error, Reason} ->
                            gun:close(ConnPid),
                            {error, Reason}
                    end;
                {error, Reason} -> {error, Reason}
            end
    end.

[essen]

At this point I have implemented the HTTP/1.1 and HTTP/2 specs. I have not tested against specific proxies although I did try against random proxies found online (proxy lists).

That said, I can see on your code that you do not provide the tunnel option. It may or may not make a difference for tinyproxy.

It sounds like squid works though, after you connect you will receive both an up and the response. After that you can use the tunnel. You are not receiving the CONNECT response currently (well, you do when you wait for the Websocket one).

[vboikotriller]

I checked this file and gun source code, but I have not found how to use tunnel option, as when I use gun:connect, I do not have StreamRef yet.

gun:open(ProxyHost, ProxyPort, #{transport => tls, protocols => [http]})
gun:await_up(ConnPid)
StreamRef = gun:connect(ConnPid, #{ host => Host, port => Port })
gun:ws_upgrade(ConnPid, ..)

[essen]

Tunnel option is given to ws_upgrade.

[vboikotriller]

ws_upgrade do not accept tunnel in options. gun:ws_upgrade(ConnPid, "/api", [], >#{tunnel => StreamRef}). Error:

                    {error,
                        {options,{ws,{tunnel,#Ref<0.2968715942.236191745.144545>}}}}
     in function  gun:ws_upgrade/4 (src/gun.erl, line 629)

The documentation says: Gun does not currently support Websocket over HTTP/2. This means websocket over SSL is not supported ?

We tried the following

{ok, ConnPid} = gun:open(ProxyHost, ProxyPort, #{transport => tcp, protocols => [http]}),
{ok, http} = gun:await_up(ConnPid),
StreamRef1 = gun:connect(ConnPid, #{host => Host, port => Port, transport => tls, protocols => [http]}),
{response, fin, 200, _} = gun:await(ConnPid, StreamRef1),
StreamRef2 = gun:ws_upgrade(ConnPid, ..)

The connection with the remote server seem to be established, but then gun raises the following error.

=SUPERVISOR REPORT==== 13-May-2021::11:45:42.010051 ===
    supervisor: {local,gun_sup}
    errorContext: child_terminated
    reason: {badarg,[{lists,split,
                            [1,#Ref<0.1124111550.4281073665.179730>],
                            [{file,"lists.erl"},{line,1447}]},
                     {gun,dereference_stream_ref,2,
                          [{file,"src/gun.erl"},{line,1316}]},
                     {gun,connected,3,[{file,"src/gun.erl"},{line,1290}]},
                     {gen_statem,loop_state_callback,11,
                                 [{file,"gen_statem.erl"},{line,1118}]},
                     {proc_lib,init_p_do_apply,3,
                               [{file,"proc_lib.erl"},{line,249}]}]}
    offender: [{pid,<0.116.0>},
               {id,gun},
               {mfargs,{gun,start_link,undefined}},
               {restart_type,temporary},
               {shutdown,5000},
               {child_type,worker}]

It looks like gun expects list, but it receives one stream instead. If we change src/gun.erl, from

dereference_stream_ref(StreamRef, #state{intermediaries=Intermediaries}) -> to dereference_stream_ref(StreamRef, _State) when is_reference(StreamRef) -> Then it kind of works, but then we start receiving not one StreamReaf, but several ones in gun_upgrade, gun_response.

[essen]

It does, see https://github.com/ninenines/gun/blob/master/src/gun.erl#L927-L929

It appears that you are trying to do this using Gun 1.3 though. It did not have the tunnel option yet I think. I am not sure it works well with the option. It was not needed for the few cases that Gun 1.3 worked with.

Anyway I've already given you the solution here:

It sounds like squid works though, after you connect you will receive both an up and the response. After that you can use the tunnel. You are not receiving the CONNECT response currently (well, you do when you wait for the Websocket one).

You need to receive the CONNECT response before you do the Websocket upgrade.