Closed lessless closed 3 years ago
essen
commented
4 years ago If you're in the disconnect function with this error reason it means the socket was closed by the server.
You can try to install an event handler in Gun to log everything it's doing, or wireshark to see what's happening on the wire, or look at what the server is doing directly.
lessless
commented
4 years ago Here is a Wireshark dump https://www.dropbox.com/s/tc1bd9e4ctdrvvb/subscribe_to_invoices.pcapng?dl=0.
I didn't see any packets with FIN flag during a session until the very end when connection is explicitly closed in code.
essen
commented
4 years ago Sorry I won't have time to look at that for a while. Please investigate on your own. Note that HTTP/2 connections close are indicated by the GOAWAY frame, it doesn't really matter who sends the FIN ultimately.
lessless
commented
4 years ago Sure, that's understnadable. Can you please tip me off from where close can be called so and I will keep digging.
Maybe looking at the calling place will give another insight.
essen
commented
4 years ago It's in the stacktrace
{current_stacktrace,
[{gun_http2,close,4,
[{file,"/Users/lessless/Code/gun/src/gun_http2.erl"},
{line,560}]},
{gun,disconnect,2,
[{file,"/Users/lessless/Code/gun/src/gun.erl"},
{line,1529}]},From where it goes to disconnect that's not clear but it's probably from the Error socket message.
lessless
commented
4 years ago @essen the execution flow goes a bunch of time through [gun:handle_common_connected on line 1299] (https://github.com/ninenines/gun/blob/master/src/gun.erl#L1299) 1.
info, {ssl,
{sslsocket,
{gen_tcp,#Port<0.8>,tls_connection,
undefined},
[<0.280.0>,<0.279.0>]},
<<0,0,4,8,0,0,0,0,0,0,0,0,9,0,0,8,6,0,
0,0,0,0,2,4,16,16,9,14,7,7>>} : connected {state,
<0.271.0>,
{up,
#Ref<0.1560276993.305922050.77247>},
"localhost",
10001,
<<"https">>,
"localhost",
10001,
[],
#{protocols =>
[http2],
retry =>
100,
retry_fun =>
fun 'Elixir.GRPC.Adapter.Gun':retry_fun/2,
tls_opts =>
[{nodelay,
true},
{cacertfile,
<<"/Users/lessless/.polar/networks/1/volumes/lnd/alice/tls.cert">>}],
transport =>
tls},
#Ref<0.1560276993.305922049.77636>,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
true,
{ssl,
ssl_closed,
ssl_error},
gun_http2,
{http2_state,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
[gun_data_h],
<<>>,
connected,
{http2_machine,
client,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
normal,
undefined,
undefined,
#{initial_window_size =>
8000000},
#{},
#{initial_window_size =>
65535},
65526,
8000000,
3,
0,
#{1 =>
{stream,
1,
<<"POST">>,
fin,
65526,
{[],
[]},
0,
undefined,
idle,
8000000,
undefined,
0,
undefined}},
[],
[],
{state,
0,
4096,
4096,
[]},
{state,
949,
4096,
4096,
[{47,
{<<"content-length">>,
<<"9">>}},
{42,
{<<"te">>,
<<"trailers">>}},
{66,
{<<"user-agent">>,
<<"grpc-elixir/0.5.0-beta.1">>}},
{66,
{<<"content-type">>,
<<"application/grpc+proto">>}},
{600,
{<<"macaroon">>,
<<"0201036C6E6402EB01030A10FF36C3159AB2BBFC1576D102BE15870E1201301A160A0761646472657373120472656164120577726974651A130A04696E666F120472656164120577726974651A170A08696E766F69636573120472656164120577726974651A140A086D616361726F6F6E120867656E65726174651A160A076D657373616765120472656164120577726974651A170A086F6666636861696E120472656164120577726974651A160A076F6E636861696E120472656164120577726974651A140A057065657273120472656164120577726974651A180A067369676E6572120867656E657261746512047265616400000620D71A63B319889CD03447CE99A30EF4A7EFC6A8F5413DB26198D3AE452C64C896">>}},
{57,
{<<":authority">>,
<<"localhost:10001">>}},
{71,
{<<":path">>,
<<"/lnrpc.Lightning/SubscribeInvoices">>}}]}},
#{1 =>
{stream,
1,
#Ref<0.1560276993.305922050.77415>,
<0.271.0>,
infinity,
["localhost",
58,
<<"10001">>],
<<"/lnrpc.Lightning/SubscribeInvoices">>,
undefined}},
#{#Ref<0.1560276993.305922050.77415> =>
1},
[]},
gun_default_event_h,
undefined,
undefined}info , keepalive , connected,
{state,
<0.271.0>,
{up,
#Ref<0.1560276993.305922050.77247>},
"localhost",
10001,
<<"https">>,
"localhost",
10001,[],
#{protocols =>
[http2],
retry => 100,
retry_fun =>
fun 'Elixir.GRPC.Adapter.Gun':retry_fun/2,
tls_opts =>
[{nodelay,
true},
{cacertfile,
<<"/Users/lessless/.polar/networks/1/volumes/lnd/alice/tls.cert">>}],
transport =>
tls},
#Ref<0.1560276993.305922049.77636>,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,true,
{ssl,
ssl_closed,
ssl_error},
gun_http2,
{http2_state,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
[gun_data_h],
<<>>,
connected,
{http2_machine,
client,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
normal,
undefined,
undefined,
#{initial_window_size =>
8000000},
#{},
#{initial_window_size =>
65535},
65535,
8000000,3,0,
#{1 =>
{stream,
1,
<<"POST">>,
fin,
65526,
{[],[]},
0,
undefined,
idle,
8000000,
undefined,
0,
undefined}},
[],[],
{state,0,
4096,4096,
[]},
{state,949,
4096,4096,
[{47,
{<<"content-length">>,
<<"9">>}},
{42,
{<<"te">>,
<<"trailers">>}},
{66,
{<<"user-agent">>,
<<"grpc-elixir/0.5.0-beta.1">>}},
{66,
{<<"content-type">>,
<<"application/grpc+proto">>}},
{600,
{<<"macaroon">>,
<<"0201036C6E6402EB01030A10FF36C3159AB2BBFC1576D102BE15870E1201301A160A0761646472657373120472656164120577726974651A130A04696E666F120472656164120577726974651A170A08696E766F69636573120472656164120577726974651A140A086D616361726F6F6E120867656E65726174651A160A076D657373616765120472656164120577726974651A170A086F6666636861696E120472656164120577726974651A160A076F6E636861696E120472656164120577726974651A140A057065657273120472656164120577726974651A180A067369676E6572120867656E657261746512047265616400000620D71A63B319889CD03447CE99A30EF4A7EFC6A8F5413DB26198D3AE452C64C896">>}},
{57,
{<<":authority">>,
<<"localhost:10001">>}},
{71,
{<<":path">>,
<<"/lnrpc.Lightning/SubscribeInvoices">>}}]}},
#{1 =>
{stream,1,
#Ref<0.1560276993.305922050.77415>,
<0.271.0>,
infinity,
["localhost",
58,
<<"10001">>],
<<"/lnrpc.Lightning/SubscribeInvoices">>,
undefined}},
#{#Ref<0.1560276993.305922050.77415> =>
1},
[]},
gun_default_event_h,
undefined,
undefined}3.
info,
{ssl,
{sslsocket,
{gen_tcp,#Port<0.8>,tls_connection,
undefined},
[<0.280.0>,<0.279.0>]},
<<0,0,8,6,1,0,0,0,0,0,0,0,0,0,0,0,0>>} : connected {state,
<0.271.0>,
{up,
#Ref<0.1560276993.305922050.77247>},
"localhost",
10001,
<<"https">>,
"localhost",
10001,
[],
#{protocols =>
[http2],
retry =>
100,
retry_fun =>
fun 'Elixir.GRPC.Adapter.Gun':retry_fun/2,
tls_opts =>
[{nodelay,
true},
{cacertfile,
<<"/Users/lessless/.polar/networks/1/volumes/lnd/alice/tls.cert">>}],
transport =>
tls},
#Ref<0.1560276993.305922049.77658>,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
true,
{ssl,
ssl_closed,
ssl_error},
gun_http2,
{http2_state,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
[gun_data_h],
<<>>,
connected,
{http2_machine,
client,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
normal,
undefined,
undefined,
#{initial_window_size =>
8000000},
#{},
#{initial_window_size =>
65535},
65535,
8000000,
3,
0,
#{1 =>
{stream,
1,
<<"POST">>,
fin,
65526,
{[],
[]},
0,
undefined,
idle,
8000000,
undefined,
0,
undefined}},
[],
[],
{state,
0,
4096,
4096,
[]},
{state,
949,
4096,
4096,
[{47,
{<<"content-length">>,
<<"9">>}},
{42,
{<<"te">>,
<<"trailers">>}},
{66,
{<<"user-agent">>,
<<"grpc-elixir/0.5.0-beta.1">>}},
{66,
{<<"content-type">>,
<<"application/grpc+proto">>}},
{600,
{<<"macaroon">>,
<<"0201036C6E6402EB01030A10FF36C3159AB2BBFC1576D102BE15870E1201301A160A0761646472657373120472656164120577726974651A130A04696E666F120472656164120577726974651A170A08696E766F69636573120472656164120577726974651A140A086D616361726F6F6E120867656E65726174651A160A076D657373616765120472656164120577726974651A170A086F6666636861696E120472656164120577726974651A160A076F6E636861696E120472656164120577726974651A140A057065657273120472656164120577726974651A180A067369676E6572120867656E657261746512047265616400000620D71A63B319889CD03447CE99A30EF4A7EFC6A8F5413DB26198D3AE452C64C896">>}},
{57,
{<<":authority">>,
<<"localhost:10001">>}},
{71,
{<<":path">>,
<<"/lnrpc.Lightning/SubscribeInvoices">>}}]}},
#{1 =>
{stream,
1,
#Ref<0.1560276993.305922050.77415>,
<0.271.0>,
infinity,
["localhost",
58,
<<"10001">>],
<<"/lnrpc.Lightning/SubscribeInvoices">>,
undefined}},
#{#Ref<0.1560276993.305922050.77415> =>
1},
[]},
gun_default_event_h,
undefined,
undefined}4.
info, keepalive, connected,
{state,
<0.271.0>,
{up,
#Ref<0.1560276993.305922050.77247>},
"localhost",
10001,
<<"https">>,
"localhost",
10001,[],
#{protocols =>
[http2],
retry => 100,
retry_fun =>
fun 'Elixir.GRPC.Adapter.Gun':retry_fun/2,
tls_opts =>
[{nodelay,
true},
{cacertfile,
<<"/Users/lessless/.polar/networks/1/volumes/lnd/alice/tls.cert">>}],
transport =>
tls},
#Ref<0.1560276993.305922049.77658>,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,true,
{ssl,
ssl_closed,
ssl_error},
gun_http2,
{http2_state,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
[gun_data_h],
<<>>,
connected,
{http2_machine,
client,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
normal,
undefined,
undefined,
#{initial_window_size =>
8000000},
#{},
#{initial_window_size =>
65535},
65535,
8000000,3,0,
#{1 =>
{stream,
1,
<<"POST">>,
fin,
65526,
{[],[]},
0,
undefined,
idle,
8000000,
undefined,
0,
undefined}},
[],[],
{state,0,
4096,4096,
[]},
{state,949,
4096,4096,
[{47,
{<<"content-length">>,
<<"9">>}},
{42,
{<<"te">>,
<<"trailers">>}},
{66,
{<<"user-agent">>,
<<"grpc-elixir/0.5.0-beta.1">>}},
{66,
{<<"content-type">>,
<<"application/grpc+proto">>}},
{600,
{<<"macaroon">>,
<<"0201036C6E6402EB01030A10FF36C3159AB2BBFC1576D102BE15870E1201301A160A0761646472657373120472656164120577726974651A130A04696E666F120472656164120577726974651A170A08696E766F69636573120472656164120577726974651A140A086D616361726F6F6E120867656E65726174651A160A076D657373616765120472656164120577726974651A170A086F6666636861696E120472656164120577726974651A160A076F6E636861696E120472656164120577726974651A140A057065657273120472656164120577726974651A180A067369676E6572120867656E657261746512047265616400000620D71A63B319889CD03447CE99A30EF4A7EFC6A8F5413DB26198D3AE452C64C896">>}},
{57,
{<<":authority">>,
<<"localhost:10001">>}},
{71,
{<<":path">>,
<<"/lnrpc.Lightning/SubscribeInvoices">>}}]}},
#{1 =>
{stream,1,
#Ref<0.1560276993.305922050.77415>,
<0.271.0>,
infinity,
["localhost",
58,
<<"10001">>],
<<"/lnrpc.Lightning/SubscribeInvoices">>,
undefined}},
#{#Ref<0.1560276993.305922050.77415> =>
1},
[]},
gun_default_event_h,
undefined,
undefined}5.
info,
{ssl,
{sslsocket,
{gen_tcp,#Port<0.8>,tls_connection,
undefined},
[<0.280.0>,<0.279.0>]},
<<0,0,8,6,1,0,0,0,0,0,0,0,0,0,0,0,0>>} : connected {state,
<0.271.0>,
{up,
#Ref<0.1560276993.305922050.77247>},
"localhost",
10001,
<<"https">>,
"localhost",
10001,
[],
#{protocols =>
[http2],
retry =>
100,
retry_fun =>
fun 'Elixir.GRPC.Adapter.Gun':retry_fun/2,
tls_opts =>
[{nodelay,
true},
{cacertfile,
<<"/Users/lessless/.polar/networks/1/volumes/lnd/alice/tls.cert">>}],
transport =>
tls},
#Ref<0.1560276993.305922049.77663>,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
true,
{ssl,
ssl_closed,
ssl_error},
gun_http2,
{http2_state,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
[gun_data_h],
<<>>,
connected,
{http2_machine,
client,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
normal,
undefined,
undefined,
#{initial_window_size =>
8000000},
#{},
#{initial_window_size =>
65535},
65535,
8000000,
3,
0,
#{1 =>
{stream,
1,
<<"POST">>,
fin,
65526,
{[],
[]},
0,
undefined,
idle,
8000000,
undefined,
0,
undefined}},
[],
[],
{state,
0,
4096,
4096,
[]},
{state,
949,
4096,
4096,
[{47,
{<<"content-length">>,
<<"9">>}},
{42,
{<<"te">>,
<<"trailers">>}},
{66,
{<<"user-agent">>,
<<"grpc-elixir/0.5.0-beta.1">>}},
{66,
{<<"content-type">>,
<<"application/grpc+proto">>}},
{600,
{<<"macaroon">>,
<<"0201036C6E6402EB01030A10FF36C3159AB2BBFC1576D102BE15870E1201301A160A0761646472657373120472656164120577726974651A130A04696E666F120472656164120577726974651A170A08696E766F69636573120472656164120577726974651A140A086D616361726F6F6E120867656E65726174651A160A076D657373616765120472656164120577726974651A170A086F6666636861696E120472656164120577726974651A160A076F6E636861696E120472656164120577726974651A140A057065657273120472656164120577726974651A180A067369676E6572120867656E657261746512047265616400000620D71A63B319889CD03447CE99A30EF4A7EFC6A8F5413DB26198D3AE452C64C896">>}},
{57,
{<<":authority">>,
<<"localhost:10001">>}},
{71,
{<<":path">>,
<<"/lnrpc.Lightning/SubscribeInvoices">>}}]}},
#{1 =>
{stream,
1,
#Ref<0.1560276993.305922050.77415>,
<0.271.0>,
infinity,
["localhost",
58,
<<"10001">>],
<<"/lnrpc.Lightning/SubscribeInvoices">>,
undefined}},
#{#Ref<0.1560276993.305922050.77415> =>
1},
[]},
gun_default_event_h,
undefined,
undefined}6.
info , keepalive , connected,
{state,
<0.271.0>,
{up,
#Ref<0.1560276993.305922050.77247>},
"localhost",
10001,
<<"https">>,
"localhost",
10001,[],
#{protocols =>
[http2],
retry => 100,
retry_fun =>
fun 'Elixir.GRPC.Adapter.Gun':retry_fun/2,
tls_opts =>
[{nodelay,
true},
{cacertfile,
<<"/Users/lessless/.polar/networks/1/volumes/lnd/alice/tls.cert">>}],
transport =>
tls},
#Ref<0.1560276993.305922049.77663>,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,true,
{ssl,
ssl_closed,
ssl_error},
gun_http2,
{http2_state,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
[gun_data_h],
<<>>,
connected,
{http2_machine,
client,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
normal,
undefined,
undefined,
#{initial_window_size =>
8000000},
#{},
#{initial_window_size =>
65535},
65535,
8000000,3,0,
#{1 =>
{stream,
1,
<<"POST">>,
fin,
65526,
{[],[]},
0,
undefined,
idle,
8000000,
undefined,
0,
undefined}},
[],[],
{state,0,
4096,4096,
[]},
{state,949,
4096,4096,
[{47,
{<<"content-length">>,
<<"9">>}},
{42,
{<<"te">>,
<<"trailers">>}},
{66,
{<<"user-agent">>,
<<"grpc-elixir/0.5.0-beta.1">>}},
{66,
{<<"content-type">>,
<<"application/grpc+proto">>}},
{600,
{<<"macaroon">>,
<<"0201036C6E6402EB01030A10FF36C3159AB2BBFC1576D102BE15870E1201301A160A0761646472657373120472656164120577726974651A130A04696E666F120472656164120577726974651A170A08696E766F69636573120472656164120577726974651A140A086D616361726F6F6E120867656E65726174651A160A076D657373616765120472656164120577726974651A170A086F6666636861696E120472656164120577726974651A160A076F6E636861696E120472656164120577726974651A140A057065657273120472656164120577726974651A180A067369676E6572120867656E657261746512047265616400000620D71A63B319889CD03447CE99A30EF4A7EFC6A8F5413DB26198D3AE452C64C896">>}},
{57,
{<<":authority">>,
<<"localhost:10001">>}},
{71,
{<<":path">>,
<<"/lnrpc.Lightning/SubscribeInvoices">>}}]}},
#{1 =>
{stream,1,
#Ref<0.1560276993.305922050.77415>,
<0.271.0>,
infinity,
["localhost",
58,
<<"10001">>],
<<"/lnrpc.Lightning/SubscribeInvoices">>,
undefined}},
#{#Ref<0.1560276993.305922050.77415> =>
1},
[]},
gun_default_event_h,
undefined,
undefined}7.
info,
{ssl,
{sslsocket,
{gen_tcp,#Port<0.8>,tls_connection,
undefined},
[<0.280.0>,<0.279.0>]},
<<0,0,8,6,1,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,22,7,0,0,0,0,0,0,0,0,1,0,0,0,11,
116,111,111,95,109,97,110,121,95,
112,105,110,103,115>>} : connected {state,
<0.271.0>,
{up,
#Ref<0.1560276993.305922050.77247>},
"localhost",
10001,
<<"https">>,
"localhost",
10001,
[],
#{protocols =>
[http2],
retry =>
100,
retry_fun =>
fun 'Elixir.GRPC.Adapter.Gun':retry_fun/2,
tls_opts =>
[{nodelay,
true},
{cacertfile,
<<"/Users/lessless/.polar/networks/1/volumes/lnd/alice/tls.cert">>}],
transport =>
tls},
#Ref<0.1560276993.305922049.77665>,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
true,
{ssl,
ssl_closed,
ssl_error},
gun_http2,
{http2_state,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
[gun_data_h],
<<>>,
connected,
{http2_machine,
client,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
normal,
undefined,
undefined,
#{initial_window_size =>
8000000},
#{},
#{initial_window_size =>
65535},
65535,
8000000,
3,
0,
#{1 =>
{stream,
1,
<<"POST">>,
fin,
65526,
{[],
[]},
0,
undefined,
idle,
8000000,
undefined,
0,
undefined}},
[],
[],
{state,
0,
4096,
4096,
[]},
{state,
949,
4096,
4096,
[{47,
{<<"content-length">>,
<<"9">>}},
{42,
{<<"te">>,
<<"trailers">>}},
{66,
{<<"user-agent">>,
<<"grpc-elixir/0.5.0-beta.1">>}},
{66,
{<<"content-type">>,
<<"application/grpc+proto">>}},
{600,
{<<"macaroon">>,
<<"0201036C6E6402EB01030A10FF36C3159AB2BBFC1576D102BE15870E1201301A160A0761646472657373120472656164120577726974651A130A04696E666F120472656164120577726974651A170A08696E766F69636573120472656164120577726974651A140A086D616361726F6F6E120867656E65726174651A160A076D657373616765120472656164120577726974651A170A086F6666636861696E120472656164120577726974651A160A076F6E636861696E120472656164120577726974651A140A057065657273120472656164120577726974651A180A067369676E6572120867656E657261746512047265616400000620D71A63B319889CD03447CE99A30EF4A7EFC6A8F5413DB26198D3AE452C64C896">>}},
{57,
{<<":authority">>,
<<"localhost:10001">>}},
{71,
{<<":path">>,
<<"/lnrpc.Lightning/SubscribeInvoices">>}}]}},
#{1 =>
{stream,
1,
#Ref<0.1560276993.305922050.77415>,
<0.271.0>,
infinity,
["localhost",
58,
<<"10001">>],
<<"/lnrpc.Lightning/SubscribeInvoices">>,
undefined}},
#{#Ref<0.1560276993.305922050.77415> =>
1},
[]},
gun_default_event_h,
undefined,
undefined}8.
info,
{ssl_closed,
{sslsocket,
{gen_tcp,#Port<0.8>,tls_connection,
undefined},
[<0.280.0>,<0.279.0>]}} : closing {state,
<0.271.0>,
{up,
#Ref<0.1560276993.305922050.77247>},
"localhost",
10001,
<<"https">>,
"localhost",
10001,
[],
#{protocols =>
[http2],
retry =>
100,
retry_fun =>
fun 'Elixir.GRPC.Adapter.Gun':retry_fun/2,
tls_opts =>
[{nodelay,
true},
{cacertfile,
<<"/Users/lessless/.polar/networks/1/volumes/lnd/alice/tls.cert">>}],
transport =>
tls},
undefined,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
true,
{ssl,
ssl_closed,
ssl_error},
gun_http2,
{http2_state,
{sslsocket,
{gen_tcp,
#Port<0.8>,
tls_connection,
undefined},
[<0.280.0>,
<0.279.0>]},
gun_tls,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
[gun_data_h],
<<>>,
closing,
{http2_machine,
client,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
normal,
undefined,
undefined,
#{initial_window_size =>
8000000},
#{},
#{initial_window_size =>
65535},
65535,
8000000,
3,
0,
#{1 =>
{stream,
1,
<<"POST">>,
fin,
65526,
{[],
[]},
0,
undefined,
idle,
8000000,
undefined,
0,
undefined}},
[],
[],
{state,
0,
4096,
4096,
[]},
{state,
949,
4096,
4096,
[{47,
{<<"content-length">>,
<<"9">>}},
{42,
{<<"te">>,
<<"trailers">>}},
{66,
{<<"user-agent">>,
<<"grpc-elixir/0.5.0-beta.1">>}},
{66,
{<<"content-type">>,
<<"application/grpc+proto">>}},
{600,
{<<"macaroon">>,
<<"0201036C6E6402EB01030A10FF36C3159AB2BBFC1576D102BE15870E1201301A160A0761646472657373120472656164120577726974651A130A04696E666F120472656164120577726974651A170A08696E766F69636573120472656164120577726974651A140A086D616361726F6F6E120867656E65726174651A160A076D657373616765120472656164120577726974651A170A086F6666636861696E120472656164120577726974651A160A076F6E636861696E120472656164120577726974651A140A057065657273120472656164120577726974651A180A067369676E6572120867656E657261746512047265616400000620D71A63B319889CD03447CE99A30EF4A7EFC6A8F5413DB26198D3AE452C64C896">>}},
{57,
{<<":authority">>,
<<"localhost:10001">>}},
{71,
{<<":path">>,
<<"/lnrpc.Lightning/SubscribeInvoices">>}}]}},
#{1 =>
{stream,
1,
#Ref<0.1560276993.305922050.77415>,
<0.271.0>,
infinity,
["localhost",
58,
<<"10001">>],
<<"/lnrpc.Lightning/SubscribeInvoices">>,
undefined}},
#{#Ref<0.1560276993.305922050.77415> =>
1},
[]},
gun_default_event_h,
undefined,
undefined}and then it moves into gun:handle_common_connected_no_input/4:
{state,<0.271.0>,
{up,
#Ref<0.1560276993.305922050.77247>},
"localhost",10001,<<"https">>,
"localhost",10001,[],
#{protocols => [http2],
retry => 100,
retry_fun =>
fun 'Elixir.GRPC.Adapter.Gun':retry_fun/2,
tls_opts =>
[{nodelay,true},
{cacertfile,
<<"/Users/lessless/.polar/networks/1/volumes/lnd/alice/tls.cert">>}],
transport => tls},
undefined,
{sslsocket,
{gen_tcp,#Port<0.8>,tls_connection,
undefined},
[<0.280.0>,<0.279.0>]},
gun_tls,true,
{ssl,ssl_closed,ssl_error},
gun_http2,
{http2_state,
{sslsocket,
{gen_tcp,#Port<0.8>,
tls_connection,undefined},
[<0.280.0>,<0.279.0>]},
gun_tls,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
[gun_data_h],
<<>>,closing,
{http2_machine,client,
#{initial_connection_window_size =>
8000000,
initial_stream_window_size =>
8000000},
normal,undefined,undefined,
#{initial_window_size => 8000000},
#{},
#{initial_window_size => 65535},
65535,8000000,3,0,
#{1 =>
{stream,1,<<"POST">>,fin,65526,
{[],[]},
0,undefined,idle,8000000,
undefined,0,undefined}},
[],[],
{state,0,4096,4096,[]},
{state,949,4096,4096,
[{47,
{<<"content-length">>,<<"9">>}},
{42,{<<"te">>,<<"trailers">>}},
{66,
{<<"user-agent">>,
<<"grpc-elixir/0.5.0-beta.1">>}},
{66,
{<<"content-type">>,
<<"application/grpc+proto">>}},
{600,
{<<"macaroon">>,
<<"0201036C6E6402EB01030A10FF36C3159AB2BBFC1576D102BE15870E1201301A160A0761646472657373120472656164120577726974651A130A04696E666F120472656164120577726974651A170A08696E766F69636573120472656164120577726974651A140A086D616361726F6F6E120867656E65726174651A160A076D657373616765120472656164120577726974651A170A086F6666636861696E120472656164120577726974651A160A076F6E636861696E120472656164120577726974651A140A057065657273120472656164120577726974651A180A067369676E6572120867656E657261746512047265616400000620D71A63B319889CD03447CE99A30EF4A7EFC6A8F5413DB26198D3AE452C64C896">>}},
{57,
{<<":authority">>,
<<"localhost:10001">>}},
{71,
{<<":path">>,
<<"/lnrpc.Lightning/SubscribeInvoices">>}}]}},
#{1 =>
{stream,1,
#Ref<0.1560276993.305922050.77415>,
<0.271.0>,infinity,
["localhost",58,<<"10001">>],
<<"/lnrpc.Lightning/SubscribeInvoices">>,
undefined}},
#{#Ref<0.1560276993.305922050.77415> =>
1},
[]},
gun_default_event_h,undefined,
undefined}At step 7 it gets into closing which is the graceful shutdown of the connection either initiated by you or the server via the GOAWAY frame.
lessless
commented
4 years ago you were right - once I stuck io:format("DebugData: ~p ~n", [DebugData]) in https://github.com/ninenines/cowlib/blob/master/src/cow_http2.erl#L262 I could saw <<"too_many_pings">>.
Is this a client or a server problem?
essen
commented
4 years ago Both. In Gun you can control this via the keepalive protocol option. Server probably has one too.
Please leave this ticket open I want to double check why you get an undef error, doesn't sound right, and perhaps increment the default keepalive value if it causes problems. This is with Nginx right? If you could give me what configuration it has for this it'd help.
lessless
commented
4 years ago Nope, this is a direct connection with between elixir-grpc/grpc and https://github.com/lightningnetwork/lnd/ (I think they're using https://pkg.go.dev/google.golang.org/grpc?tab=doc)
You can find instructions on how to spin up the server in steps 1-3 of https://github.com/elixir-grpc/grpc/issues/155
Then you can:
mix.exs to point to local versions of gun and cowlib so it will be easier to debug: defp deps do
[
{:grpc, github: "elixir-grpc/grpc"},
{:gun, path: "../gun", override: true},
{:cowlib, path: "../cowlib", override: true},
]
endUnistream.working_subscriptionEverytime I make changes to gun or cowlib I just relaunch shell and those chagnges are picked up.
essen
commented
4 years ago Not sure you understood. Too many pings is a standard HTTP/2 error. It's probably the server you linked that sent it. It's most likely configurable as well, or there's a hardcoded value. Either way it's not a bug per se. You can configure keepalive as mentioned above to avoid the problem.
lessless
commented
4 years ago oh yeah. I set keepalive to one hour and it connection didn't go down yet :)
thanks a lot!
essen
commented
4 years ago I'm leaning toward disabling the ping by default.
lessless
commented
4 years ago Connection was stable for last 20 hours. I think I can move on with those settings.
Last question - is it possible to disable ping in current version (I wasn't able to find typespec for timeout())?
essen
commented
4 years ago It's a standard type that includes infinity.
essen
commented
3 years ago I've disabled the ping in all protocols. It was already disabled in HTTP/1.1 (non-standard), and for Websocket it's a new feature so I'm just being conservative. Closing, thanks!
Hi,
I'm trying to drill down into root cause of https://github.com/elixir-grpc/grpc/issues/155 and since I'm not an Erlang expert I decided to post my finding here.
So, under conditions described in https://github.com/elixir-grpc/grpc/issues/155
gun_http2:closeis called fromgun:disconnect:I managed to cause a crash (hehe) in
gun_http2:closeand got following message:Then I saw that protocol stream wasn't fully printed and managed to print it out from
gun:disconnectdoes this shed any light on the nature of that
{stream_error, closed}error?