REST Websocket proxy doesnt respond ping with pong

kornpow commented 4 years ago

Background

When using the rest websocket proxy, some websocket clients send periodic pings, and expect to receive pongs, or else it will close the connection. It doesn't appear as if LND is responding to those pings, so additional flags need to be used to prevent accidental closes of the websocket.

I am trying to use the REST '/v2/router/htlcevents' endpoint stream.

Your environment

version of lnd "version": "0.10.1-beta commit=clock/v1.0.0-250-ga39c91fbcb9b002c93cae3432771b295ea4a942c-dirty", "commit_hash": "a39c91fbcb9b002c93cae3432771b295ea4a942c",
which operating system (uname -a on *Nix) ubuntu
version of btcd, bitcoind, or other backend bitcoind
any other relevant environment details python3.8 using the websocket package

Steps to produce

This is the stack overflow page that described the result of my issue: https://github.com/miguelgrinberg/python-socketio/issues/310

Code example:

import websockets
import asyncio
import ssl
import logging
logger = logging.getLogger('websockets')
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler())

macaroon = codecs.encode(open(f'{LND_DIR}/data/chain/bitcoin/{CHAIN}/admin.macaroon', 'rb').read(), 'hex').decode()

headers = {'Grpc-Metadata-macaroon': macaroon}

ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ssl_context.load_verify_locations('/home/sako09038/.lnd/tls.cert')

async def htlcevents2():
    ws = await websockets.connect("wss://10.0.0.111:8080/v2/router/htlcevents?method=GET", ping_timeout=None,ping_interval=1, ssl=ssl_context, extra_headers=headers, max_size=1000000000)
    # future = asyncio.run_coroutine_threadsafe(pinger(ws), loop)
    print('waiting')
    await asyncio.sleep(1)
    print('priming')
    await ws.send(json.dumps({}).encode('UTF-8'))
    async for message in ws:
        print('receiving')
        try:
            hi = await asyncio.wait_for(ws.recv(), timeout=5)
            hi = json.loads(hi)
            hi = hi['result']
            await ws.ping()
            pprint(hi)
        except asyncio.TimeoutError:
            print('timeout!')
        except asyncio.CancelledError:
            print("cancelled?")

async def main():
    loop = asyncio.get_event_loop()
    await htlcevents2()

asyncio.run(main())

When I opened my websocket as such I received a "no pong received error":

client > Frame(fin=True, opcode=2, data=b'{}', rsv1=False, rsv2=False, rsv3=False)
client > Frame(fin=True, opcode=9, data=b'j\xd1\xcb\xd8', rsv1=False, rsv2=False, rsv3=False)
client ! timed out waiting for pong
client ! failing OPEN WebSocket connection with code 1011
client - state = CLOSING
client > Frame(fin=True, opcode=8, data=b'\x03\xf3', rsv1=False, rsv2=False, rsv3=False)
client ! timed out waiting for TCP close
client x closing TCP connection
client ! timed out waiting for TCP close
client x aborting TCP connection
client - event = connection_lost(None)
client - state = CLOSED

However, opening up the socket as so, telling it to ignore missing pongs, I am able to keep the socket alive long term:

ws = await websockets.connect("wss://10.0.0.111:8080/v2/router/htlcevents?method=GET", ping_timeout=None,ping_interval=1, ssl=ssl_context, extra_headers=headers, max_size=1000000000)

What do you think? Is it bad that its not responding the pings with pongs? According to the linked stackoverflow the issue is non-compliance with: https://tools.ietf.org/html/rfc6455#section-5.5.3

guggero commented 4 years ago

It is correct that the WebSocket proxy doesn't respond to ping messages. If that means it's not fully compliant with the RFC then that should probably be changed. When testing with the node-js implementation (https://github.com/websockets/ws) this issue never came up as that library doesn't seem to care about ping/pong requests (or at least doesn't require them).

To get your code working: If you use ping_timeout=None,ping_interval=99999, do you still see instability? As long as the client library doesn't expect a response to pings, the connection should remain open and stable.

We'll try to get a fix for the pong responses in, hopefully for v0.11.1.

apotdevin commented 3 years ago

Any plans for this feature to be in some next release? Would be awesome to have since there's currently no way to check if a websocket connection is still open

apotdevin commented 3 years ago

When running LND behind an NGINX proxy unless you add some long timeouts to the nginx config it automatically cancels the websocket connection if it has no activity for a while. This could also be avoided with the ping-pong messages so that you can forcefully keep open the connection even if no messages are being transmitted through the websocket (apart from the ping-pong)

lessless commented 3 years ago

I spent few days debugging exactly the same problem with gRPC https://github.com/ninenines/gun/issues/230#issuecomment-704823789

lightningnetwork / lnd