search

writer / writer-framework

No-code in the front, Python in the back. An open-source framework for creating data apps.
https://dev.writer.com/framework/introduction
Apache License 2.0
1.33k stars 76 forks source link

Service large amount of data results in timeout and app non response #140

Open thondeboer opened 1 year ago

thondeboer commented 1 year ago

I am trying to serve quite a bit of information but get timeouts as shown below and app becomes unresponsive

I tried to make some of the data invisible, since it is not needed until user wants to see it, but that does not help.

Any ideas on how to avoid this timeout? Or can we extend the teimeout somewhere in the settings>

Task exception was never retrieved
future: <Task finished name='Task-108' coro=<get_asgi_app.<locals>._handle_state_enquiry_message() done, defined at /home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/streamsync/serve.py:265> exception=ConnectionClosedError(None, Close(code=1011, reason='keepalive ping timeout'), None)>
Traceback (most recent call last):
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 979, in transfer_data
    await asyncio.shield(self._put_message_waiter)
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/streamsync/serve.py", line 279, in _handle_state_enquiry_message
    await websocket.send_json(response.model_dump())
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/starlette/websockets.py", line 173, in send_json
    await self.send({"type": "websocket.send", "text": text})
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/starlette/websockets.py", line 85, in send
    await self._send(message)
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in sender
    await send(message)
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 320, in asgi_send
    await self.send(data)  # type: ignore[arg-type]
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 635, in send
    await self.ensure_open()
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 944, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: sent 1011 (unexpected error) keepalive ping timeout; no close frame received
Task exception was never retrieved
future: <Task finished name='Task-185' coro=<get_asgi_app.<locals>._handle_state_enquiry_message() done, defined at /home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/streamsync/serve.py:265> exception=ConnectionClosedError(None, Close(code=1011, reason='keepalive ping timeout'), None)>
Traceback (most recent call last):
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 979, in transfer_data
    await asyncio.shield(self._put_message_waiter)
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/streamsync/serve.py", line 279, in _handle_state_enquiry_message
    await websocket.send_json(response.model_dump())
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/starlette/websockets.py", line 173, in send_json
    await self.send({"type": "websocket.send", "text": text})
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/starlette/websockets.py", line 85, in send
    await self._send(message)
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in sender
    await send(message)
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 320, in asgi_send
    await self.send(data)  # type: ignore[arg-type]
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 635, in send
    await self.ensure_open()
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 944, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: sent 1011 (unexpected error) keepalive ping timeout; no close frame received
Task exception was never retrieved
future: <Task finished name='Task-109' coro=<get_asgi_app.<locals>._handle_state_enquiry_message() done, defined at /home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/streamsync/serve.py:265> exception=ConnectionClosedError(None, Close(code=1011, reason='keepalive ping timeout'), None)>
Traceback (most recent call last):
  File "/home/tdeboer/.cache/pypoetry/virtualenvs/rnayx-eqgCBscA-py3.10/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 979, in transfer_data
    await asyncio.shield(self._put_message_waiter)
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:
.
.
.
thondeboer commented 1 year ago

I guess what we really need is server-based paging of large dataframes like we can do in Dash. Could implement ourserlves I guessif we can catch some of the events, like sort on a dataframe

FabienArcellier commented 1 year ago

I don't know pretty well the architecture. I think the websocket link timeout is 20 seconds on uvicorn. This setting is not editable in the current implementation.

You may try temporarily to modify the line 404 of streamsync.serve:server in your venv to see how it impact your situation and add ws_ping_timeout ?

uvicorn.run(asgi_app, host=host,
                port=port, log_level=log_level, ws_max_size=MAX_WEBSOCKET_MESSAGE_SIZE, ws_ping_timeout=600)

Could you share more details about the volume your application is trying to fetch using the browser inspector ?

image

FabienArcellier commented 1 year ago

Did you try to bind visibility on a variable in the state and keep the state empty till the visibility variable is triggerd on the backend ?

def handle_timer_tick(state: StreamsyncState):
    if state['tick_visible'] is False:
        return

    df = state["random_df"]
    for i in range(5):
        df[f'pgcf_{i+1}'] = np.around(np.random.rand(10, 1), decimals=9)
    state["random_df"] = df

def trigger_auto_refresh(state: StreamsyncState):
    state["tick_visible"] = not state["tick_visible"]
thondeboer commented 1 year ago

Yeah, I'll consider doing this to create the data until requested. Will also look at the messages as you suggested above

FabienArcellier commented 9 months ago

I have described a design to chunk frame inside the websocket payload. It will ensure we keep the frame below a size and highly reduce the risk of timeout : https://github.com/FabienArcellier/streamsync/issues/12

It involves adding an encoder / decoder on backend / frontend to rebuild the frame.