Web Socket for Web UI 'Positions' disconnecting

laisee commented 8 months ago

Web Socket connections dropping for 'Positions' after web app open for some time. See screenshots in attached doc. The disconnections happen after about 15-20 mins in Web UI.

power-trade-issues-websockets-disconnection.pdf

laisee commented 8 months ago

comments on this one: "... a new positions api and there’s a cache issue on the server side that publishes positions. .. A fix is in test."

klugjo commented 8 months ago

Thanks @laisee having a look

kernelbee commented 8 months ago

@laisee : it can be when we re-deploy API backend... please provide env (url) and time of occurance (next time maybe)

klugjo commented 8 months ago

@laisee the GUI will try to reconnect periodically (with randomized, increasing time intervals as detailed here: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/)

Unless we have an outage, a web app refresh should solve this

laisee commented 8 months ago

@klugjo Yes, a full refresh of the web page fixes the issue. I'm not seeing any behavior to re-connect at increasing intervals. Will test again and share details on date/time, browser, OS, etc. Until now I am using brave on OSX/M2.

laisee commented 8 months ago

From M: "We just moved to a new positions api and there’s a cache issue on the server side that publishes positions. This may be delayed. A fix is in test"

laisee commented 7 months ago

Any updates on this? I can still see 'Positions' websocket seems to fail on web UI if the user does nothing for 10 mins or longer. The current code doesn't seem to be re-trying or re-connecting for the disconnected socket.

klugjo commented 7 months ago

Fixed in TEST environment and testing, will let you know when deployed to Production. Thanks

klugjo commented 7 months ago

Hi @laisee the new web app version is available in PROD, can you confirm this is fixed ?

laisee commented 7 months ago

Will give it a try now - saw the notification about new version earlier today

On Wed, 7 Feb 2024 at 15:23, Klughertz Jonathan @.***> wrote:

Hi @laisee https://github.com/laisee the new web app version is available in PROD, can you confirm this is fixed ?

— Reply to this email directly, view it on GitHub https://github.com/Power-Trade/api-docs/issues/8#issuecomment-1931429418, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNBV2R4XGZD6ZUPYCPI2F3YSMTWZAVCNFSM6AAAAABCSY4R6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZRGQZDSNBRHA . You are receiving this because you were mentioned.Message ID: @.***>

laisee commented 7 months ago

Tested latest version:

The original issue around 'Positions' websocket failure is no longer seen. When net connections are dropped by turning off WIFI for my macbook the 'Orders' web socket turns red after couple of minutes. Did not see any other channel turning off although I waited for 6 mins total.

Observations:

'Orders' web socket automatically reconnects once net access restored by turning WIFI back on. Took about 1-2 mins to go from RED to ORANGE to GREEN
No other websockets turned red. Should this be happening for all disconnected channels?
Doing 'soft' refresh with Cmd/R reloads the web page but it sticks on "40%" for about 3-4 mins

Doing 'hard' refresh with Cmd/Shift/R reloads page without delays, in about 3-6 secs

laisee commented 7 months ago

Ran the test for a bit longer, leaving the 'Trade' tab open and checking every few mins on the "connected" UI display

After about 8-10 mins the 'Positions' web socket went red, as before, and stayed red. In other words it didnt re-connect automatically. See below for UI display. I then turned WIFI and 'Orders' web socket turned orange then red. Once I turned WIFI back on the 'Orders' web socket re-connnected automatically.

The 'Positions' web socket did not reconnect until I made a hard refresh of the page. Still seems to be an issue with this one.

klugjo commented 7 months ago

Hi @laisee

Thanks a lot for the feedback. I added a few new things to help:

Idle status:

After one hour, the app will go idle and close all connections. There is a button to reactivate things without reloading

Disconnect status:

This happens if your network goes down. When it comes back online, an app reload is mandatory as its too hard to know what went offline or not

A fix for that Positions not reconnecting

I think I have found a bug in the reconnection flow but can't be 100% sure as I am not able to reproduce the error consistently. I appreciate your help and patience in testing that again

laisee commented 7 months ago

Thanks for the updates.Will retry a few tests and idle sessions to see how it behaves now.

laisee commented 7 months ago

Tested latest version and its working for the most part, but the disconnection after X mins still happening even though I changed nothing about the net connection or browser.

See below, the websockets for 'Order' and 'Positions' disconnected after some time (< 20 mins) of inactivity.

Showing full view of the web page

Now I turn off WIFI to fully disconnect the app and app responds correctly in 1-2 secs

Showing full view of the web page while WIFI is OFF

Switching WIFI to ON restores all websocket connections immediately.

Hope this helps.

laisee commented 7 months ago

Did one more test. I closed my mac book (OSX/M2) and opened again after 10 secs. Web UI looked like this

I guess its a 'sleep' mode entered for networking when laptop is closed. And likewise the automatic 'sleep' on networking done by OSX for laptop left open.

Observations:

Only seems to affect 'Orders' (sometimes) and 'Positions' (almost all times) though I have not done systematic tests
PT App doesn't seem to detect this change automatically - needs a hard reload to get all web sockets reconnected
PT App operates at less than 100% functionality while in this state. I can book trades, which are filled or added to order book but position data is missing and changes to my account health are not provided or updated. This could be dangerous to PT and/or Traders when adding trades with no idea of position/health impact.

How it looks on account info when positions websocket is disconnected

laisee commented 7 months ago

I tested the latest Web UI and its working nicely when loss of network connection occurs. The new UI displays are good and keep the traders informed and in control IMHO.

Let me know if any other testing would be useful, otherwise this looks complete from my side.

klugjo commented 7 months ago

Thanks @laisee I am closing this then. We can open a new ticket if needed

laisee commented 7 months ago

Cool. UI works nicely for me now.

On Wed, 14 Feb 2024 at 14:46, Klughertz Jonathan @.***> wrote:

Thanks @laisee https://github.com/laisee I am closing this then. We can open a new ticket if needed

— Reply to this email directly, view it on GitHub https://github.com/Power-Trade/api-docs/issues/8#issuecomment-1943173416, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNBV2S435PJ5H62WSTCYFTYTRMU7AVCNFSM6AAAAABCSY4R6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBTGE3TGNBRGY . You are receiving this because you were mentioned.Message ID: @.***>

Power-Trade / api-docs

Web Socket for Web UI 'Positions' disconnecting #8