search

alpacahq / alpaca-trade-api-python

Python client for Alpaca's trade API
https://pypi.org/project/alpaca-trade-api/
Apache License 2.0
1.72k stars 529 forks source link

[WARNING] data websocket error, restarting connection: no close frame received or sent #588

Open kimboox44 opened 2 years ago

kimboox44 commented 2 years ago

Is there an existing issue for this?

Current Behavior

websocket connection on sip is unusable for me, a connection lost every 10 seconds, one of tow errors

some suggested that api already reconnect automatically, that's not a solution because you're missing 5sec of timeout before the connection actually drop plus reconnection+subscription time, that leaves gaps in 1minute bars by the time you start receiving again. for 24h run i got a total of 290 candles on AAPL (because of this error) out of possible 960 candles that 70% lost data.

my guess:

Expected Behavior

Expected to work

Steps To Reproduce

simple 1minute barse subscription for all symbols

Anything else?

No response

umitanuki commented 2 years ago

I am also experiencing this while comparing with CoinbasePro websocket. I've just run some 10 minutes and 2 of these where data stalled for some seconds or maybe 10+, while Coinbase Pro connection keeps runnnig without problem

ovresko commented 2 years ago

I can't understand Alpaca focusing in adding more features/markets while the core of thier business (data) is getting worse everyday. i'm thinking of switching, for the same problem, i'm sure they're dropping random connections to reduce load. they don't care anymore

ccnlui commented 2 years ago

This is a challenge inherent to streaming data with websocket, if the client is not consuming data at the rate that the server is producing, client is bound to get disconnected. This is a pretty good article of why this happens.

Some suggestions:

ovresko commented 2 years ago

This is a challenge inherent to streaming data with websocket, if the client is not consuming data at the rate that the server is producing, client is bound to get disconnected. This is a pretty good article of why this happens.

Some suggestions:

  • dont subscribe to more data than necessary
  • make sure handler is not doing heavy computation and it is reading as fast as possible from the websocket
  • avoid running client in VMs or containers
  • try the more performant go-sdk

for me i'm subscribing to 10 to 20 symbols trades max on a 64Gb RAM 8Cores VPS nothing else, and yet connection drop every 5 to 20 seconds (1006,1011), if it's a websocket client issue we would experience the same for other websockets, for example our binance subscribtion run with all available coins +500 live, for 7 months with no disconnections (python also). i've analyzed the traffic from alpaca, and it's always ALWAYS dropping around 10th 15th or 20th second on a new minute and not when receiving bars at second 0 to 5 where traffic is higher and bellow 20sec ping timout. our system designed around alpaca's websocket so switching is a nasty task. i've tried subscribing to only one symbol with no significant volume, and yet it drops at 15th second. i'm giving up

umitanuki commented 2 years ago

100% agree that this needs to be addressed. As I reported, I'm also comparing with coinbase pro websocket connection and it doesn't disconnect.

With my internal hat on, Alpaca's backend team is working on it and we are hoping to get this sorted.

@ovresko you mentioned it got worse. Do you know around when it got worse, and was it not like this before?

raja commented 2 years ago

@umitanuki @ovresko @kimboox44 To help better address the issue, can you see if you can reproduce the undesirable interimittent disconnects by using https://github.com/hashrocket/ws with the same subscription message you are using with alpaca python sdk. This will help us better understand if this is specific to the websocket client implementation in our python sdk (which I believe it may be) vs our streaming endpoint.

ovresko commented 2 years ago

@raja i did try ws, for bars it just stops same behaviour, after a minute without any messages but it did hold the connection for 1 trades subscription (no bars) for 20 minutes now

tried in two seperate networks (DSL, 4G) same, and i don't see anything useful except response data meaning low level logs

cmd_ZKTzSxLJMB

ovresko commented 2 years ago

I tried using bare minimum subscription using websockets, and it's workign fine so far with 1000 live trades and bars subscription. i think it's a problem with python api too InkedCode_grxSmkNCxR_LI

ovresko commented 2 years ago

there are some data providers that offers dynamic price update interval on ws, live/100ms/500ms/1s to be set at subscription to address the situation where clients can't keep up, that would solve all performance related issues (just opt for lower rate)

drew887 commented 2 years ago

Hey just to give an update here since its been over a week.

While we are still investigating this issue, we have recently merged a PR that we hope will help lessen the frequency and severity of this issue in the mean time for most users. We also have a second PR merged allowing users to configure the options we pass to the websocket library we use under the hood themselves if these defaults wont work for them.

I will try and get a new release out soon so that these changes are available (EDIT: v2.0.0 has been released and contains these changes) but all I can ask is that you please continue to be patient with us in the meantime while we investigate :sweat:.

sshcli commented 1 year ago

Still facing this problem in the new python-py SDK