minute aggregate bars missing

[connor-roche]

I set up a WebSocket connection to grab 1 min aggregates for a number of stocks:

    alpaca_api = alpaca.REST(
        "key",
        "secretkey",
        api_version="v2",
    )
    stream = alpaca.StreamConn(
        "key", "secretkey",
    )

    symbols = {
        symbol: Scalper(
            api=alpaca_api, symbol=symbol, allotted_liquidity=args.allotted_liquidity
        )
        for symbol in args.symbols
    }

    @stream.on(r"^AM")
    async def on_bars(conn, channel, data):
        logger.info(f"{data.symbol} new bar: {data}")
        if data.symbol in symbols:
            symbols[data.symbol].add_new_closing_price(new_closing_price=data.close)

    @stream.on(r"trade_updates")
    async def on_trade_update(conn, channel, data):
        logger.info(f"trade udpate: {data}")
        symbol = data.order["symbol"]
        if symbol in symbols:
            symbols[symbol].on_trade_update(trade_data=data)

    async def check_for_flatten():
        flatten_time = pd.Timestamp.now(tz="America/New_York").replace(
            hour=15, minute=59, second=0
        )
        while True:
            if pd.Timestamp.now(tz="America/New_York") > flatten_time:
                logger.info("time to flatten then exit")
                alpaca_api.cancel_all_orders()
                alpaca_api.close_all_positions()
                sys.exit(0)
            await asyncio.sleep(20)

    channels = ["trade_updates"] + [f"AM.{symbol}" for symbol in args.symbols]
    loop = stream.loop
    loop.run_until_complete(
        asyncio.gather(stream.subscribe(channels), check_for_flatten(),)
    )
    loop.close()

This is similar to the work done here: https://github.com/alpacahq/example-scalping by @umitanuki. However, when I run the script it doesn't return bars every minute for all the symbols. Some minute aggregates are skipped for some symbols and it changes for different symbols each minute as well.

I tried this with 5 and 15 symbols and still, the same issue persists. Is this on the polygon side or is there a check to make sure the WebSocket returns info for every minute aggregate? Also, if others have had this issue then could it be best to move to REST API with multiple threads running, one for each symbol?

[jaggas]

I think there is a volume threshold to publish a minute bar, maybe 100 shares or so. If you look at heavy volume symbols like SPY or QQQ you will see minute bars at every interval. The bars can be pretty spread out in PM/AH due to low volume.

[shlomiku]

Hi @connor-roche it is generally not good practice to do the REST calls for every symbol in a separate thread ( you could use it as a backup if you want) but exactly for that purpose you have the WS service. having said that, if you use the latest package version (or >=0.48) you are using the alpaca WS, not the polygon WS. to use the polygon WS you need to specify it directly alpaca.StreamConn("key", "secretkey", data_stream='polygon')

a follow up to @jaggas 's comment: when are you experiencing these issues (is it PM/AH)? and is it for highly traded stocks or not?

[connor-roche]

I tried this with the Polygon WS and Alpaca WS and still got the same result. While this issue is much more exaggerated in pre-market and after-market (I get a response from the SPY minute interval WS every 5-8 mins) I am still experiencing the issue during market hours. I ran my script with a number of high volume stocks and had this problem for a decent amount of them. Here are some symbols where I ran into the issue: AMZN / DIS / AMD / SHOP / TSLA / GOOG / GOOGL / ZM / FB just to name a few

Keep in mind it doesn't happen every minute. Some minutes it might return a bar for TSLA and the next minute it might not. For instance, I ran my script for 22 mins and got 17 TSLA bars back spread out through the 22 mins.

If you would like I can send you my script for greater insight.

Thanks for all your help thus far @jaggas & @shlomikushchi!

[Peppershaker]

having this exact issue

[drew887]

Hey closing this one out due to inactivity, and I believe this is todo with the api/ws itself and not so much the SDK here