Megathread: Streamer issues, DXFeed licensing, missing event types & symbols

[Graeme22]

Hi folks, I wanted to let everyone know about some developments that will affect a lot of users of the SDK.

Recently, a lot of users have been having issues with the streamer (#135, #137, #141) It turns out this is due to some changes that have been implemented upstream in the API. For a bit of background reading see the Javascript SDK's page: tastytrade/tastytrade-api-js#7, tastytrade/tastytrade-api-js#9, tastytrade/tastytrade-api-js#40.

Long story short, there ~are~ were two types of streamers supported in this SDK, the DXFeed and the DXLink streamer. DXFeed was an older protocol which now appears to be completely defunct. DXLink is the new protocol, and most users had already migrated to it. Not too big of a deal up to this point.

However, it appears that Tastytrade is now providing two different websockets for use with DXLink. The first, obtained by hitting the /quote-streamer-tokens endpoint, has been around for a long time. It is the endpoint used by official Tastytrade applications (I confirmed it's still being used by the Tasty web platform with a MITM just now), and doesn't have many restrictions with respect to event types, instrument types, etc. The second, newer one, /api-quote-tokens, is designed specifically for us API users.

Apparently, the /quote-streamer-tokens endpoint is now reserved for internal usage, and it is considered a violation of DXFeed TOS to use it. According to Devin Moss from TT:

"Distributing quote data is something that puts tastytrade at some risk of breeching [sic] licensing agreements, especially if they were to make it available to all users with API access. tastytrade has no control over what API users do with those quotes, which poses a real possibility that abusers could use the data to source their own proprietary projects."

Well, there's definitely plenty of people using the data "to source their own proprietary projects," though I thought that was kind of the whole purpose of the API... It certainly appears most users are using it alongside Tastytrade in a way that seems to fit within the API's intended purpose.

So where does that leave us? Well, to start off, some "normal" users will be unaffected. Those who are not trading indices nor futures and don't use greeks will hopefully be able to use the new websocket without problems. It's already mostly implemented in #140.

However, there are many users who are relying on the streamer for other kinds of instruments (especially SPX afaict) or events, like greeks. These users could continue to use the old DXLink websocket; however, Devin has also said:

"For the time being, please only use the [/api-quote-tokens] endpoint. API users who rely on the /quote-streamer-tokens endpoint will be flagged and put on delayed quotes."

So this is pretty bad news for a lot of users. I guess it was too good to be true that simply creating a Tastytrade account was enough to get real-time data for any instrument you could want? Anyways, that's pretty much where we're at for now. It doesn't seem like Tastytrade will change this, but maybe if we get enough people to email them something can be worked out.

[Graeme22]

So here's the plan. The main branch and the version v8.0+ releases for this repository will follow the DXFeed TOS. However, I've created a branch advanced-streamer for people who need the old functionality, which I'll also release periodically as v7.1+. I'll do my best to keep it working if possible.

[some-person-0]

Thanks for all your hard work and the update. I view the loss of the ability to get greeks as an absolute disaster!

[OperationalFallacy]

This is a good overview, @Graeme22 By the way, some users are completely fine with the delayed quotes for the automated trading via API.

There is also a technical problem with using dxfeed streaming quotes within API services on a backend: client.connect('wss://demo.dxfeed.com/dxlink-ws') - unlike previous implementation, this is strictly designed for the web and app clients. It is still possible to use it in the backend, but it looks akward.

I hope TT will figure out some pricing for API streaming. Of course, there is an avenue to sign up directly with Dxfeed; the problem is that the whole industry and Dxfeed are not particularly interested in catering to small API development shops.

I'm also considering abstracting quote service in the backend, so any provider can be plugged in if needed.

The problem with that TT backend is too tied up to the formats and structures of dxfeed, I suspect it will be very hard to separate them. I haven't looked into that too much yet. It's interesting to hear what others are planning to do.

[joshuakoh7]

So here's the plan. The main branch and the version v8.0+ releases for this repository will follow the DXFeed TOS. However, I've created a branch advanced-streamer for people who need the old functionality, which I'll also release periodically as v7.1+. I'll do my best to keep it working if possible.

Please consider a larger buffer for the advanced streamer because subscribing TAS for the whole SPX chain is unstable with too many subscribe calls

Also, consider sending a different user-agent header instead of python requests

[Graeme22]

Please consider a larger buffer for the advanced streamer because subscribing TAS for the whole SPX chain is unstable with too many subscribe calls

So that's actually not in my control at all. Personally I haven't had problems with whole SPX chain, but you could consider stitching together two separate streamer if it's not working for you.

Also, consider sending a different user-agent header instead of python requests

The v7.1 release is already using a random, realistic user agent!

[joshuakoh7]

Please consider a larger buffer for the advanced streamer because subscribing TAS for the whole SPX chain is unstable with too many subscribe calls

So that's actually not in my control at all. Personally I haven't had problems with whole SPX chain, but you could consider stitching together two separate streamer if it's not working for you.

Are you subscribing to TAS of the full chain? I was using the old repo under the old tastyworks package before switching to DXLinkStreamer here and I was able to chunk 10k symbols per subscribe call with the old streamer. With the new streamer I can do max of 2k before I hit the buffer size. Not sure if this is the reason for the streamer dropping more often (sending too many subscriptions calls).

Also, consider sending a different user-agent header instead of python requests

The v7.1 release is already using a random, realistic user agent!

Not for the logging in to tastytrade though (TT could be flagging via that)

[Graeme22]

Are you subscribing to TAS of the full chain?

Nope, I was doing Greeks and quotes.

I was using the old repo under the old tastyworks package before switching to DXLinkStreamer here and I was able to chunk 10k symbols per subscribe call with the old streamer. With the new streamer I can do max of 2k before I hit the buffer size. Not sure if this is the reason for the streamer dropping more often (sending too many subscriptions calls).

Not much that can be done here, though it seems like as it's something you only have to do once it shouldn't be too big of a deal? Granted I don't know your use case.

Not for the logging in to tastytrade though (TT could be flagging via that)

Ha, good point! Would you be able to do a PR to fix that?

[kaidaniel82]

@Graeme22 Hi all, can someone pls confirm if VXX, UVIX, and VIXY are working? I've been facing issues with an older version of the API (6.4) that has become very unreliable in receiving those quotes for the past few days. It was working fine for many month. I like to know if this could be related to the issue you are describing.

I connect to the DxLink service up to four times simultaneously through parallel processes, e.g. each process is independently connected to Tasty. It has always worked until the last few days.

[Graeme22]

That's probably the issue. Could you see if the latest (7.2) fixes it? Those are ETFs so no reason they shouldn't work normally.

[kaidaniel82]

Thank you for your note,

I'm looking into it now. I need to perform a minor surgery on it, as I had forked the repository to implement a mechanism that inserts an SSL certificate into the Streamer class to connect with the websocket. By the way, this would be a great, fundamental feature.

Additionally, it's not so easy to test since it sometimes works and sometimes doesn't over the past few days.

[Graeme22]

You're welcome to open a PR, though I'm not sure why we'd need to secure quotes streams, maybe you could enlighten me on that?

[kaidaniel82]

I had to set up a complete RnD project on Windows Server 2022 to figure out why the websocket for live data was delivering no data or unreliable data. The same code ran fine on my Windows 11 computer, Linux , and Mac. As soon as I switched to Win Server 2022, it didn't work.

I received Python error messages indicating that there was something wrong with the SSL certificate, or perhaps they weren't properly found on the server. Consequently, I assumed that the upstream library needed them.

After I added these cert.pem files to the websocket.connect, it worked flawlessly for months for me and many of my colleagues.

[joshuakoh7]

Got an email from TT, seems like the new API should now support indices and futures? Can anyone confirm?

[Graeme22]

Got an email from TT, seems like the new API should now support indices and futures? Can anyone confirm?

As of now, I was able to get quotes for futures options but not indices. Greeks did not work.

[Graeme22]

I had to set up a complete RnD project on Windows Server 2022 to figure out why the websocket for live data was delivering no data or unreliable data. The same code ran fine on my Windows 11 computer, Linux , and Mac. As soon as I switched to Win Server 2022, it didn't work.

I received Python error messages indicating that there was something wrong with the SSL certificate, or perhaps they weren't properly found on the server. Consequently, I assumed that the upstream library needed them.

After I added these cert.pem files to the websocket.connect, it worked flawlessly for months for me and many of my colleagues.

Interesting! Could you open a PR to add this behavior optionally? Eg with a flag to turn it on. And maybe add what you just said to the docs on the streamer as well?

[ferdousbhai]

Is there a way to get the bid and ask price of an asset without using the streamer?

I can't get the quote examples from the docs to run using the DXLinkStreamer, I get "TastytradeError: Connection timed out".

[Graeme22]

Is there a way to get the bid and ask price of an asset without using the streamer?

I can't get the quote examples from the docs to run using the DXLinkStreamer, I get "TastytradeError: Connection timed out".

Nope, streamer is required for live prices. What version of the SDK are you using?

[ferdousbhai]

Hi @Graeme22, I was using version 8.0, from Jupyter notebook.

I just tried running it from an IDE and now I see that the issue is related to SSL certificate. I installed certificates, and now it's working. Thanks!

[kaidaniel82]

Hi @Graeme22, I was using version 8.0, from Jupyter notebook.

I just tried running it from an IDE and now I see that the issue is related to SSL certificate. I installed certificates, and now it's working. Thanks!

Looks like the reported issue which I highlighted 2 weeks ago...

[Graeme22]

Looks like the reported issue which I highlighted 2 weeks ago...

Could be! @ferdousbhai what OS are you on?

@kaidaniel82 could you open a PR with your fix?

[kaidaniel82]

Looks like the reported issue which I highlighted 2 weeks ago...

Could be! @ferdousbhai what OS are you on?

@kaidaniel82 could you open a PR with your fix?

I opened a PR on advanced-streamer branch.

import ssl
ssl_context = ssl.create_default_context(cafile=cert_path)
async with DXLinkStreamer(session, ssl_context=ssl_context) as data_feed:
....

[Graeme22]

import ssl ssl_context = ssl.create_default_context(cafile=cert_path) async with DXLinkStreamer(session, ssl_context=ssl_context) as data_feed:

Thanks @kaidaniel82! Did you find that it was necessary to use specific settings for the SSL context? Or was it just the presence of the context that made everything work?

EDIT: I had a client with this issue on macOS, and was able to resolve like so:

import certifi
import ssl
async with DXLinkStreamer(session, ssl.SSLContext(cafile=certifi.where())):
    pass

[kaidaniel82]

Just the presence of the context fixed all issues.

BTW I believe my suggestion and the one you mentioned are essentially the same. Certifi packages the certificates in the site-packages folder and provides the path to the certificates using the where() command. I have also used this method elsewhere in my app, specifically when using a Discord API in my product, where I use Certifi due to i ran into the same error message. Based on the development history, I previously downloaded the certificates from Mozilla for Tasty API and stored them in my app as files. These are, to my knowledge, the same certificates that Certifi includes. Both methods should work and, as far as I understand, have the same effect.

[fisher8-jake]

Can anyone confirm if the streamer is working for options on indices (e.g. SPY)?

[Quenos]

Can anyone confirm if the streamer is working for options on indices (e.g. SPY)?

Hi, The streamer works every symbol that can be traded on TastyTrade. So, SPY and other indices are available.

[fisher8-jake]

I have not been able to stream SPY options quotes. The program just hangs awaiting a quote from the streamer.

[Graeme22]

I have not been able to stream SPY options quotes. The program just hangs awaiting a quote from the streamer.

Can you post your code, SDK version, etc.?

[fisher8-jake]

session = ProductionSession(username, password)    
async with DXLinkStreamer(session) as streamer:
# Initial subscription to SPY
print("Subscribing to stream...")
await streamer.subscribe(EventType.QUOTE, [symbol])
print(f"Subscribed to {symbol} stream.")
quote = await streamer.get_event(EventType.QUOTE)
print(f"Receiving quotes: {quote.eventSymbol} has ask price of {quote.askPrice}")

The program hangs forever on the quote line.

[Quenos]

And what are you using as symbol.

It would be helpful to give the full function and the call to that function.

[fisher8-jake]

Here is a simplified version of the code, but captures all the essentials.

import asyncio
import os
import datetime as dt
import pandas as pd
from tastytrade import ProductionSession, Account, DXLinkStreamer
from tastytrade.dxfeed import EventType
from datetime import datetime
import pandas_market_calendars as mcal
import pytz
from dotenv import load_dotenv

async def main(session, symbol, market_close):
    async with DXLinkStreamer(session) as streamer:
        # Initial subscription to SPY
        print("Subscribing to stream...")
        await streamer.subscribe(EventType.QUOTE, [symbol])
        print(f"Subscribed to {symbol} stream.")

        # Continuously listen for quotes
        print("Starting while loop for stream.")
        # resp = streamer._connect()

        while dt.datetime.utcnow() < market_close:
            # Get the current time in that timezone
            # new_york_time = datetime.now(new_york_tz)  # Market close given in UTC time
            print("In the loop, awaiting quote...")
            print(f"The event type is {EventType.QUOTE}")
            # resp = streamer.listen(EventType.QUOTE)
            quote = await streamer.get_event(EventType.QUOTE)
            print(f"Receiving quotes: {quote.eventSymbol} has ask price of {quote.askPrice}")

load_dotenv()
symbol = 'SPY'
# symbol = 'TSLA'
username = os.environ.get('TastyTrade_Username')
password = os.environ.get('TastyTrade_Password')
account_num = os.environ.get('TastyTrade_AccountNum')
session = ProductionSession(username, password)

expiration = dt.date.today() + dt.timedelta(days=1)

# Create a calendar object for NYSE
nyse = mcal.get_calendar('NYSE')
# Create a timezone object for New York
new_york_tz = pytz.timezone('America/New_York')
current_time_ny = datetime.now(new_york_tz)
current_date_ny = current_time_ny.date()

# Get the trading days for the current month
# Adjust the start and end dates as needed to cover the current date
SPY_time_offset = 15  # SPY options close 15 minutes AFTER market
start_date = current_date_ny.replace(day=1)
end_date = current_date_ny.replace(day=1).replace(month=current_date_ny.month % 12 + 1) - dt.timedelta(days=1)
schedule = nyse.schedule(start_date=start_date, end_date=end_date)
day_schedule = schedule[schedule.index == pd.to_datetime(current_date_ny)]
market_close = day_schedule['market_close'].iloc[0]
market_close = market_close.replace(tzinfo=None)  # This is given in UTC
market_close = market_close + dt.timedelta(minutes=SPY_time_offset)

# Run Streamer
if __name__ == "__main__":
    asyncio.run(main(session=session, symbol=symbol, market_close=market_close))

[Quenos]

I ran your code and I'm getting the SPY quote back.

Quote(eventSymbol='SPY', eventTime=0, sequence=0, timeNanoPart=0, bidTime=0, bidExchangeCode='Q', askTime=0, askExchangeCode='Q', bidPrice=Decimal('540.71'), askPrice=Decimal('540.74'), bidSize=721, askSize=1037)

[fisher8-jake]

Ok thanks, I uninstalled the package and re-installed. Working again, appreciate the help.

[Graeme22]

Ok thanks, I uninstalled the package and re-installed. Working again, appreciate the help.

Great! By the way, some of the functions you were creating are already included in the SDK, you may find them useful: https://tastyworks-api.readthedocs.io/en/latest/tastytrade.html#tastytrade.utils.now_in_new_york

[ferdousbhai]

Looks like the reported issue which I highlighted 2 weeks ago...

Could be! @ferdousbhai what OS are you on?

@kaidaniel82 could you open a PR with your fix?

MacOS

[marwinsteiner]

Am I correct in saying that at the moment the DXLink WebSocket can only be used to stream Quote data and none of the other market events like Greeks or Profile which users of the tasty API have access to? Do I need to go through DXFeed to get the option chain, for instance? Not directly related to the SDK but would love any general pointers.

[Graeme22]

Am I correct in saying that at the moment the DXLink WebSocket can only be used to stream Quote data and none of the other market events like Greeks or Profile which users of the tasty API have access to? Do I need to go through DXFeed to get the option chain, for instance? Not directly related to the SDK but would love any general pointers.

Nope! As long as you're using the SDK, it'll behave like one of the official Tasty clients and you can get quotes, greeks, candles and more with no problems.

[marwinsteiner]

Thanks Graeme... I guess nobody knows when they will add Greeks to their DXLink offering? From the tasty documentation, which I only discovered today:

[inanisvitae]

import ssl ssl_context = ssl.create_default_context(cafile=cert_path) async with DXLinkStreamer(session, ssl_context=ssl_context) as data_feed:

Thanks @kaidaniel82! Did you find that it was necessary to use specific settings for the SSL context? Or was it just the presence of the context that made everything work?

EDIT: I had a client with this issue on macOS, and was able to resolve like so:

import certifi
import ssl
async with DXLinkStreamer(session, ssl.SSLContext(cafile=certifi.where())):
    pass

Hi @Graeme22 , Firstly, thanks for all your effort. I ran into this issue as well on mac, but I was able to fix it this way. I just have a question. If I want to deploy my app with this DXLinkStreamer on cloud such as gcp or AWS, do I still need this bit or not?

[Graeme22]

Firstly, thanks for all your effort. I ran into this issue as well on mac, but I was able to fix it this way. I just have a question. If I want to deploy my app with this DXLinkStreamer on cloud such as gcp or AWS, do I still need this bit or not?

Hi, glad to help!

I think probably not, I've been deploying to Heroku without any issues so I suspect it's a Mac-only thing.

[kaidaniel82]

Firstly, thanks for all your effort. I ran into this issue as well on mac, but I was able to fix it this way. I just have a question. If I want to deploy my app with this DXLinkStreamer on cloud such as gcp or AWS, do I still need this bit or not?

Hi, glad to help!

I think probably not, I've been deploying to Heroku without any issues so I suspect it's a Mac-only thing.

@Graeme22 i ran into this issue on my windows server. This forced me to used the method i described...

[Graeme22]

@Graeme22 i ran into this issue on my windows server. This forced me to used the method i described...

Confirmed not Mac-only! TL;DR use Linux 😁

[inanisvitae]

Hey @Graeme22 , Ran into a warning regarding this, DeprecationWarning: ssl.SSLContext() without protocol argument is deprecated. async with DXLinkStreamer(s, ssl.SSLContext(cafile=certifi.where())) as streamer: I suppose we just ignore it? Thanks

[Graeme22]

Hey @Graeme22 , Ran into a warning regarding this, DeprecationWarning: ssl.SSLContext() without protocol argument is deprecated. async with DXLinkStreamer(s, ssl.SSLContext(cafile=certifi.where())) as streamer: I suppose we just ignore it? Thanks

Yup! If you ever get an actual error instead of a warning, you can follow the instructions here: https://stackoverflow.com/a/72810634/13938613

[pangyuteng]

Thanks Graeme... I guess nobody knows when they will add Greeks to their DXLink offering?

fyi. greeks is avilable. 🤫 https://github.com/tastyware/tastytrade/blob/97e1bc6632cfd4a15721da816085eb906a02bcb0/docs/data-streamer.rst#L76