bmello4688
commented
2 years ago Hi Steve,
So I have found that the tda api is very buggy and glitchy. The stale data could be because no trades are happening if you are outside of market hours. I’m looking into a better solution such as getting the thinkorswim api, but it will be some time.
I have created a plug-in for QuantConnect’s Lean Engine. It may help fix some of your problems.
Brian
Brian
From: Steven Fischer @.> Sent: Tuesday, February 1, 2022 12:14:12 PM To: bmello4688/TDAmeritradeApi @.> Cc: Subscribed @.***> Subject: [bmello4688/TDAmeritradeApi] Issues with streaming data disconnecting and stale data (Issue #8)
Hi there - first off, thank you for your continued work. I've built my bot around your project and am having success using the TDA data feed for about 1200 tickers.
I'm experiencing some frustrating issues with the streaming of live chart data (likely due to TDA API glitchiness). Streaming chart minute data seems to be duplicated and stale, and I get occasional app-killing disconnects. I'm curious if you've seen these issues or have any guidance on troubleshooting.
The issues
- Occasional but serious "ADMIN did not respond" disconnect. I'm investigating how to catch this in my app to pause and resubscribe - but right now it kills my app cold.
[image]https://user-images.githubusercontent.com/4407258/152004369-487146df-99de-436a-8430-daef76c7d835.png
-
Fresh minute chart data will stop arriving unless I periodically resubscribe. Data arrives, but it doesn't reflect current minute data. I find I need to resubscribe every two minutes or less to keep fresh data flowing.
-
Minute chart data contains "old" data. Here's a screenshot of what comes back in the marketData array for a few symbols. Notice duplicate data entries for the same minute. After my app runs for a long time, there are many duplicate chart records for each symbol. I'm not sure if this is a contributing factor to my other issues - but it feels like old data is hanging out causing a problem.
[image]https://user-images.githubusercontent.com/4407258/152014550-e75459d2-363a-446c-90d9-754d7803cb75.png
My setup:
- I'm using similar code to the test project (see below)
- I'm streaming between 300 and 1500 symbols by repeatedly calling subscribe with chunks of
- I use two TDA accounts with separate TDA client instances (different TDA API cust key, different TDA broker account).
- One account is used exclusively for streaming minute chart data, the other account handles live quote streaming, buy/sell orders, and account data updates.
if (mdt == MarketDataType.Charts) {
_lastCandleTime = _clockService.Now;
if (_recentCandles == null)
_recentCandles = new List<(string, DateTime, DateTime)>();
lock (_recentCandles)
_recentCandles = Enumerable.TakeLast(_recentCandles, 5000).ToList();
foreach (var key in marketData[MarketDataType.Charts].Keys)
{
try
{
MinuteChartData[] minutes = marketData[MarketDataType.Charts][key].Data.ToArray();
List<MinuteChartData> minuteList = new List<MinuteChartData>(minutes);
// echo out every minute returned
foreach (var min in minuteList)
{
await SaveStreamLog($"{ _clockService.Timestamp() },{ min.Symbol },ALL-MIN,{ min.ChartTime.ToShortTimeString() }");
}
var minute = minuteList.OrderBy(m => m.ChartTime).LastOrDefault();
var diff = Math.Abs(DateTime.Now.AddHours(TIMEZONE_HOURS_OFFSET * -1).Subtract(minute.ChartTime).TotalSeconds);
if (diff < 100)
{
quoteCount++;
Console.ForegroundColor = ConsoleColor.Green;
}
if (_zenService.ExtendedLogging)
Console.WriteLine(quoteCount + " " + minute.Symbol + "- " + minute.ChartTime.ToShortDateString() + " " + minute.ChartTime.ToLongTimeString() + " " + minute.ClosePrice + " " + diff);
Console.ForegroundColor = ConsoleColor.White;
var candle = new Shared.Candle()
{
Close = (decimal)minute.ClosePrice,
Open = (decimal)minute.OpenPrice,
DateTime = minute.ChartTime,
High = (decimal)minute.HighPrice,
Low = (decimal)minute.LowPrice,
Volume = (long)minute.Volume
};
await SaveStreamLog($"{ _clockService.Timestamp() },{ minute.Symbol },MIN,{ minute.ChartTime.ToShortTimeString() }");
_recentCandles.Add((minute.Symbol, minute.ChartTime, _clockService.Now));
var ticker = _zenService.LiveTickers.Where(t => t.Name == minute.Symbol).LastOrDefault();
if (ticker != null)
{
// is this a new candle?
if (candle.DateTime > ticker.LastCandleSeen)
{
lock (ticker)
ticker.LastCandleSeen = candle.DateTime;
// add to in-memory database
AddMinuteCandle(minute.Symbol, candle, false).Wait();
CalculateMostRecentCandles(minute.Symbol).Wait();
// immediately process the ticker
if (Math.Abs(_clockService.Now.Subtract(candle.DateTime.AddHours(TIMEZONE_HOURS_OFFSET)).TotalMinutes) <= 2)
{
Thread newThread = new Thread(ProcessTicker);
newThread.IsBackground = true;
newThread.Start(ticker);
}
}
}
}
catch (Exception ex)
{
Console.WriteLine("STREAMING ERROR");
Console.WriteLine(ex);
}
}
}`— Reply to this email directly, view it on GitHubhttps://github.com/bmello4688/TDAmeritradeApi/issues/8, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB3WTQSIZU6SKXGZ22RKBWTUZAIGJANCNFSM5NJYCG5A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.***>
StevenFischerWI
Hi there - first off, thank you for your continued work. I've built my bot around your project and am having success using the TDA data feed for about 1200 tickers.
I'm experiencing some frustrating issues with the streaming of live chart data (likely due to TDA API glitchiness). Streaming chart minute data seems to be duplicated and stale, and I get occasional app-killing disconnects. I'm curious if you've seen these issues or have any guidance on troubleshooting.
The issues
Fresh minute chart data will stop arriving unless I periodically resubscribe. Data arrives, but it doesn't reflect current minute data. I find I need to resubscribe every two minutes or less to keep fresh data flowing.
Minute chart data contains "old" data. Here's a screenshot of what comes back in the marketData array for a few symbols. Notice duplicate data entries for the same minute. After my app runs for a long time, there are many duplicate chart records for each symbol. I'm not sure if this is a contributing factor to my other issues - but it feels like old data is hanging out causing a problem.
My setup:
if (mdt == MarketDataType.Charts) {