TickMessage.TotalVolume exceeded int.MaxValue

[Nucs]

Currently TickMessage.TotalVolume is of type int. I've received the following tick:

Timestamp	Last	LastSize	TotalVolume	Bid	Ask	TickId	BasisForLast	TradeMarketCenter	TradeConditions
15:05.2	0.0127	39370079	2160408936	0.0125	0.0135	318215	C	56	1

Might I suggest upgrading LastSize to 'long' as-well.

[mathpaquette]

@Nucs did you contact IQFeed? this is definitely an error from their data.

[Nucs]

No, how are you sure it is an error?

[mathpaquette]

@Nucs it happened to me in the past and it was corrupted data from their side. 2 billion its a lot lot of volume. Whats the ticker?

[mathpaquette]

https://finviz.com/screener.ashx?v=131&o=-averagevolume

[Nucs]

2 billion its a lot lot of volume Yeah it is.. we need a less destructive way to handle bad data if that's the case We couldn't find what ticker that is

[mathpaquette]

changing data type is a very impacting change for very rare exceptions like this. On top of that, you already need to try/catch every lookup requests because you can receive unordered data or NO_DATA exception. for me this has nothing to do with the library itself.

[mathpaquette]

https://github.com/QuantConnect/Lean/blob/master/ToolBox/IQFeed/IQFeedFileHistoryProvider.cs#L278

[mathpaquette]

Like I said, pretty sure this is corrupted data from their end and I won't merge it for now. Please come back with specific tickers and we can escalate that with IQFeed. It's better to have an overflow exception and discard the data than hiding something under the carpet.

[mathpaquette]

@Nucs ill close the issue for now, please dont hesitate to reopen once you have more detail on the tickers.

[gribunin]

@mathpaquette Hello, yesterday our service crashed because it seems exactly the same issue. Here is the exception info:

Application: IqFeedManager.exe
CoreCLR Version: 5.0.20.51904
.NET Version: 5.0.0
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OverflowException: Value was either too large or too small for an Int32.
   at System.Number.ThrowOverflowOrFormatException(ParsingStatus status, TypeCode type)
   at System.Number.ParseInt32(ReadOnlySpan`1 value, NumberStyles styles, NumberFormatInfo info)
   at IQFeed.CSharpApiClient.Lookup.Historical.Messages.TickMessage.Parse(String message)
   at IQFeed.CSharpApiClient.Lookup.Common.BaseLookupMessageHandler.ProcessMessages[T](Func`2 parserFunc, Func`2 errorParserFunc, Byte[] message, Int32 count)
   at IQFeed.CSharpApiClient.Lookup.Historical.HistoricalMessageHandler.GetTickMessages(Byte[] message, Int32 count)
   at IQFeed.CSharpApiClient.Lookup.Common.BaseLookupFacade.<>c__DisplayClass4_0`1.<GetMessagesAsync>g__SocketClientOnMessageReceived|1(Object sender, SocketMessageEventArgs args)
   at IQFeed.CSharpApiClient.Socket.SocketClient.ProcessReceive(SocketAsyncEventArgs e)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pNativeOverlapped)

I think it was the error on IqFeed side though, because the service was requesting the set of tickers it always requests -- nothing unusual.

Also the corresponding call was in try/catch block with general catch(Exception), but anyway the unhandled exception crashed the app. -- it wasn't caught:

try {
 ... =   await lookupClient.Historical.GetHistoryTickDatapointsAsync(...
 } catch (Exception ex) {
 ...
 }

Was it supposed to be caught in the try/catch block?