streamer.py crashing on feed_data

[love-reading]

When using DXLinkStreamer class and connecting to dxfeed with subscription .. intermittently the parsing crashes at streamer.py

async def _map_message(self, message) -> None: """ Takes the raw JSON data, parses the events and places them into their respective queues.

    :param message: raw JSON data from the websocket
    """
    try:
        for item in message:

   msg_type = item.pop('eventType') 

AttributeError: 'str' object has no attribute 'pop'

This happens once every other time I start the application and stop it and restart it. Its not consistently reproducible but happens enough times regularly to be worrisome.

DEBUG LOGS WHEN IT FAILS <<<<<

2024-04-17 18:35:34,245 [DEBUG] received: {'type': 'FEED_CONFIG', 'channel': 7, 'dataFormat': 'COMPACT', 'aggregationPeriod': 0.0, 'eventFields': {'Quote': ['eventType', 'eventSymbol', 'eventTime', 'sequence', 'timeNanoPart', 'bidTime', 'bidExchangeCode', 'bidPrice', 'bidSize', 'askTime', 'askExchangeCode', 'askPrice', 'askSize']}}

2024-04-17 18:35:34,386 [DEBUG] received: {'type': 'FEED_DATA', 'channel': 7, 'data': ['Quote', ['Quote', 'AAPL', 0, 0, 0, 0, 'Q', 168.4, 51.0, 0, 'Q', 168.46, 10.0, 'Quote', 'SPX', 0, 0, 0, 0, '', 4979.73, 'NaN', 0, '', 5064.47, 'NaN']]}

{'type': 'FEED_DATA', 'channel': 15, 'data': ['Trade', ['Trade', 'AAPL', 0, 1713384001197, 0, 553584, 'Q', 168.0, -1.38, 14584.0, 19830, 50851471.0, 8601512193.17428, 'ZERO_DOWN', True]]}

DEBUG LOGS WHEN IT WORKS <<<<<

{'type': 'CHANNEL_REQUEST', 'channel': 7, 'service': 'FEED', 'parameters': {'contract': 'AUTO'}} {'type': 'CHANNEL_OPENED', 'channel': 7, 'service': 'FEED', 'parameters': {'contract': 'AUTO', 'subFormat': 'LIST'}} {'type': 'FEED_CONFIG', 'channel': 7, 'dataFormat': 'FULL', 'aggregationPeriod': 0.0, 'eventFields': {'Quote': ['eventType', 'eventSymbol', 'eventTime', 'sequence', 'timeNanoPart', 'bidTime', 'bidExchangeCode', 'bidPrice', 'bidSize', 'askTime', 'askExchangeCode', 'askPrice', 'askSize']}}

2024-04-17 19:09:28,507 [DEBUG] received: {'type': 'FEED_CONFIG', 'channel': 15, 'dataFormat': 'FULL', 'aggregationPeriod': 0.0, 'eventFields': {'Trade': ['eventType', 'eventSymbol', 'eventTime', 'time', 'timeNanoPart', 'sequence', 'exchangeCode', 'price', 'change', 'size', 'dayId', 'dayVolume', 'dayTurnover', 'tickDirection', 'extendedTradingHours']}}

How do I consistently get the same type of feeds from dxfeed so as to not crash the application?

The code is subscribing to a few equities and SPX options.

[love-reading]

Also, crashes on the receipt of the greek, quote, trade .. examples below

{'type': 'FEED_DATA', 'channel': 3, 'data': ['Greeks', ['Greeks', '.SPXW240417C4690', 0, 0, 7358932107072110592, 1713384899097, 0, 331.776491363427, 0.1921169629886611, 0.999664504062865, 0.0, -0.113472567367694, 0.00267629112451684, 0.0]]}

2024-04-17 20:39:00,505 [DEBUG] received: {'type': 'FEED_DATA', 'channel': 3, 'data': ['Greeks', ['Greeks', '.SPXW240417C5600', 0, 0, 7358932107072110592, 1713384899097, 0, 0.0413056993726286, 0.1853860861385492, 0.0, 0.0, -0.0413056993726286, 0.0, 0.0, 'Greeks', '.SPXW240417C6400', 0, 0, 7358932107072110592, 1713384899097, 0, 0.0229268737200528, 0.1853860861385492, 0.0, 0.0, -0.0229268737200528, 0.0, 0.0, 'Greeks', '.SPXW240417P6400', 0, 0, 7358932107072110592, 1713384899097, 0, 1377.945005255165, 0.1853860861387242, -0.999664504062865, 0.0, -0.0229268737200528, -0.00365208170509761, 0.0, 'Greeks', '.SPXW240417P6000', 0, 0, 7358932107072110592, 1713384899097, 0, 978.051188448577, 0.1853860861387242, -0.999664504062865, 0.0, -0.0320567753204841, -0.00342382659852901, 0.0, 'Greeks', '.SPXW240417C5700', 0, 0, 7358932107072110592, 1713384899097, 0, 0.0389694270195378, 0.1853860861385492, 0.0, 0.0, -0.0389694270195378, 0.0, 0.0, 'Greeks', '.SPXW240417P5800', 0, 0, 7358932107072110592, 1713384899097, 0, 778.104312486926, 0.1853860861387242, -0.999664504062865, 0.0, -0.0366541677637997, -0.00330969904524471, 0.0, 'Greeks', '.SPXW240417C6000', 0, 0, 7358932107072110592, 1713384899097, 0, 0.0320567753204841, 0.1853860861385492, 0.0, 0.0, -0.0320567753204841, 0.0, 0.0, 'Greeks', '.SPXW240417P6200', 0, 0, 7358932107072110592, 1713384899097, 0, 1177.998089826491, 0.1853860861387242, -0.999664504062865, 0.0, -0.0274847991406659, -0.00353795415181331, 0.0, 'Greeks', '.SPXW240417C5800', 0, 0, 7358932107072110592, 1713384899097, 0, 0.0366541677637997, 0.1853860861385492, 0.0, 0.0, -0.0366541677637997, 0.0, 0.0, 'Greeks', '.SPXW240417P5900', 0, 0, 7358932107072110592, 1713384899097, 0, 878.077745850665, 0.1853860861387242, -0.999664504062865, 0.0, -0.0343508544559654, -0.00336676282188686, 0.0, 'Greeks', '.SPXW240417C6200', 0, 0, 7358932107072110592, 1713384899097, 0, 0.0274847991406659, 0.1853860861385492, 0.0, 0.0, -0.0274847991406659, 0.0, 0.0]]}

Quote: 2024-04-17 20:39:00,511 [DEBUG] received: {'type': 'FEED_DATA', 'channel': 7, 'data': ['Quote', ['Quote', 'AAPL', 0, 0, 0, 0, 'Q', 166.75, 'NaN', 0, 'Q', 170.0, 'NaN', 'Quote', 'SPX', 0, 0, 0, 0, '', 4979.73, 'NaN', 0, '', 5064.47, 'NaN']]}

Trade: 2024-04-17 20:39:00,615 [DEBUG] received: {'type': 'FEED_DATA', 'channel': 15, 'data': ['Trade', ['Trade', 'AAPL', 0, 1713384001197, 0, 554671, 'Q', 168.0, -1.38, 14584.0, 19830, 50901210.0, 8609898993.78193, 'ZERO_DOWN', True, 'Trade', 'SPX', 0, 1713389037732, 0, 49202, '', 5022.21, -29.2, 'NaN', 19830, 'NaN', 'NaN', 'ZERO_DOWN', False]]}

[love-reading]

However, when it receives the following messages, example: Quote and Greeks, streamer.py is able to process them just fine.

2024-04-17 20:46:59,295 [DEBUG] received: {'type': 'FEED_DATA', 'channel': 7, 'data': [{'eventType': 'Quote', 'eventSymbol': '.SPXW240417C1400', 'eventTime': 0, 'sequence': 0, 'timeNanoPart': 0, 'bidTime': 0, 'bidExchangeCode': 'C', 'bidPrice': 3615.7, 'bidSize': 'NaN', 'askTime': 0, 'askExchangeCode': 'C', 'askPrice': 3639.7, 'askSize': 'NaN'}, {'eventType': 'Quote', 'eventSymbol': '.SPXW240417C2400', 'eventTime': 0, 'sequence': 0, 'timeNanoPart': 0, 'bidTime': 0, 'bidExchangeCode': 'C', 'bidPrice': 2615.3, 'bidSize': 'NaN', 'askTime': 0, 'askExchangeCode': 'C', 'askPrice': 2639.3, 'askSize': 'NaN'}, {'eventType': 'Quote', 'eventSymbol': '.SPXW240417P2400', 'eventTime': 0, 'sequence': 0, 'timeNanoPart': 0, 'bidTime': 0, 'bidExchangeCode': 'C', 'bidPrice': 0.0, 'bidSize': 'NaN', 'askTime': 0, 'askExchangeCode': 'C', 'askPrice': 0.05, 'askSize': 'NaN'}, {'eventType': 'Quote', 'eventSymbol': '.SPXW240417P3400', 'eventTime': 0, 'sequence': 0, 'timeNanoPart': 0, 'bidTime': 0, 'bidExchangeCode': 'C', 'bidPrice': 0.0, 'bidSize': 'NaN', 'askTime': 0, 'askExchangeCode': 'C', 'askPrice': 0.05, 'askSize': 'NaN'}, {'eventType': 'Quote', 'eventSymbol': '.SPXW240417C3900', 'eventTime': 0, 'sequence': 0, 'timeNanoPart': 0, 'bidTime': 0, 'bidExchangeCode': 'C', 'bidPrice': 1115.3, 'bidSize': 'NaN', 'askTime': 0, 'askExchangeCode': 'C', 'askPrice': 1136.6, 'askSize': 'NaN'}]}

2024-04-17 20:46:59,296 [DEBUG] received: {'type': 'FEED_DATA', 'channel': 3, 'data': [{'eventType': 'Greeks', 'eventSymbol': '.SPXW240417C1400', 'eventTime': 0, 'eventFlags': 0, 'index': 7358932107072110592, 'time': 1713384899097, 'sequence': 0, 'price': 3620.89626770391, 'volatility': 0.1921169629886611, 'delta': 0.999664504062865, 'gamma': 0.0, 'theta': -0.0315122330030135, 'rho': 0.000798892872990101, 'vega': 0.0}, {'eventType': 'Greeks', 'eventSymbol': '.SPXW240417C2400', 'eventTime': 0, 'eventFlags': 0, 'index': 7358932107072110592, 'time': 1713384899097, 'sequence': 0, 'price': 2621.16185927152, 'volatility': 0.1921169629886611, 'delta': 0.999664504062865, 'gamma': 0.0, 'theta': -0.0544705710844937, 'rho': 0.001369530639411603, 'vega': 0.0}, {'eventType': 'Greeks', 'eventSymbol': '.SPXW240417P2400', 'eventTime': 0, 'eventFlags': 0, 'index': 7358932107072110592, 'time': 1713384899097, 'sequence': 0, 'price': 0.0544705710844937, 'volatility': 0.1921169629886199, 'delta': 0.0, 'gamma': 0.0, 'theta': -0.0544705710844937, 'rho': 0.0, 'vega': 0.0}, {'eventType': 'Greeks', 'eventSymbol': '.SPXW240417P3400', 'eventTime': 0, 'eventFlags': 0, 'index': 7358932107072110592, 'time': 1713384899097, 'sequence': 0, 'price': 0.0779813714914326, 'volatility': 0.1921169629886199, 'delta': 0.0, 'gamma': 0.0, 'theta': -0.0779813714914326, 'rho': 0.0, 'vega': 0.0}, {'eventType': 'Greeks', 'eventSymbol': '.SPXW240417P4885', 'eventTime': 0, 'eventFlags': 0, 'index': 7358932107072110592, 'time': 1713384899097, 'sequence': 0, 'price': 0.1348164780519596, 'volatility': 0.1921169629886199, 'delta': -5.62721840134e-81, 'gamma': 1.472460421865321e-80, 'theta': -0.1348164780519596, 'rho': -1.613197338167427e-83, 'vega': 4.07253643082336e-80}, {'eventType': 'Greeks', 'eventSymbol': '.SPXW240417C3900', 'eventTime': 0, 'eventFlags': 0, 'index': 7358932107072110592, 'time': 1713384899097, 'sequence': 0, 'price': 1121.56129508346, 'volatility': 0.1921169629886611, 'delta': 0.999664504062865, 'gamma': 0.0, 'theta': -0.0899565387298935, 'rho': 0.002225487289043854, 'vega': 0.0}]}

[Graeme22]

This may be related to #137. Could you see if #140 fixes your issue?

[love-reading]

This may be related to #137. Could you see if #140 fixes your issue?

Seems like DXLink sometimes sends output as lists and sometimes as dicts .. could that be causing it?

How should I use #140 to test if it fixes the crash?

[love-reading]

@Graeme22 Is it a known issue that DXLinkStreamer would crash when we receive lists?

[love-reading]

reviewed code of #140, the _map_message in streamer.py still tries to pop eventType and would fail on the string received sporadically from dxFeed

Quote: 2024-04-17 20:39:00,511 [DEBUG] received: {'type': 'FEED_DATA', 'channel': 7, 'data': ['Quote', ['Quote', 'AAPL', 0, 0, 0, 0, 'Q', 166.75, 'NaN', 0, 'Q', 170.0, 'NaN', 'Quote', 'SPX', 0, 0, 0, 0, '', 4979.73, 'NaN', 0, '', 5064.47, 'NaN']]}

Trade: 2024-04-17 20:39:00,615 [DEBUG] received: {'type': 'FEED_DATA', 'channel': 15, 'data': ['Trade', ['Trade', 'AAPL', 0, 1713384001197, 0, 554671, 'Q', 168.0, -1.38, 14584.0, 19830, 50901210.0, 8609898993.78193, 'ZERO_DOWN', True, 'Trade', 'SPX', 0, 1713389037732, 0, 49202, '', 5022.21, -29.2, 'NaN', 19830, 'NaN', 'NaN', 'ZERO_DOWN', False]]}

[Graeme22]

reviewed code of #140, the _map_message in streamer.py still tries to pop eventType and would fail on the string received sporadically from dxFeed

You're probably right, however #140 actually uses a different streamer websocket so it's worth a shot.

[Graeme22]

Looks like this has to do with the 'dataFormat' being either COMPACT or FULL. If it's still present in the new streamer then I'll fix it ASAP.

[love-reading]

Thank you @Graeme22 handling the compact and full message types will unblock the crash.

Reviewed #140 to look at the new streamer websocket. Tested this end point and it does not support (yet according to tasty) getting market data for indices (e.g SPX)

[Graeme22]

That's problematic! A lot of people are relying on SPX quotes. I think that's sufficient reason to leave this unmerged for now.

[Graeme22]

Hi @love-reading , I think this is actually a separate issue as you detailed. Thanks for opening! Looking into fixing it now.

[Graeme22]

Ok now the data formatting issue should be fixed in #140