[Bug] Some websocket server responses no longer include exchange or mic code

[stephenrs]

Describe the bug

Recent changes to the “subscribe-status” event that the websocket server emits cause the event to no longer contain the mic code. In the past, the event included the mic code, but suddenly it stopped including it about 10 days ago.

So, any websocket implementations that rely on the presence of mic code to resolve duplicates and symbol ambiguity when managing websocket connections are currently non-functional.

There is currently no way to guarantee symbol uniqueness in a system that respects the subscribe-status event.

This might affect other websocket server responses, but I have only tested the subscribe-status event.

To Reproduce

Subscribe to the websocket by sending a message like this:

{"action": "subscribe", "params": {"symbols": [{"symbol": "NSEI", "mic_code": "XNSE"}]}}

Log the subscribe-status response, which will look like this:

{“event”:”subscribe-status”,”status”:”ok”,”success”:[{“symbol”:”NSEI”,”type”:”INDEX”}],”fails”:null}

As you can see, the subscribe-status response does not include the mic code.

Expected behavior

The subscribe-status event should include the mic code to eliminate the possibility of ambiguity.

mic code was previously included in the event, but for some reason, it now is not included.

However, it should be noted that this is inconsistent with websocket price updates, which include mic code, for example:

{"event":"price","symbol":"NSEI","currency":"INR","exchange":"NSE","mic_code":"XNSE","type":"Index","timestamp":1695790395,"price":19569.25}

Additional context

This was previously discussed and resolved here: https://github.com/twelvedata/twelvedata-python/issues/62 and here: https://github.com/twelvedata/twelvedata-python/issues/61

A similar issue has been raised here: https://github.com/twelvedata/twelvedata-python/issues/68

It appears that guaranteeing symbol uniqueness just isn't a priority for the dev team, unfortunately.

[stephenrs]

Has anyone taken note of this problem?

I would really appreciate some information about this, I have been without websocket updates for almost 2 weeks. I first reported the problem through your help system more than a week ago, but I still have not received any useful information about the status. This was an urgent problem two weeks ago, so is extremely urgent now.

So, it would be great if you can let me know the following:

1. Have you confirmed the issue on your end?
2. If so, are you planning to fix it?
3. If so, when?
4. If not, why?

Thanks!

[stephenrs]

I guess this is not going to get fixed.

[midasSSS]

Indices do not contain mic_code in response. Other instrument types do, for instance, equities.

[stephenrs]

Indices do not contain mic_code in response. Other instrument types do, for instance, equities.

Why did you introduce this inconsistency? It breaks existing websocket implementations that rely on having a consistent way to process and manage symbols. What benefit does it serve?

This just makes the twelvedata websocket server harder to work with and less reliable. Guaranteeing the uniqueness of symbols is a high priority for a system like this, otherwise you're introducing the possibility of random problems for no reason.

For example, in my implementation which has been live for more than 8 months, in order to adapt to this change I have to write additional code to handle non-index symbols differently than index symbols. This will make the code more complex and brittle than it was before. A large system that subscribes to websocket feeds shouldn't know or care what type of symbol it is receiving data for.

Also, even if I make this change to the code, there will still be no way to guarantee uniqueness with index symbols. So, this is still an active bug in the websocket implementation.

EDIT: Is it too hard to add back the exchange and mic code to the subscribe-status event? It's hard to understand why you would want to make your product worse rather than better.

[midasSSS]

This inconsistency is related to the fact that indices do not trade; therefore, it would be misleading to indicate MIC codes for this category.

[stephenrs]

This inconsistency is related to the fact that indices do not trade; therefore, it would be misleading to indicate MIC codes for this category.

Whether or not they trade isn't relevant in this context. Indices are all tied to an exchange....which is why exchange and mic code are listed here: https://api.twelvedata.com/indices?show_plan=true

It's much more important that your dataset is consistent than whether or not an instrument trades.

EDIT: Another way to think about this is to focus on who publishes the index data, rather than on whether or not it trades. It's the same for all index markets around the world. For example, the NSE exchange publishes NIFTY index data, and the CBOE publishes S&P (SPX) data.

So, the trading aspect just isn't relevant and it's not at all misleading to associate indexes with exchanges - it's a reflection of the market data reality.

[stephenrs]

Just trying to be helpful...

So, we can think of a symbol specification like "XNSE:NSEI" as an indication of [SOURCE]:[INSTRUMENT]

We are data processors, and it's not our job to tell people anything about whether or not they can trade a symbol. Our job is to tell them where the data came from (the SOURCE), and what the data represents (the INSTRUMENT), and nothing more.

So, the Twelvedata NIFTY (NSEI) index data is coming from the XNSE source, and downstream consumers of the API and the Websockets need to know that information.

[stephenrs]

One more thing to try to be helpful...

All of this points to the reason why using MIC:SYMBOL is the absolute best way to refer market data - because it means you will never have to worry about duplicates, because MIC is universally unique globally. So, another example...

The dollar index is the most popular index in the world and everybody knows it as DXY. However, there are several different versions of the DXY that come from different sources. The ICE exchange has a DXY, Dow Jones/FXCM has a DXY, and there are some custom versions of the DXY like TVC:DXY.

Symbols are duplicated all the time, and so are exchange codes, but MIC codes are not. Tradingview is one of the most popular platforms in the world and they have 4 different DXY instruments from 4 different sources, as you can see in the attached screenshot.

So again, the MIC code designation in a symbol has nothing to do with how or whether an instrument can be traded. It's intended to be used purely as a description of the origin of the data.

EDIT: Now, imagine that you are Tradingview...if your data vendor isn't telling you the source of the DXY index data you're buying from them, then how are you going to make your platform work properly?

EDIT: Just in case it's not yet clear...

• Client System: "Hey Twelvedata, thanks for the subscription-status message, so I know the websocket subscription for DXY is confirmed, but which source is the data coming from?"

• Twelvedata: "Don't worry about it, I told you the type is 'INDEX'"

• Client System: "But I have 4 DXY subscriptions pending, for 4 different sources, and all of them are type 'INDEX'. Can't you just tell me what the source of the data is?"

• Twelvedata: "No, that's all I'm willing to tell you about this."

• Client System: "Come on man, each one of these pending DXY subscriptions will trigger pending actions that are waiting for the subscription-status message, and I don't know which actions to trigger."

• Twelvedata: "Sorry. Good luck."

[stephenrs]

I just went back and read this old issue https://github.com/twelvedata/twelvedata-python/issues/61, and I realize that I've been trying to make this same point for almost a year.

I don't mean to be rude, but I guess I just don't understand why you refuse to keep consistency in your API, and why you insist on trying to use exchange codes to reference symbols when this approach won't work reliably (As noted in #61, there are 5 exchanges with the code BSE, so how are you going to resolve duplicates?)...and now you've decided to even get rid of the exchange codes for indices.

You have to make sure access to data is consistent everywhere in your system and that it guarantees uniqueness - so API and Websocket consumers know that they will always get exactly the data they requested. Using symbol and type is obviously not enough. Using symbol and exchange is better, but it's still not enough. The only way to provide this guarantee is to use symbol and MIC code...and it's such a simple thing to do.

Again, I don't mean to be rude, but this is a bit frustrating and it has cost me a lot of time. I'm not sure why this very important point isn't understood yet. Can you explain? Do you disagree?

[stephenrs]

Also, for reference, most traders already know how indexes work and that you can't trade them directly. And newer traders who don't already know this will find out quickly from their broker when they try to trade one....and their broker might offer them ways to trade an index indirectly as described here: https://www.investopedia.com/ask/answers/how-to-invest-in-an-index/

So this is just not something that a data vendor should be thinking about at all.

[stephenrs]

Indices do not contain mic_code in response. Other instrument types do, for instance, equities.

It turns out that this is not true, unfortunately. Here are some examples of the subscribe-status event for equities that don't include any exchange/mic source information:

{"event":"subscribe-status","status":"ok","success":[{"symbol":"APC","type":"COMMON_STOCK"}],"fails":null}

{"event":"subscribe-status","status":"ok","success":[{"symbol":"AMZ","type":"COMMON_STOCK"}],"fails":null}

{"event":"subscribe-status","status":"ok","success":[{"symbol":"1HTA","type":"COMMON_STOCK"}],"fails":null}

So, this is an even bigger problem than I thought it was and the subscribe-status event is essentially useless, unfortunately.

[stephenrs]

Well, I've done my best to give some useful information, but I guess you guys just don't care, and it seems you don't care much about the quality of your product.