Kraken only receives one ticker update, then nothing afterwards

evan-coygo commented 4 years ago

Exchange Kraken

Subscription type Ticker

Describe the bug Only one update is received, no subsequent updates occur even though the data is updating on the exchange.

To Reproduce

   const ccxws = require('ccxws');
    const client = new ccxws.kraken({
      reconnectIntervalMs: 15 * 60 * 1000
    });
    client.on('ticker', (ticker, market) => {
      console.log('ccxws TICKER update, market=', market, 'ticker=', ticker);
    });

    // give it a moment to load the symbols map internally
    await timeout(3000);
    const ccxwsMarket = { id: 'XXMRXXBT', base: 'XMR', quote: 'BTC' };
    client.subscribeTicker(ccxwsMarket);

Expected behavior A new ticker update event is received on every change of the data. Currently only the first ticker is received when I run the example above.

Additional context I added some logs to the ccxws kraken-client processMsg() method, and it appears to only be receiving heartbeat messages after the first initial ticker payload. This would imply that the exchange itself isn't sending the data, but I find that hard to believe. Kraken API status page shows all systems functional.

bmancini55 commented 4 years ago

A higher volume market such as { id: "XXBTZUSD", base: "BTC", quote: "USD" } is sending updates. The XMR/BTC market may just not have any top of book bid/ask activity.

evan-coygo commented 4 years ago

here's a screencap of running that example above w/the kraken UI in a browser, watch the top of book changes. ask changes between 0.006877, 0.006878, 0.006879 but no events are emitted by ccxws

ccxws kraken

bmancini55 commented 4 years ago

From what you've describe, it sounds like the Kraken ticker API doesn't send down data on order book changes, but on ticks (trade executions). The API docs aren't clear on how best bid and ask are updated. For ref: https://docs.kraken.com/websockets/#message-subscribe

I would do what you did and add some logging to the raw socket stream. If Kraken isn't sending intermittent updates then there isn't much we can do from a library perspective.

evan-coygo commented 4 years ago

Yes it looks like you're right, this behavior differs depending on the exchange unfortunately. This is what I've found, in case it's helpful to others:

Sends ticker events on every match (trade), not when order book updates:

Coinbase Pro
Kraken

Sends ticker event on every ask/bid update in order book:

Binance
Bittrex
Gemini

Kraken DOES suppoort getting live top of book updates, but that uses their spread feed, not the ticker feed.

@bmancini55 I'm on a journey to access live tickers on each exchange right now so I can help contribute back to this lib if you're willing to accept. I've just added a new PR to update the Gemini ticker implementation I added a few months ago to support live tickers, and to add support for the Kraken spread API here https://github.com/altangent/ccxws/pull/147/files

bmancini55 commented 4 years ago

Thanks for doing an audit of that functionality!

Would you mind splitting out the Kraken changes into a separate PR?

evan-coygo commented 4 years ago

@bmancini55 moved into #150 and #151 for Gemini and Kraken respectively

evan-coygo commented 4 years ago

@bmancini55 can we get this closed out and PR merged? I've been using the branch in my fork for a couple weeks now and it's all working smoothly

bmancini55 commented 4 years ago

Sorry for the delay, other priorities were pressing.

The big hold up with the PR is that I'm disinclined to start including one-off and especially monkey-patched features into the library. I'm not saying this functionality shouldn't be added, but I'm hesitant to add bloat the API, especially in a way that doesn't consider the other 20+ exchanges.

As you've documented above, we have a few scenarios at play here:

Some exchanges emit an update on each tick only
Some exchanges emit an update on each tick or top of book change
Some exchanges have exclusive top of book feeds

Some of my thoughts:

Ticker is a poorly chosen name for the ticker event on some exchanges and this name carried through to CCXWS. To me this implies that it changes on ticks. If it's changing with top of book updates, this is an aggregated Level 1 feed, or a market snapshot feed.
This library was originally written to standardize the lower level raw data feeds. For example, connect to trade and orderbook streams. Point being, not a lot of consideration was given toward tickers, spread, or level 1 data. I digress.
A spread feed could be a nice addition to the library as it's provides a way to get a raw top of book information, at least for the exchanges that support that feed. It presents some challenges though:
- For exchanges that don't support a spread feed it might be mean two things:
  - If the exchanges has a "ticker" feed that does send top of book updates, the "ticker" feed could be tapped to emit "spread" events
  - What if there is not "ticker" feed? Do we just use the l2snapshot or l2update feeds to internally maintain a "spread"?
- Do we even really care about a "raw" spread feed at this level?
A more appropriate change might be modifying the definition of the existing "ticker" functionality to include "spread" updates.
- This would really only impact exchanges that only do tick updates and have a separate spread feed. Binance may be one of these (https://github.com/binance-exchange/binance-official-api-docs/blob/master/web-socket-streams.md#individual-symbol-book-ticker-streams)
- I think there is an argument to be made for making ticker synonymous with "level1" information as many exchanges already treat their "ticker" feeds as a "level1" market info feed.
- The "ticker" event could deprecated in favor of the new "level1" event, which is more consistent with the information that is displayed on websites and market data nomenclature.

evan-coygo commented 4 years ago

@bmancini55 yeah I understand the hesitation. Here are my thoughts:

Seperating "ticker" and "spread"

I think splitting "ticker" and "spread" (or perhaps "topOfBook"?) into their own defined feeds in the library's public API might be the most appropriate action. The functionality differs between exchanges and I think it'd be good to specify what you're subscribing to so that you don't run into confusing scenarios where you're expecting a "ticker" event to only be emitted upon fill events, like if you're using "ticker" to gather data for your own candlestick charts.

Ideally it would be a complete Level 1 event, like client.on("l1update"), but from memory I don't think many exchanges that emit these events on every top of book change include the "last price" and "last size" that are part of a Level 1 definition. So either "spread' or "topOfBook" would seem most appropriate to me.

Potential issues

One issue w/a new event type is that you'd run into backwards-compatibility issues for consumers already using the library who are relying on those exchanges that emit "ticker" on every top of book change instead of only on fills like a "ticker" really should be. I think if you're creating a new event type, you should really make sure that you're conforming to the correct event on each exchange, so breaking backwards compatibility is the correct move here IMO.

In the interim

With all of that in mind, especially breaking backwards compatibility, it seems like the "ideal" solution should be delayed until the next major release that you've been discussing, a "ccxws 2.0" that will make major changes to the internal and probably public APIs. Since that is probably going to take awhile, I'd really like if we could get this merged in and released in the meantime since it does add useful functionality and there already concessions in other parts of the codebase. That seems like the most pragmatic approach in the interim, but I suppose I'm biased since it'd make my life easier no longer pointing to a fork in our builds :smile:

nudabagana commented 3 years ago

Any updates? Will #151 Kraken fix be merged? Not sure if I should wait or just drop the lib and implement kraken ws myself.

bmancini55 commented 3 years ago

This issue is on the list to address in the next few weeks. PR #151 has some issues that were mentioned above, so it will not be merged as-is.

evan-coygo commented 3 years ago

@nudabagana I ended up just maintaining a full order book via the l2snapshot and l2update events. When you have a full order book you can get real-time updates to any level of the order book, including top-level ask and bid spread

nudabagana commented 3 years ago

@evan-coygo That's what I tried. But order book desyncs after 5-10 minutes. I get all new orders, but some old orders remain. It seems that sometime it doesn't send orders with 0 amount. (I'm also using cex/binance/bitmex/bitstamp and they all work fine, only kraken has this problem)

bmancini55 commented 3 years ago

@nudabagana the Kraken order book issue has been reported in the past and we thought it was addressed but there may have been a regressioon introduced with #155. I'll create another issue to track the issue there as an immediate priority.

@evan-coygo if you've found a suitable workaround for the ticker issue, I'd like to close this issue and PR. I'm also happy to create a new feature issue to explore either a top of book event or wiring it into the ticker event for the exchanges where it is supported.

evan-coygo commented 3 years ago

yes I'll go ahead and close it

altangent / ccxws