Kraken order book goes out of sync

[bmancini55]

Exchange Kraken

Subscription type L2 order book

Describe the bug Reported in #142, possibly caused by #155. Kraken order books go out of sync after 5-10 minutes. This may be caused by dropping updates in the update stream. Per Kraken documentation, the stream includes tuples of (\<price>, \<volume>, \<timestamp>). The price attribute may be duplicated for a single update event.

Need to perform some research as to why the OB goes out of sync. It is unclear if the values in a single update are additive or replace the existing values. The existing logic in the library assumes that the last timestamp is the most recent cumulative volume at the price point.

[nudabagana]

Made minimal reproduction in CodeSandbox: https://codesandbox.io/s/node-js-o9xko?file=/src/index.js

[bmancini55]

Finally had some time to spend dig into this today. I was able to reproduce the issue and add debugging to see why there are checksum failures. I will post a sandbox with the reproduction code before applying the fix.

A few minor fixes should enable order books to run correctly.

• CCXWS needs to broadcast the timestamp per update point
  • Points should only be updated when the timestamp is newer than the existing point
  • I've captured out of order updates that would invalidate newer updates, so checking the timestamp before updating a point is important
  • CCXWS currently sets the update timestamp to the maximum timestamp point in the update, which provides no value at all
• CCXWS needs to broadcast the checksum per update, this one is obvious
• CCXWS is ONLY sending the first of several possible update objects that can happen in the same event. This is obviously problematic in that entire update blocks are missed. For example, update 1 is how most come in, but update 2 has two blocks:

[270,{"b":[["11260.50000","0.00000000","1596221402.104952"],["11228.70000","2.60111463","1596221103.546084","r"]],"c":"1281654047"},"book-100","XBT/USD"]

[270,{"a":[["11277.30000","1.01949833","1596221402.163693"]]},{"b":[["11275.30000","0.17300000","1596221402.163680"]],"c":"1036980588"},"book-100","XBT/USD"]

[bmancini55]

Fix has been released in https://github.com/altangent/ccxws/releases/tag/v0.36.0

BTW, I added a prototype order book implementation ported from our internal Go library. We have tentative plans to add order books to this library and figured this was a good time to start that work. Because it's experimental it's not exported in the main API yet, but you give it a try with the checksum validation using the code below:

/* eslint-disable no-console */
const ccxws = require("ccxws");
const KrakenOrderBook = require("ccxws/src/orderbooks/KrakenOrderBook");
const KrakenOrderBookPoint = require("ccxws/src/orderbooks/KrakenOrderBookPoint");
let ob;

const client = new ccxws.kraken();
let market = { id: "XXBTZUSD", base: "BTC", quote: "USD" };
client.subscribeLevel2Updates(market);

client.on("l2snapshot", (snap) => {
  let asks = snap.asks.map(p => new KrakenOrderBookPoint(p.price, p.size, p.timestamp));
  let bids = snap.bids.map(p => new KrakenOrderBookPoint(p.price, p.size, p.timestamp));
  ob = new KrakenOrderBook(asks, bids);
});

client.on("l2update", (update, _, msg) => {
  console.log(JSON.stringify(msg));

  const before = ob.snapshot(10);
  let expectedChecksum = update.checksum;

  for (let { price, size, timestamp } of update.asks) {
    ob.update(false, price, size, timestamp);
  }

  for (let { price, size, timestamp } of update.bids) {
    ob.update(true, price, size, timestamp);
  }

  const checksum = ob.checksum();
  if (checksum !== expectedChecksum) {
    const after = ob.snapshot(10);
    console.log(checksum, expectedChecksum);
    console.log(JSON.stringify(before, 0, null));
    console.log(JSON.stringify(after, 0, null));
    process.exit(1);
  }
});