Bitfinex Terminal

Bitfinex Terminal offers market data as live data streams and is built on top of Dazaar. It offers first class support for Algo Traders by offering market data over a distributed database to everyone.

There are free and paid market data streams available. They give algo traders and data scientists easy and fast access to our historical trading data. Trading data is easy to download and replicate. It is shared on a P2P network - like BitTorrent - but with a database interface. Also, unlike with BitTorrent, you won't have to download everything until you can use it, streaming time ranges is supported. And if you decide to download more data, e.g. from an earlier timeframe, it will of course preserve the data ordered. Free data comes in a nice B-Tree. Order and funding books are in compressed append-only logs, both guarantee fast access.

Our paid streams give access to historical orderbook data and historical funding book data. Every five seconds we record the data of the funding and order books, which gives an insight into important characterisitcs of the market, i.e. liquidity over time. The data is compressed using an effective algorithm, which allows for fast data transfer and small storage footprints.

Table of Contents

1. How to use it?
2. Support for Algo Traders
3. Example: Get a Full Snapshot of the Free BTCUSD trades, and keep following new updates
4. Example: Query Free Candles Data
5. Example: Access the Paid Orderbooks with USDT
6. Example: Access the Paid Orderbooks with LND
7. Example: Access Historical Funding Book Data with LND
8. Example: Query Local Orderbook Data
9. Example: Download and Query Historical Funding Stats
10. Tutorial: Backtest your Trading Strategies with Bitfinex Terminal & Honey Framework
11. Tutorial: Execute your Trading Strategy with the Honey Framework and Bitfinex Terminal
12. Article: Learn more about Dazaar
13. Video: Screencast for Free Data Streams

Every data stream in Bitfinex Terminal and also Dazaar has a unique id on the network. Imagine it like the url of the data stream. You can directly use ids or you can use Dazaar Cards, which are available for download. We have prepared a few examples that will show how to use it. Bitfinex Terminal currently offers a mix of free and paid data streams:

• Historical Candles Data (free)
• Historical Trades Data (free)
• Historical Funding Stats Data (free)
• Historical Orderbooks Data (paid)
• Historical Fundingbooks Data (paid)

Paid streams will require you to send a specific amount of crypto currency to a defined address. Dazaar will start replicating the whole dataset in the background unless sparse-mode is enabled. That means your database query is resolved with a high priority, while it additionally downloads the whole dataset in the background. For a guided example, see this short how-to guide.

Bitfinex Terminal is also supported by the Honey Framework. You can easily backtest your Honey Framework Strategies on Bitfinex Terminal with the Honey Framework backtesting tools. And you can also decide to trade on your own or to sell the trading signals on the Dazaar network with bfx-hf-strategy-dazaar.

Examples

Also make sure to check our tutorial material :)

Example: Get a Full Snapshot of the BTCUSD trades, and keep following new updates

In our first example we will prepare a full copy of the Bitfinex trades data. We also want to keep following new updates that come in. We will walk through a small set of code, as if we were writing it. The full example can be found at examples/trades-full-copy.js.

As a first step we have to require our dependencies, one of it are the terms of service that we can read at https://github.com/bitfinexcom/bitfinex-terminal/tree/master/terms-of-use:

const dazaar = require('dazaar')
const swarm = require('dazaar/swarm')

const Hyperbee = require('hyperbee')
const keyEncoding = require('bitfinex-terminal-key-encoding')
const terms = require('bitfinex-terminal-terms-of-use')

And we create a Dazaar market. Our data will be stored at dbs/full-trades

const market = dazaar('dbs/full-trades')

Then we download the Dazaar Card for the stream and load it. The option live: true keeps the connection open, so we keep listening for updates after the data is fully synced. If we would set sparse to true, we would just download requested data that we request in a query. This way, we make a full copy. We also accept the Bitfinex Terminal terms of use by loading them into Dazaar, after we have read them:

const card = require('../cards/free/trades/bitfinex.terminal.btcusd.trades.json')
const buyer = dmarket.buy(card, { live: true, sparse: false, terms })

In the next step a lot happens. We register an event listener for the feed event. When it is emitted, we know that the feed is ready for consuming. We also set up a Hyperbee instance, which will provide us a nice interface to access the data. We pass the db instance to a function called doQuery which we will highlight next.

buyer.on('feed', function () {
  console.log('got feed')

  const db = new Hyperbee(buyer.feed, {
    keyEncoding,
    valueEncoding: 'json'
  })

  doQuery(db)
})

The function doQuery will make a request for us. It will request the required data prioritized and print it to our console. We select all trades with a timestamp larger than October 2018, 9:00 UTC and 2019. We also limit the results to 10. The SQL equivalent to this query would be:

SELECT * from trades
WHERE timestamp >= 2018-10-10T09:00:00.000Z
  AND timestamp <= 2019-10-10T09:00:00.000Z
LIMIT 10

This is our query:

function doQuery (db) {
  db.createReadStream({
    gte: { timestamp: new Date('2018-10-10T09:00:00.000Z') },
    lte: { timestamp: new Date('2019-10-10T09:00:00.000Z') },
    limit: 10
  }).on('data', (d) => {
    console.log(d)
  })
}

To start everything, we have to join the P2P swarm:

swarm(buyer)

And for demonstration purposes we also log the downloaded elements, so we can see the progress:

setInterval(() => {
  if (!buyer.feed) return

  console.log('data feed length is', buyer.feed.length, 'elements')
}, 1000)

This example code will download the whole BTCUSD trades dataset from Bitfinex Terminal. If the connection is reset, it will just resume where it stopped. We also make a request for a timerange and print it to the console. The timerange is prioritized and downloaded first. When all downloading is finished, we keep the socket open to receive the latest updates. That is a lot that is done for us behind the scenes.

The full example can be found at examples/candles-sparse-select.js. Our setup is similar to the trades example, we require our dependencies and set up a database that is stored on disk:

const dazaar = require('dazaar')
const swarm = require('dazaar/swarm')

const Hyperbee = require('hyperbee')
const keyEncoding = require('bitfinex-terminal-key-encoding')

const market = dazaar('dbs/sparse-candles')

This time we load a Dazaar Card for candles. We also enable the sparse mode, that means, just the data we directly request is downloaded. We also accept the Bitfinex Terminal terms of use by loading module for it into Dazaar:

const card = require('../cards/free/candles/bitfinex.terminal.btcusd.candles.json')
const terms = require('bitfinex-terminal-terms-of-use')
const buyer = dmarket.buy(card, { sparse: true, terms })

Our event listener looks exactly the same as the one from the previous example:

buyer.on('feed', function () {
  console.log('got feed')

  const db = new Hyperbee(buyer.feed, {
    keyEncoding,
    valueEncoding: 'json'
  })

  doQuery(db)
})

For our query we define a timeframe and the candle type. We also reverse the results, so we should get the newest entries first. The equivalent SQL query is:

SELECT * from candles
WHERE candle = '5m'
 AND timestamp >= 2018-10-10T09:00:00.000Z
 AND timestamp <= 2019-10-10T09:00:00.000Z
LIMIT 10
ORDER BY timestamp DESC

Here is our query:

function doQuery (db) {
  db.createReadStream({
    gte: { candle: '5m', timestamp: new Date('2018-10-10T09:00:00.000Z') },
    lte: { candle: '5m', timestamp: new Date('2019-10-10T09:00:00.000Z') },
    limit: 10,
    reverse: true
  }).on('data', (d) => {
    console.log(d)
  })
}

In this example we did a sparse-enabled select on the data, so just the data we selected is downloaded. The full example can be found in examples/candles-sparse-select.js.

wget https://github.com/bitfinexcom/bitfinex-terminal/tree/master/cards/paid/orderbooks/bitfinex.terminal.btcusd.orderbook.json

npm install dazaar @dazaar/payment-eth \
  bitfinex-terminal-order-book bitfinex-terminal-terms-of-use

One particular module we install is the terms of use module. You can read the terms of use here: https://github.com/bitfinexcom/bitfinex-terminal/tree/master/terms-of-use. Later we will pass that module to the dazaar market to accept the terms.

We create a file called test-orderbooks.js and load the modules we just installed with npm:

const dazaar = require('dazaar')
const swarm = require('dazaar/swarm')
const Payment = require('@dazaar/payment-eth')
const terms = require('bitfinex-terminal-terms-of-use')

const Orderbook = require('bitfinex-terminal-order-book')

We also load the Dazaar Card of of the data stream:

const card = require('./bitfinex.terminal.btcusd.orderbook.json')

When we open the file bitfinex.terminal.btcusd.orderbook.json we can see the Dazaar Card. The id field refers to the data stream, and the payment section gives instructions for payment. You can see that it requires 2 cents USDT for one hour of access:

  "amount": 0.02,
  "unit": "hours",
  "interval": 1

Similar to Free Bitfinex Terminal data we set up a database and load the card. We also accept the terms of use by passing them as an argument:

const market = dazaar('./dbs/terminal-orderbook-btcusd')
const buyer = market.buy(card, { sparse: false, terms })

When our local client is bootstrapped, we print the payment informations to the console:

buyer.ready(function () {
  console.log('Pay to ' + Payment.tweak(buyer.key, card))

  swarm(buyer)
})

It will print something like Pay to 0xaf2333... to the console. When we pay 2 cents USDT to that address, we get one hour of access to the stream and our replication starts. But for now, we just continue to write our example program.

When we made the payment, as soon as our payment is received, the feed event is triggered, and we start replicating the data. In case we have already payed, the feed event is triggered, too. For example purposes, we also do a database query with the orderbook query api:

buyer.on('feed', function () {
  console.log('got feed')

  const o = new Orderbook(buyer.feed)
  doQuery(o)
})

Our query function will fetch the entries of the last 5 minutes. It is the equivalent to the SQL query SELECT * from <table> WHERE timestamp > $TIMESTAMP_5_MINUTES_AGO. The option live keeps the stream open, so each new entry is printed to the console:

async function doQuery (o) {
  const start = new Date().getTime() - (1000 * 60 * 5)

  const s = o.createReadStream({ start, live: true })

  for await (const data of s) {
    console.log('--->', data)
  }
}

When we now start or program with node test-orderbooks.js it will print an ETH address to the console. When we pay 0.02 USDT to it, the streaming of data will start. Additionally older data is replicated in the background, for later use. When we want to consume more data, we can pay any time. If we pay before the hour ends, it will be added on top of our available time.

The full code example can be found in examples/payed-orderbooks.js.

wget https://github.com/bitfinexcom/bitfinex-terminal/tree/master/cards/paid/orderbooks/bitfinex.terminal.btcusd.orderbook.json

{
  "method": "Lightning",
  "currency": "SATS",
  "amount": 100,
  "unit": "hours",
  "interval": 1,
  "minSeconds": 1
}

npm install dazaar @dazaar/payment-lightning  \
  bitfinex-terminal-order-book bitfinex-terminal-terms-of-use \
  hyperbee

const dazaar = require('dazaar')
const swarm = require('dazaar/swarm')
const Payment = require('@dazaar/payment-lightning')
const terms = require('bitfinex-terminal-terms-of-use')
const Orderbook = require('bitfinex-terminal-order-book')

One important module is the terms of use module bitfinex-terminal-terms-of-use. You can read the terms of use here: https://github.com/bitfinexcom/bitfinex-terminal/tree/master/terms-of-use. If you agree with the terms, we can continue to code and pass the module content later to the Dazaar market.

We also have to load our Dazaar Card:

const card = require('./bitfinex.terminal.btcusd.orderbook.json')

Similar to the previous examples, we set up a database:

const market = dazaar('dbs/terminal-orderbook-btcusd-lnd')

And we also initiate the Dazaar market. Please note that you have to insert the terms of use here, if you agree with them:

const buyer = market.buy(card, { sparse: false, terms })

When our market setup from above is finished, the ready function is called as a callback. In the next step we request an invoice for 100 satoshi, which will give us an hour to access and download data. The invoice is printed to the console, and can be paid with any Lightning wallet:

buyer.ready(function () {
  console.log('ready')

  const payment = new Payment(buyer)
  payment.requestInvoice(100, function (err, invoice) {
    if (err) console.log(err)
    console.log(invoice)
  })

  swarm(buyer)
})

When the payment has arrived its destination, the stream becomes available for Dazaar. On our side a feed event is emitted, and we know that we can send our database queries. Of course the feed event is also emitted when we have made a payment in the past that is still valid.

buyer.on('feed', function () {
  console.log('got feed')

  const o = new Orderbook(buyer.feed)
  doQuery(o)
})

Our query will be very simple, the SQL equivalent is:

SELECT * from orderbooks
WHERE timestamp > $TIMESTAMP_10_MINUTES_AGO

In addition, we keep the stream open and listen for ne data coming in with live: true. There are other query modifier available for more complex queries, namely end and limit.

limit will limit the result set to the defined amount and with end a we can set the upper bound for our timeseries query.

async function doQuery (o) {
  const s = o.createReadStream({
    start: new Date().getTime() - (1000 * 60 * 10),
    live: true
  })

  for await (const data of s) {
    console.log('--->', data)
  }
}

When we now run the code, which is also located at examples/payed-orderbooks-lnd.js, it will print a Lightning invoice.

Once we pay the invoice, we get access, the query is exectuted and the results printed to the console. As sparse mode is set to false, the db also replicates the whole dataset in the background.

The full code example can be found in examples/payed-orderbooks-lnd.js.

Of course its possible to query the local copy of the orderbook data you've paid for. In essence, we'll just remove the part where we connect to the servers. That means, we will modify examples/payed-orderbooks.js a little bit.

As we don't connect to Dazaar network we need to require less modules:

const dazaar = require('dazaar')
const terms = require('bitfinex-terminal-terms-of-use')

const Orderbook = require('bitfinex-terminal-order-book')

As in the previous example, we require our card and setup Dazaar. Note that the used database path dbs/terminal-orderbook-btcusd is equal to the one used in the previous example, where we paid and downloaded data using the ETH payment provider:

const card = require('../cards/paid/orderbooks/bitfinex.terminal.btcusd.orderbook.json')
const market = dazaar('dbs/terminal-orderbook-btcusd')
const buyer = market.buy(card, { sparse: false, terms })

When the buyer.ready callback is called, we initialise the database query API and run our query:

buyer.ready(function () {
  const o = new Orderbook(buyer.feed)
  doQuery(o)
})

Our query is the equivalent of a SELECT * from <table> LIMIT 5 query in SQL:

async function doQuery (o) {
  const s = o.createReadStream({ limit: 5 })

  for await (const data of s) {
    console.log('--->', data)
  }
}

Lets say we want to do a range query, with a max of 5 results. The equivalent in SQL would be:

SELECT * from <table> WHERE timestamp > $TIMESTAMP_5_MINUTES_AGO AND timestamp < $NOW LIMIT 5

We would replace our query in doQuery in this way:

const start = new Date().getTime() - (1000 * 60 * 5)
const end = Date.now()

const s = o.createReadStream({ limit: 5, start, end  })

The full example can be found at examples/payed-orderbook-local.js.

Example: Access Historical Funding Book Data with LND

npm install dazaar @dazaar/payment-lightning  \
  bitfinex-terminal-funding-book-encoding bitfinex-terminal-terms-of-use \
  hyperbee

wget https://github.com/bitfinexcom/bitfinex-terminal/tree/master/cards/paid/funding/bitfinex.terminal.funding.books.json

const dazaar = require('dazaar')
const swarm = require('dazaar/swarm')
const Hyperbee = require('hyperbee')
const Payment = require('@dazaar/payment-lightning')
const { keyEncoding, valueEncoding } = require('bitfinex-terminal-funding-book-encoding')

const terms = require('bitfinex-terminal-terms-of-use')

One important module is the module bitfinex-terminal-terms-of-use. You can read the terms of use here: https://github.com/bitfinexcom/bitfinex-terminal/tree/master/terms-of-use. Just when you agree with the terms you can continue with the tutorial. In order to agree, we have to pass the terms into the Dazaar buy function later.

Similar to previous examples we require the card and set up the database. We will query the USD funding book. This time we enable sparse mode. That means, just the data we request in a query is downlaoded. Sparse mode gives faster results to ad-hoc queries, but it does not download the other data that is not part of the query in the background for us. Please also note that we are passing the terms of use here to accept them.

const card = require('../cards/paid/funding/bitfinex.terminal.funding.books.json')

const market = dazaar('dbs/terminal-fundingbook-usd-lnd')
const buyer = market.buy(card, { sparse: true, terms })

The next step is to request an invoice for payment. It is printed to the console:

buyer.ready(function () {
  console.log('ready')

  const payment = new Payment(buyer)
  payment.requestInvoice(100, function (err, invoice) {
    if (err) console.log(err)
    console.log(invoice)
  })

  swarm(buyer)
})

Once payment has happened and there is still time left to download and query data, the feed event is emitted, and we can query the data:

buyer.on('feed', function () {
  const db = new Hyperbee(buyer.feed, {
    keyEncoding,
    valueEncoding
  })

  doQuery(db)
})

The function doQuery runs the actual query of the data that also replicates the data that is replicated. As we are in sparse-mode, the data not related to our query is not replicated in the background:

function doQuery (db) {
  db.sub('USD').createReadStream({
    limit: 10,
    reverse: true
  }).on('data', (d) => {
    console.log(JSON.stringify(d, null, 2))
  })
}

You can find the full example in examples/payed-fundingbooks-lnd.js.

Example: Download and Query Historical Funding Stats

npm install dazaar bitfinex-terminal-terms-of-use \
  hyperbee bitfinex-terminal-funding-encoding

const dazaar = require('dazaar')
const swarm = require('dazaar/swarm')

const Hyperbee = require('hyperbee')

const terms = require('bitfinex-terminal-terms-of-use')

const { keyEncoding, valueEncoding } = require('bitfinex-terminal-funding-encoding')

const market = dazaar('dbs/funding-stats')

We require the Dazaar Card which points to the data stream. In case you wonder, the card is available here:

const card = require('../cards/free/funding-stats/bitfinex.terminal.funding.stats.json')

To access the data we have to accept the Terms of Service. You can read them here. If you agree with the terms, you pass them together with the Dazaar Card as an argument to the market.buy function. Please note that we also set sparse to false, so the whole database will get replicated in the background. In case we would set sparse to true, just the data resolving from our query would be replicated.

const buyer = market.buy(card, { sparse: false, terms })

As next steps we are going to initiate the database and run our query. When we connect to the remote peers, a feed event is emitted. Starting from that point we can query the database using hyperbee:

buyer.on('feed', function () {
  console.log('got feed')

  const db = new Hyperbee(buyer.feed, {
    keyEncoding,
    valueEncoding
  })

  doQuery(db)
})

The Funding Stats database uses sub-databases. For each funding pair there is a sub-database that can be accessed using the db.sub() command. The doQuery function will query funding stats for USD. We will query the last three entries for the timeframe between the third of January and the fifth of January 2021, 9 AM. The SQL quivalent for the query we run is:

SELECT * from funding_stats
WHERE f_currency = 'USD'
  AND datetime >= '2021-01-03T09:00:00.000Z'
  AND datetime <= '2021-01-05T09:00:00.000Z'
LIMIT 3
ORDER BY datetime DESC

Our hyperbee query looks like this, it logs the results to the console:

function doQuery (db) {
  db.sub('USD').createReadStream({
    gte: new Date('2021-01-03T09:00:00.000Z'),
    lte: new Date('2021-01-05T09:00:00.000Z'),
    limit: 3,
    reverse: true
  }).on('data', (d) => {
    console.log(d)
  })
}

The last step for receiving the funding stats is to connect to the remote peers, and starting to download and query the data:

swarm(buyer)

That's it! When you run the code, it will download the whole funding stats database in the background. In addition it runs a query for USD funding stats to select a few data entries. You can find the full code in examples/funding-stats.js.

Tutorials

Our tutorials can be found in the folder articles.

• Tutorial: Backtest your Trading Strategies with Bitfinex Terminal & Honey Framework
• Tutorial: Execute your Trading Strategy with the Honey Framework and Bitfinex Terminal

Articles

• Introducing Dazaar

Videos

• Video: Screencast for Free Data Streams