update Dockerfile

[dmitry-ee]

• Make result image way smaller (from 924m to 57m)
• Faster docker build (from 70s to 26s)
• Proper SIG handling (if you are trying to make docker kill or docker rm, container will hang) via tini (mostly because node shouldn't run with pid=1)

[dmitry-ee]

Hey @Tucsky, finally i made things work. Docker CMD isn't really good way to pass environment variables, so i decided to add full support for them inside config.js.

After start it will look like this:

PID:  6
[init] reading config.json...
overriding 'port' to '3000' via env 'PORT'
overriding 'storage' to 'influx' via env 'STORAGE'
overriding 'influxUrl' to 'localhost:5000' via env 'INFLUX_URL'
overriding 'filesLocation' to '/app/data' via env 'FILES_LOCATION'
[storage] Using "influx" storage solution

Hope it's nice solution for you

[Tucsky]

Fantastic work sir, Im not using docker myself but Ive tested it, and it works pretty nicely so Im just going to trust you and approve the PR :)

[Tucsky]

Do you use this branch for anything in particular ? Collecting or broadcasting trade data ? Im just curious... Asking because Im planning to merge this https://github.com/Tucsky/SignificantTrades/tree/feature/multiserver into feature/server soon (feedback appreciated 😄 ) This will allow to watch multiple pair on 1 instance, major changes including the pair dimension that will reference the remote pair by default in the storage. Say you watch BTCUSD,BTCUSDT,BTCUSD-PERPETUAL it will create three files (instead of "normalizing" the pair to BTCUSD like today). There will be a mapping option tho. Just letting u know, as I am not aware of many ppl using it but here you are.

[dmitry-ee]

Nice to hear that! While ago i did custom docker-wrapper around branch feature/server and using it for data collecting (i have another app which listens to all websockets and sends filtered data to kafka and then to elasticsearch). It would be amazing if there's would be one application for all of this work.

Regarding branch multiserver: Amazing work! Is it would be your main server branch?

I can help you to resolve environment mapping and docker builds :)

[Tucsky]

filtered data to kafka and then to elasticsearch would be amazing if there's would be one application for all of this work.

Well it definetly can, did you try writing an adapter for that yet ? (like an src/storage/kafka.js KaftaStorage or src/storage/elasticsearch.js) Or you currently using an external tool the use the files created by the FileStorage maybe? Ive zero clue what kafka is 🤷‍♂️

Yeah Im going to make another repo for the server, once I find the multiserver branch to be stable enough because its a big mess rn with all the branches. Still lot of room for issues with this new branch, there is so many pairs! Ive added kraken futures and huobi futures also could be some bad amount calculations around contract values.

I can help you to resolve environment mapping and docker builds :)

This is what Im talking about in config.json

{
  "mapping": {
    "BTCUSD": "(?:XBT|BTC)-?USDT?-?(?:PERPETUAL)",
    "ETHUSD": "ETH-?USDT?-?(?:PERPETUAL)"
  },
  "pairs": "XBTUSD,BTCUSDT,BTCUSDT-PERPETUAL,ETHUSD,ETHUSDT,ETHUSDT-PERPETUAL"
}

There is actually 3 possible names for exemple take "BTCUSDT-PERPETUAL" in this example the mappedPair is "BTCUSD" (as per the mapping regexp) the localPair is "BTCUSDT-PERPETUAL" (this symbol does not exists in binance, I added -PERPETUAL to be able to distinguish the btcusdt spot symbol and btcusdt futures) the remotePair (the one that is passed to the ws subscription) is "btcusdt"

I only did that for binance tho. All the other exchanges have different symbol name (that are unique for spot + futures + swap combined)

The localPair should use same naming convention for all exchanges pair-SWAP instead of -PERPETUAL or -PERP then use symbol-THIS_WEEK symbol-NEXT_WEEK symbol-QUARTER symbol-BI_QUARTER symbol-THIS_MONTH for the futures contracts the localPair -> remotePair thing is done in the formatProduct method within each exchange

[dmitry-ee]

Or you currently using an external tool the use the files created by the FileStorage maybe? Ive zero clue what kafka is 🤷‍♂️

I'm using own-written application which listens for your SignificantTrades websocket then transforms data into JSON and then sends JSON to Kafka (Kafka it's kind of MQ/streaming server). I'll try to make JSON-transformer and Kafka storage

There is actually 3 possible names for exemple

May i ask: in previous versions we used parameter pair: "BTCUSD" without any knowledge about localPairs (XBTUSD, BTCUSDT, etc). Isn't it too complicated to supply all localPairs without knowing what's really inside each exchange product list?

I think (as a simple user) i just want to define which CRYPTO/FIAT pair(s) and which exchange(s) i want to listen.

Maybe it's better to define regexp mapping inside each exchange listener?

You made this to fetch -SPOT/SWAP and -QUARTER/WEEK (time-based) features at the same time and to display them equally? Maybe you can achieve your goal by adding flag (eg fetchTimeBasedFeatures: false) I suppose they are sharing different market volume and bringing different market impact and we should have option to split them between each other

[Tucsky]

May i ask: in previous versions we used parameter pair: "BTCUSD" without any knowledge about localPairs (XBTUSD, BTCUSDT, etc).

Exactly, so in the previous all I was doing was replacing USDT by USD and XBT by BTC. And maybe also trimming the -PERPETUAL for bybit That was ok until I want to track more pairs right :) Products were hardcoded, always USD no way to track only USDT markets, you type BTCUSD you don't know if you are getting a future market, perp or spot and we target remotePair, thu the need normalise the local pair for easy targeting

• -PERPETUAL or -SWAP suffix for all perps markets
• maybe no XBT in the localPair (in favor of "BTC")
• Spot will remain just the symbol name no suffix.
• The time based markets, its going to be complicated, since they need to be refreshed at expiration, the symbol name change etc... for thoses the localPair should equal remotePair if not maybe upper case the local part for consistency.

While letting the possibility to match a symbol as named by the exchange.

I think (as a simple user) i just want to define which CRYPTO/FIAT pair(s) and which exchange(s) i want to listen.

Absolutely, the exchanges option remain unchanged

Theroricaly in order to get the same behavior as today you would need to set the pairs to

pairs: "BTCUSD,XBTUSD,BTCUSDT,BTCUST-PERPETUAL,BTCUSDT-PERPETUAL"
"mapping": {
  "BTCUSD": "(?:XBT|BTC)-?USDT?-?(?:PERPETUAL)",
}

but again BTC has a lot of markets lot of pairs should be must easier to target

[dmitry-ee]

I think with more exchanges more formatting problems will come. Have you considered to use ccxt library? It's a most solid and modern library for crypto exchanges communication at this time.

Check out small product fetcher:

const ccxt = require('ccxt')

const myExchanges = ["binance", "bitmex", "ftx", "okex"]
// const myExchanges = ["binance"]
const keysToExtract = [
  "symbol", "base", "quote", "type",
  // "spot", "swap", "features", "option"
]

async function loadAllMarkets () {
  let result = []
  for (const e of myExchanges) {
    let exchange = eval(`new ccxt.${e}()`)
    await exchange.loadMarkets()
    let markets = Object.values(exchange.markets).map(market => keysToExtract.map(keyName => market[keyName]))
    result.push({ "exchange": e, "markets": markets })
  }
  console.log(JSON.stringify(result))
}

loadAllMarkets()

output and Object.values(exchange.market) example for binance

ccxt should format all pairs equally and tell you is this spot/swap/futures/option pair then you can handle them differently. Yeah library has about 10Mb size, but worth to use.

Hope that could help you