Global cache for Exchange.loadMarkets()

[ggecy]

Hello,

in our project we are using many exchange instances - one user in our system can have multiple access keys to multiple exchanges and there are many users. Each ccxt exchange object however needs to load the markets separately and holds a copy of them, which is a lot of duplicate data clogging the memory, a lot of traffic and it takes a long time to load the markets.

Would it be possible to have a global cache for markets per exchange type? Maybe enabled by some parameter during exchange creation? So e.g. if I have 20 instances of binance exchange, they would all share the same markets and currencies data?

Thanks. Gabriel

[kroitor]

@ggecy hi! What's your programming language?

[ggecy]

@kroitor Hi, most of the code is in Java, but the part using ccxt library runs on nodejs and javascript.

[kroitor]

@ggecy thx, just asking, since it depends on the language... We don't have the global cache in our immediate priorities, but you can still do that in Node.js / JavaScript. I'll post an example on how to share that data across multiple exchange instances asap.

[kroitor]

@ggecy please, see this example:

• https://github.com/ccxt/ccxt/blob/master/examples/js/shared-load-markets.js

const ccxt = require ('ccxt')

const globalIds = [ 'binance', 'poloniex', 'bittrex', 'bitstamp' ]
const globalExchanges = {}

async function loadExchange (id) {
    try {
        const exchange = new ccxt[id] ({
            'enableRateLimit': true,
        })
        await exchange.loadMarkets ()
        globalExchanges[id] = exchange
    } catch (e) {
        // throw e // uncomment to break the entire program on any error
        // console.log (e) // print the exception and ignore this exchange
    }
}

async function main () {

    // initialize unique global exchange instances first
    await Promise.all (globalIds.map (async (id) => loadExchange (id)))
    console.log ('Loaded global exchanges:', Object.keys (globalExchanges))

    // load user exchanges and keys from a database or configure these via JSON
    const users = {
        'user1': {
            'binance': { 'apiKey': 'USER1_BINANCE_API_KEY', 'secret': 'USER1_BINANCE_SECRET' },
            'poloniex': { 'apiKey': 'USER1_POLONIEX_API_KEY', 'secret': 'USER1_POLONIEX_SECRET' }
        },
        'user2': {
            'poloniex': { 'apiKey': 'USER2_POLONIEX_API_KEY', 'secret': 'USER2_POLONIEX_SECRET' },
            'bittrex': { 'apiKey': 'USER2_BITTREX_API_KEY', 'secret': 'USER2_BITTREX_SECRET' }
        },
        'user3': {
            'bittrex': { 'apiKey': 'USER3_BITTREX_API_KEY', 'secret': 'USER3_BITTREX_SECRET' },
            'bitstamp': { 'apiKey': 'USER3_BITSTAMP_API_KEY', 'secret': 'USER3_BITSTAMP_SECRET' }
        }
    }

    // initialize local exchanges per user
    const localExchanges = {}
    for (const userId in users) {
        const userExchanges = {}
        for (const exchangeId in users[userId]) {
            if (exchangeId in globalExchanges) {
                const globalExchange = globalExchanges[exchangeId]
                const exchange = new ccxt[exchangeId] ({
                    'enableRateLimit': true,
                    // 'verbose': true, // uncomment for debug output
                    ... users[userId][exchangeId],
                });
                [
                    'ids',
                    'markets',
                    'markets_by_id',
                    'marketsById',
                    'currencies',
                    'currencies_by_id',
                    'baseCurrencies',
                    'quoteCurrencies',
                    'symbols',
                ].forEach ((propertyName) => {
                    exchange[propertyName] = globalExchange[propertyName]
                })
                userExchanges[exchangeId] = exchange
            }
        }
        localExchanges[userId] = userExchanges
    }

    // print the loaded exchanges per user
    for (const userId in localExchanges) {
        console.log ('Loaded', userId, Object.keys (localExchanges[userId]))
    }
}

main ()

Let us know if that does not answer your question.

[ggecy]

Thanks, I will try it and let you know.

[ggecy]

I took just the copy properties part since our logic is a bit more complex, it's working fine, thank you.

[kroitor]

@ggecy thx for the feedback!