search

tardis-dev / tardis-node

Convenient access to tick-level real-time and historical cryptocurrency market data via Node.js
https://docs.tardis.dev/api/node-js
Mozilla Public License 2.0
293 stars 65 forks source link

implement fineGrainCache mechanism #26

Closed hackape closed 4 years ago

hackape commented 4 years ago

implement #25, submitted as draft.

thaaddeus commented 4 years ago

HI, this looks good, but I must give it some thought how it affects everything as for example first thing I notice is that it will cause API to receive much more requests for cases that are different to yours (requesting multiple symbols across many channels) and that right now is main blocker for me. I know it's behind the flag, but still. BTW, have you observed if perf is still good with merging logic from multiple files vs one file?

hackape commented 4 years ago

No I haven't done any benchmark yet.

But I've already observed test fail. data feed API for bitfinex cannot distinguish the symbol filter since raw data are identified by chanId. And when merging multiple symbol stream back to one, like in the test BTCUSD and BTCUST, the merged stream end up emitting duplicate message, cus the same raw chanId message presents in both BTCUSD and BTCUST streams.

hackape commented 4 years ago

For the bitfinex case it's really beyond my control if server side cannot support.

But for this one:

it will cause API to receive much more requests for cases that are different to yours (requesting multiple symbols across many channels)

I'm not sure how bad the impact is on server side. But does reusing the same TCP connect solve the problem? That can be done on client side.

have you observed if perf is still good with merging logic from multiple files vs one file?

I'll post benchmark later.

thaaddeus commented 4 years ago

Yes, bitfinex is tricky one, reusing TCP does not help much (client already does that, keep-alive), will think about it.