Open goodboy opened 4 years ago
Probably worth reading up on some professional grade systems:
Did a little more research on timescaledb
and found this reddit thread which criticized its performance :crying_cat_face:
On the brighter side, Ameobea
suggested the very cool looking tectonicdb for L2 tick data:
Did some decent starter research on what systems will work for production grade data feed collaboration.
I'm going to list some top picks and what I deem to be pros/cons; comes with a check-mark if we have an existing integration.
numpy
on the suit fronts
[ ] #495
[ ] the new
arcticdb
, a replacement for the priorarctic
project (see below)np.ndarray
s as well or potentially write apolars
adapter?[ ] the original
arctic
by the "man group" which allows serializing dfs and np arrays.marketstore
but is built on mongodbpd.DataFrame
: https://arctic.readthedocs.io/en/latest/tickstore/cryptofeed
backend impl: https://github.com/bmoscon/cryptofeed/blob/6cf185de641cd63b180f2a34cf52c0773e5961a3/cryptofeed/backends/arctic.py[ ]
QuestDb
which seems to claim QuestDB is the fastest open source time series databasejava
...cryptofeed
backend impl: https://github.com/bmoscon/cryptofeed/blob/6cf185de641cd63b180f2a34cf52c0773e5961a3/cryptofeed/backends/quest.py[ ]
greptimedb
: https://github.com/GreptimeTeam/greptimedblegacy-ish FOSS projects that probably aren't specialized enough..
influxdb
is the gold standard in TSDBscryptofeed
backend impl: https://github.com/bmoscon/cryptofeed/blob/6cf185de641cd63b180f2a34cf52c0773e5961a3/cryptofeed/backends/influxdb.pytimescaledb
is an extension on top of postgreskafka
Additional Resources
cryptofeed
's long list of backends which includes some serious projects i hadn't seen before.Thots