databento / databento-python

The official Python client library for Databento
https://docs.databento.com
Apache License 2.0
137 stars 22 forks source link

Is there a self-hosted tsdb (`dbz`?) and any existing `docker` container? #9

Closed goodboy closed 1 year ago

goodboy commented 1 year ago

Hey y'all,

very very interested in trying out your tsdb but am wondering if there's any containerization work that's been done and is at least partially maintained?

If not happy to whip something up and publish it for others to try.

Salud!

cjdsellers commented 1 year ago

Hi @goodboy

Thanks for your interest!

To clarify, this library doesn't actually include a time series database. It provides a thin client to our HTTP API for historical data streaming and batch download requests.

Could you tell us more about your specific use case?

goodboy commented 1 year ago

@cjdsellers ah yes my apologies, i hadn't read through your service docs enough 😹

I was under the impression the project was more along the lines of something like marketstore where you were providing the tsdb and client; i see now that the backend is all in your prop service yes?

the project i am working around is in search of something more reliable (cough with maybe less go and more rust) for our tsdb. i had seen this project after snooping your nautilus project and thought based on the client docs it looked like a great fit 😉

To clarify, this library doesn't actually include a time series database. It provides a thin client to our HTTP API for historical data streaming and batch download requests.

I presume dbz is used in your service and/or can be used without it as a form of tsdb though right? I haven't looked into that code base yet, but I guess we're after something that is foss, can be self-hosted, and more or less has parity perf and functionality with marketstore.

Sorry this derailed so much from the original question, i'll adjust the OP title.

goodboy commented 1 year ago

Ah, diggin a little i'm going to guess the http service is wrapping the dbz-python stuff yah?

rkrn commented 1 year ago

Yes.

Keep in mind dbz is not a database management system (like Clickhouse) nor an embedded database (like RocksDB). It's a presentation layer protocol (like SBE) that can be used both as a file storage format or for messaging.

You could use it as a flat file, backing storage for a server — which is basically what we've done for our API backend — and get good performance out of it.

If you have an interest in dbz, keep in mind we'll be releasing major changes soon.

goodboy commented 1 year ago

@rkrn sweet sounds good.

Yeah, longer run if it becomes stable enough to be used in a similar way (with wrapping) as your http layer, we'll probably take it seriously.

I'm sure y'all know about the competition but just for the record these are the tsdbs we've got eyes on atm.

I look forward to seeing the major updates have in store 🏄🏼