search

njaard / sonnerie

A simple timeseries database
Other
264 stars 19 forks source link

Mentioning SQLite3 as the chosen backend #2

Closed lweberk closed 5 years ago

lweberk commented 5 years ago

Hi,

really interesting project.

Perhaps it would be relevant to mention it uses SQLite3 as backend. I did not notice it until the backup section, which made me look into the dependencies in the Cargo.toml. For those looking for something embeddable it might be of relevance, like it was for me.

Best regards, lwk

njaard commented 5 years ago

It's true, metadata is stored in sqlite3. In what manner would this affect your ability to use it? What are the constraints specifically?

lweberk commented 5 years ago

The first one is more like an inconvenience than a deal breaker. We would have to compile on our ARMv6/7 targets which is prohibitively slow and limited or manually bending the linking process against a foreign arch libsqlite3 while cross-compiling which I currently would not know how to. Alternatively emulate ARM on x86_64 in a VM. All of them are not very appealing, though fiddling with the linking might be actually not so far fetched. All in all, our use-case is so simple, that we have ditched sqlite3 for just sequential files in a past case already, to avoid this issue altogether.

Second downside is the amount of IO for the write of an entry is not transparent and would require digging into it in a case by case basis, as well as as the sqlite3 file gets bigger. Relevant due to wear leveling on the SD cards. Currently we are purchasing industrial grade microSD's to have built-in wear leveling at the hardware level so we do not burn a hole into the storage right away. But still, it would be nice to have a clearer picture on that given sqlite3 being conceived for a lot more than monotonic time series data, relinquishing the potential for optimizing for the use-case.

Don't get me wrong, I like this project a lot. From a preliminary perspective it is just not the right fit for our use case. This issue was more like a drive by report of me tangentially touching the project while going through potential time series storage solutions.

njaard commented 5 years ago

Even without the sqlite, I would say Sonnerie's disk access patterns are not really suitable for SSD. For the kind of high-throughput applications Sonnerie is designed for, designing for SSDs and HDDs is actually pretty different. I'm not sure how I would design a tsdb for use with SSDs.

Note that the actual time series data isn't stored in sqlite but in the "blocks" file, which is not sqlite.

njaard commented 5 years ago

I tried to go into a little more detail about when sonnerie should be used, making note of Sqlite.