V2.3.x release plan - Githubissues

rosedblabs / rosedb

Lightweight, fast and reliable key/value storage engine based on Bitcask.

https://rosedblabs.github.io

Apache License 2.0

4.62k stars 634 forks source link

V2.3.x release plan #253

Closed roseduan closed 1 year ago

roseduan commented 1 year ago

[x] optimize merge operation, provide an option to rebuild the index after the merge is done @Jeremy-Run
discussion: #251
[x] refactor index module, use in-memory btree as backend.
[ ] ~~disk hash index~~
I try to use diskhash as the index, but I found that the change will be a lot, so I will give up this issue.
If some user acquires this feature, I recommend LotusDB instead.
[ ] maybe more...

taroim commented 1 year ago

Are you planning to add support for TTL function?

taroim commented 1 year ago

Also, could you consider adding support for self-repair function of WAL files?

weijiew commented 1 year ago

If you want to add TTL (Time-to-Live) support, I can implement this. @roseduan

roseduan commented 1 year ago

Are you planning to add support for TTL function?

Are you going to use rosedb? can you describe your case more clearly? So I can design this feature in the next release.

roseduan commented 1 year ago

Also, could you consider adding support for self-repair function of WAL files?

What do you mean self-repair? You can add an issue in WAL project and tell me more about it, thanks.

taroim commented 1 year ago

Are you planning to add support for TTL function?

Are you going to use rosedb? can you describe your case more clearly? So I can design this feature in the next release.

We are currently attempting to utilize RoseDB for storing meteorological data, primarily focused on storing forecast data for the last 10 days. These data necessitate daily scheduled tasks such as collection, parsing, and collaborative element coordination. Additionally, the regular removal of outdated data is particularly crucial due to the substantial data volume, which amounts to no less than 30GB per day.

roseduan commented 1 year ago

Are you planning to add support for TTL function?

Are you going to use rosedb? can you describe your case more clearly? So I can design this feature in the next release.

We are currently attempting to utilize RoseDB for storing meteorological data, primarily focused on storing forecast data for the last 10 days. These data necessitate daily scheduled tasks such as collection, parsing, and collaborative element coordination. Additionally, the regular removal of outdated data is particularly crucial due to the substantial data volume, which amounts to no less than 30GB per day.

If the data amounts are no less than 30GB per day, I think you should pay attention to the restart speed of RoseDB, because rosedb will iterate all data to rebuild the index in memory, it may take a long time if the data grows larger.

BTW, the TTL and expire methods are easy to support, I will add them to our development plan if you need them.

taroim commented 1 year ago

The restart speed is indeed relatively slow, We've developed middleware to achieve distributed management of RoseDB. At present, use 16 testing nodes, storing data around 120GB. Therefore, I'm particularly attentive to the efficiency and stability of the data merging functionality.

We have experimented with various storage engines, including BoltDB, Badger, VictoriaMetrics, and more. After comparing them, we found that RoseDB is relatively straightforward, with high read-write efficiency ~~~ it's quite astonishing!

roseduan commented 1 year ago

OK thanks, I will add the TTL support in the next release.

roseduan commented 1 year ago

V2.3.0 has released: https://github.com/rosedblabs/rosedb/releases/tag/v2.3.0