[X] I had searched in the issues and found no similar feature requirement.
Description
As BanyanDB grows and deals with bigger datasets, it's important to manage data efficiently and optimize storage to keep costs down. Right now, BanyanDB doesn't differentiate between "hot" data that gets accessed frequently and "cold" data that's barely touched.
After discussing with @HHoflittlefish777, we propose the implementation of a hot-warm-cold data management architecture within BanyanDB. This architecture would categorize data based on access patterns and move them between different storage tiers:
Hot: Data that is frequently accessed and requires fast read/write speeds. This should be stored on the fastest storage media available.
Warm: Data that is accessed less frequently but may still need to be readily available. This could be stored on lower-cost, lower-speed media. It applies a force merge to optimize the data files.
Cold: Rarely accessed data that can be archived, preventing any changes from being made to it. This should be stored on the cheapest storage media, such as cold HDDs or even offsite/cloud storage.
Contributor: @HHoflittlefish777
Next Steps
[ ] Discuss the viability and scope of the proposed feature
[ ] Draft a detailed design document
[ ] Review and approve the design document
[ ] Implement the feature
[ ] Perform thorough testing, including performance and resilience tests
[ ] Update documentation to reflect the new architecture
[ ] Create tutorials or guides for users to understand how to work with the new data tiers
Use case
No response
Related issues
No response
Are you willing to submit a pull request to implement this on your own?
[ ] Yes I am willing to submit a pull request on my own!
sollhui
commented
8 months ago
Search before asking
Description
As BanyanDB grows and deals with bigger datasets, it's important to manage data efficiently and optimize storage to keep costs down. Right now, BanyanDB doesn't differentiate between "hot" data that gets accessed frequently and "cold" data that's barely touched.
After discussing with @HHoflittlefish777, we propose the implementation of a hot-warm-cold data management architecture within BanyanDB. This architecture would categorize data based on access patterns and move them between different storage tiers:
Contributor: @HHoflittlefish777
Next Steps
Use case
No response
Related issues
No response
Are you willing to submit a pull request to implement this on your own?
Code of Conduct