[DISCUSS]: Roadmap 2024

chunshao90 commented 6 months ago

Here are some of the features I have planned for 2024. Everyone is welcome to discuss.

Distributed Solution

Meta Procedure Module
- Refine operations such as retries and rollbacks for different procedures.
Automated Load Balancing Based on Real Load
- Implement dynamic load information collection mechanisms, with horaedb uploading real load data.
- Incorporate advanced scheduling algorithms to balance load distribution based on actual load conditions.
Increase chaos testing to validate system robustness.

Query Optimization

Optimize PQL by supporting operator pushdown to reduce data retrieval costs.
Add SST-level sorted keys to improve data-fetching efficiency during queries.
Pre-Aggregation and Pre-Downsample.

Storage Optimization

Improve query performance for both high and low cardinality by enhancing storage formats
- Introduce inverted indexes.
- Organize data along timelines.
- Implement a columnar storage Memtable.
Compaction Offload
- Offload compaction tasks to dedicated nodes or services to minimize the impact on real-time query nodes.
Import opendal.

Other

horaecontrol cluster command-line tool, supporting functions such as automated deployment, monitoring integration, and fault diagnosis.
Optimize the code directory structure.

Need to discuss

~~- Evaluate the feasibility of reimplementing horaemeta with Rust to unify the development language and improve development efficiency.~~

ShiKaiWi commented 6 months ago

I guess Pre-Aggregation and Pre-Downsample is also an attractive feature to help improve the performance of the queries involving massive data points.

tanruixiang commented 6 months ago

import opendal
file structure simplify

kcrazy commented 2 months ago

Support Windows system.

jiacai2050 commented 2 months ago

Support Windows system.

Hi, would you give us more background about your user case?

We haven't test horaedb on Windows since there is no one asking for this feature.

kcrazy commented 2 months ago

Support Windows system.

Hi, would you give us more background about your user case?

We haven't test horaedb on Windows since there is no one asking for this feature.

Although my production environment is Linux, my development and testing environments are on Windows. Sometimes I also perform tasks on Windows.

If there is a native support version for Windows, it would be more convenient to use, and this is also a factor I consider in whether to use horaedb.

jiacai2050 commented 2 months ago

Thanks, that's sounds reasonable for me.

In theory, IMO horaedb don't use any platform specific features, so it may work on Windows, but I can't ensure that, I think we need someone to try build/run on windows, would you like to give it a try? I create a tracking issue here.

kcrazy commented 2 months ago

Thanks, that's sounds reasonable for me.

In theory, IMO horaedb don't use any platform specific features, so it may work on Windows, but I can't ensure that, I think we need someone to try build/run on windows, would you like to give it a try? I create a tracking issue here.

ok

apache / horaedb