search

GreptimeTeam / greptimedb

An open-source, cloud-native, unified time series database for metrics, logs and events with SQL/PromQL supported. Available on GreptimeCloud.
https://greptime.com/
Apache License 2.0
4.36k stars 315 forks source link

GreptimeDB Roadmap 2024 #3412

Open tisonkun opened 9 months ago

tisonkun commented 9 months ago

You may find the roadmap blog at https://greptime.com/blogs/2024-02-29-greptimedb-2024-roadmap.

Overview

This roadmap outlines the objectives our team is pursuing and the visions we harbor for our collective community.

The evolution of GreptimeDB in 2024 is marked by a suite of main feature updates. These enhancements are a testament to our ongoing commitment to excellence, driven by feedback from our community and the latest requirements in real-world scenarios.

Our roadmap for the year includes significant advancements that promise to elevate the capabilities of GreptimeDB and enrich the user experience.

Join Force

For tasks with tracking issues available, directly diving into the issue and commenting would be desirable.

For open-ended tasks or if you have any other ideas to improve GreptimeDB, don't hesitate to comment below, ask on GitHub Discussion or join the GreptimeDB Community on Slack.

Details

Metrics Engine

A new engine designed for observability scenarios. Its primary aim is to handle a large number of small tables, making it particularly suitable for using Prometheus metrics. By utilizing synthetic wide tables, this new Engine offers the capability to store metric data and reuse metadata, rendering "tables" atop it more lightweight and overcoming some of the limitations of the existing Mito engine, which is considered too heavy for such tasks.

Inverted Index

DataFlow (GreptimeFlow)

A lightweight stream computing component capable of performing continuous aggregation on GreptimeDB data streams. It can be embedded into the GreptimeDB Frontend or deployed as a separate service within the GreptimeDB cluster.

Here is a flow job can be submitted in the form of SQL:

CREATE TASK avg_over_5m WINDOW_SIZE = "5m" AS SELECT avg(value) FROM table WHERE time > now() - 5m GROUP BY time(1m);

Full-text Index

In SQL statements, you can use the MATCHES function to perform full-text searches, which is especially useful for log analysis. The MATCHES function supports full-text searches on String type columns. Here’s an example of how it can be used:

SELECT * FROM logs WHERE MATCHES(message, 'Barack OR Obama');

Region Migration

It allows migrating regions between Datanodes, facilitating the relocation of hot data and the horizontal scaling of load balancing.

Logs Engine

A storage engine designed specifically for the characteristics of log data, sharing most of GreptimeDB's architecture and capabilities, such as the SQL query layer, data sharding, distributed routing, querying indexing, and compression ability. This enables GreptimeDB to become a unified system offering optimized storage and a consistent access experience for Metrics and Logs data based on a multi-engine architecture.

tisonkun commented 9 months ago

@fengjiachun I remember that "Region Migration" is under development, do we have some issues to linked here?

fengjiachun commented 9 months ago

@fengjiachun I remember that "Region Migration" is under development, do we have some issues to linked here?

I added RFC and tracking issue link.

tisonkun commented 9 months ago

@fengjiachun Cool! So it even has been implemented and would be delivered at 0.7?

fengjiachun commented 9 months ago

@tisonkun Yes

tisonkun commented 4 months ago

No longer valid. Need an update.

killme2008 commented 4 months ago

Yes. We need to update it but keep it open.

@fengjiachun Please update it next week

fengjiachun commented 3 months ago

I'll update it in the next few days.