search

opencurve / curve

Curve is a sandbox project hosted by the CNCF Foundation. It's cloud-native, high-performance, and easy to operate. Curve is an open-source distributed storage system for block and shared file storage.
https://opencurve.io
Apache License 2.0
2.34k stars 526 forks source link

Curve Roadmap 2023 #2207

Open ilixiaocui opened 1 year ago

ilixiaocui commented 1 year ago

Update at 2023-07-24

CurveFS Feature

progress expected describe related
Doing Q3 Rename performance optimization pending (The design scheme has been confirmed, but there is a lack of developers to complete the function development. If you want to participate in the development of this function, please contact us)
Done Q2 Metadata performance optimization when multiple mounts enable cto consistency(In the cto consistency scenario, the client's metadata cache is invalid, resulting in slow metadata operations) Done in v2.6
progress expected describe related
Doing Q2 Support CurveBS as data storage backend(In terms of function, the data deletion function needs to be completed. performance and stability need to be tested) pending (The development and basic testing of the single-client mounting mode has been completed in master branch, and the multi-client mounting mode still has data synchronization problems, If you want to participate in the development of this function, please contact us)
Doing Q3 Support HDFS SDK(hadoop users can use CurveFS as HDFS storage)
Delay NULL Support hot and cold data hierarchical storage and life cycle management Depending on the feature above
Doing Q3 Support online migration of a cluster across two IDCs(eg. MDS/ETCD/metaservers)
Q4 Support single replica storage backend (We can use various distributed kv cache systems to replace the S3 storage backend and implement a single copy mode to support temporary data storage services)
progress expected describe related
Done Q2 Support AI scene(It is necessary to study the IO model of various stages of AI and perform targeted optimization) We have completed the docking and tuning of AI business scenarios in NetEase, and the performance and cost have achieved significant results. In the second half of the year, there will be specific output of relevant cases and optimization solutions, so stay tuned
Delay NULL Support hot data storage of ES\Clickhouse(Need to focus on performance optimization) Depending on the feature Support CurveBS as data storage backend

CurveBS Feature

progress expected describe related
Done Q2 Integrate various tools of CurveBS into new tool https://github.com/opencurve/curve/tree/master/tools-v2
Done Q2 Compatible with librbd library interface, no need to compile QEMU and Libvirt when using Curve volume Basically completed, the beta version is expected to be released in early August
Done Q2 Hybrid storage engine supporting SSD/NVMe and HDD It has been used in NetEase, the beta version is expected to be released in early August
Done Q2 Supports single-cluster multi disk type storage pools, such as all-flash pools and HDD/SSD hybrid storage pools It has been used in NetEase, the beta version is expected to be released in early August
Delay NULL Support EC pending (The design scheme has been confirmed, but there is a lack of developers to complete the function development. If you want to participate in the development of this function, please contact us)
Delay NULL Support single machine scale in cluster pending (If you want to participate in the development of this function, please contact us)
Doing Q3 Support volume snapshot store in the CurveBS cluster Basically completed, the beta version is expected to be released in August
Doing Q3 Support authentication in Curve system Basically completed, the beta version is expected to be released in August
Doing Q3 Support online migration of a cluster across two IDCs(eg. MDS/ETCD/Chunkservers)
progress expected describe related
Doing Q3 Brpc/Braft optimization(Reduce software layer latency. Improve the stability of the cluster running under high load, for example, the heartbeat timeout caused by high load further leads to frequent switching of the replication group leader)
Doing Q3 Performance analysis and optimization in mixed flash scenarios(NVME+HDD)
Delay NULL Performance analysis tool
progress expected describe related
Done Q2 curveadm support the client boots up and automatically mounts Done in v0.3

Cloud Native

progress expected describe related
Done Q2 Support CurveBS operator(Including cluster deployment, daily operation and maintenance, and visualization of monitoring indicators) https://github.com/opencurve/curve-operator
Done Q2 Support CurveFS operator(Including cluster deployment, daily operation and maintenance, and visualization of monitoring indicators) https://github.com/opencurve/curve-operator

CI

progress expected describe related
Done Q2 CI Stability Optimization of CurveBS We are using the new stability toolset now in CI
Q4 CI visualization optimization
DemoLiang commented 1 year ago

我觉得可以考虑优先增加下块存储的开机自启动和自动挂载，当时的bs客户端重启后必须手动启动docker还有手动挂载块设备

shentupenghui commented 1 year ago

我觉得优先云原生化部署和可观测方面特性，减少客户使用运维成本

shawn0915 commented 1 year ago

Support hot and cold data hierarchical storage and life cycle management

期待此特性Release，尤其是热数据IO性能调优提升。

ilixiaocui commented 1 year ago

我觉得可以考虑优先增加下块存储的开机自启动和自动挂载，当时的bs客户端重启后必须手动启动docker还有手动挂载块设备

good idea!

ilixiaocui commented 1 year ago

我觉得优先云原生化部署和可观测方面特性，减少客户使用运维成本

  1. What exactly does it mean to optimize cloud-native deployment? Currently only csi is supported, the focus of this year in roadmap is the support for operator.

  2. What exactly does optimization of observable properties mean? Currently using the form of promethues+grafana. Does it mean that the display form is not enough? Or is the indicator description not enough? or something else？

ilixiaocui commented 1 year ago

Support hot and cold data hierarchical storage and life cycle management

期待此特性Release，尤其是热数据IO性能调优提升。

Welcome to continue to pay attention, it would be even better if we can participate together :)

Wine93 commented 1 year ago

Hi, guys, we will discuss CTO optimization draft at 2023-02-21 15:00-16:00. You can click the tencent meeting link below to join us: https://meeting.tencent.com/dm/hTU86nag6WG0

design documents

caoxianfei1 commented 1 year ago

Hello, everyone, we will discuss CurveBS deployment solution on Kubernetes at 2023-02-24 16:00-17:00 You can click the tencent meeting link below to join us: https://meeting.tencent.com/dm/TlpXDnFYcpqI

Design document for reference： Curve-BS云原生部署方案.pdf

Discussion Group image

legionxiong commented 1 year ago

我觉得可以考虑优先增加下块存储的开机自启动和自动挂载，当时的bs客户端重启后必须手动启动docker还有手动挂载块设备

已有方案并通过验证， 近期会提交到社区 review。

legionxiong commented 1 year ago

我觉得可以考虑优先增加下块存储的开机自启动和自动挂载，当时的bs客户端重启后必须手动启动docker还有手动挂载块设备

已有方案并通过验证， 近期会提交到社区 review。

Curve: https://github.com/opencurve/curve/pull/2285 CurveAdm: https://github.com/opencurve/curveadm/pull/201

Cyber-SiKu commented 1 year ago

我觉得可以考虑优先增加下块存储的开机自启动和自动挂载，当时的bs客户端重启后必须手动启动docker还有手动挂载块设备

已有方案并通过验证， 近期会提交到社区 review。

Curve: #2285 CurveAdm: opencurve/curveadm#201

您是做的chunkserver自动重启挂载吗？

lizonglingo commented 1 year ago

今天在试用过程中有一些问题想了解下：

  1. client 关机/重启后，需要 restart container 再进入手动挂载，退出容器后还需要手动再挂载到系统上。能不能改进 client 重启自动挂载的支持？
  2. 关于 UI 交互界面，可能对于运维人员来说，对 pool、user、usage 等集群、存储池、用户、容量的信息最好能清晰明了方便的查看，所以后续有没有 Web 交互集群的支持（如 Longhorn UI 那种，平替 curveadm、curve 命令）？
ilixiaocui commented 1 year ago

今天在试用过程中有一些问题想了解下：

  1. client 关机/重启后，需要 restart container 再进入手动挂载，退出容器后还需要手动再挂载到系统上。能不能改进 client 重启自动挂载的支持？
  2. 关于 UI 交互界面，可能对于运维人员来说，对 pool、user、usage 等集群、存储池、用户、容量的信息最好能清晰明了方便的查看，所以后续有没有 Web 交互集群的支持（如 Longhorn UI 那种，平替 curveadm、curve 命令）？

1： The curveadm develop version has been supported, but the current client crash and manual stop need to manually unmount the mount point 2：There is already a preliminary version of the console: https://github.com/opencurve/curve-meetup-slides/pull/82

SeanHai commented 1 year ago

Hi, guys, we will discuss authentication in Curve system at 2023-05-23 16:00-17:00. You can click the tencent meeting link below to join us: https://meeting.tencent.com/dm/4SH79fdWD4Y6