[Feature][Server] Discard ZooKeeper using raft

DylanMao1996 commented 2 years ago

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

1.Dolphinscheduler is too heavy to have a ZooKeeper external. For lighter use and easier deployment, is it possible to do something like Kafka? Ditching ZooKeeper and using raft protocol for lighter weight?

2.The DolphinScheduler metadata store is now PG and mysql. Both require external installation. Can you integrate with Rocksdb? It has a rocksdb embedded in it. Data is stored in the HDFS to ensure metadata security (HDFS multi-version replication).

Use case

No response

Related issues

No response

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

github-actions[bot] commented 2 years ago

Hi:

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.

caishunfeng commented 2 years ago

Good idea. Can you provide some specific design for using raft replace zk?

DylanMao1996 commented 2 years ago

Dolphinscheduler currently has distributed locks removed, allowing multiple masters to process them in parallel, but intermediate operations still involve locks. Is there a way to leverage raft's idea? The State event event is listened for as long as the command is generated and then thrown to the master thread pool for distribution.

------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2021年11月4日(星期四) 下午4:29 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [apache/dolphinscheduler] [Feature][Server] Discard ZooKeeper using raft (Issue #6680)

Good idea. Can you provide some specific design for using raft replace zk?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

caishunfeng commented 2 years ago

Dolphinscheduler currently has distributed locks removed, allowing multiple masters to process them in parallel, but intermediate operations still involve locks. Is there a way to leverage raft's idea? The State event event is listened for as long as the command is generated and then thrown to the master thread pool for distribution.

I think this is just a part of schedule optimization, but the main function of zk is registry center.

DylanMao1996 commented 2 years ago

If it is just the same as the registry, I can understand that zK's Watch is used. Can it be like Kafka? What about Controller Quorum? You can cancel ZK and add a DolphinScheduler JPS process.

------------------ 原始邮件 ------------------ 发件人: "apache/dolphinscheduler" @.>; 发送时间: 2021年11月4日(星期四) 下午4:40 @.>; @.**@.>; 主题: Re: [apache/dolphinscheduler] [Feature][Server] Discard ZooKeeper using raft (Issue #6680)

Dolphinscheduler currently has distributed locks removed, allowing multiple masters to process them in parallel, but intermediate operations still involve locks. Is there a way to leverage raft's idea? The State event event is listened for as long as the command is generated and then thrown to the master thread pool for distribution.

I think this is just a part of schedule optimization, but the main function of zk is registry center.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

apache / dolphinscheduler