search

datavane / datasophon

The next generation of cloud-native big data management expert , Aims to help users rapidly build stable, efficient, and scalable cloud-native platforms for big data.
https://datasophon.github.io/datasophon-website/
Apache License 2.0
1.01k stars 363 forks source link

[Feature] Spark3问题和Spark2集成建议 #500

Open 954031894 opened 5 months ago

954031894 commented 5 months ago

Search before asking

Description

在使用1.2.1版本的过程中通过datasophon部署安装spark3中没有master角色，看了一下安装Spark3指南，说没有master角色，感觉应该是Spark3没有适配完成吗？ 希望集成一个Spark2版本，因为安装的很多组件对spark的要求还是2.x

当然，我自己也会尝试一下我的这些建议

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

thomasg19930417 commented 5 months ago

The onYarn mode should be used here. In theory, this is just a client, but the role of historyserver should be missing currently.

954031894 commented 5 months ago

If the on Yarn mode is indeed like this, but when I deploy Spark3, even the workers cannot deploy it. There is no such option, only the client. I don't know if it's a version issue on my end or what caused it

iShot_2024-01-25_09 25 45
954031894 commented 5 months ago

I understand the reason now. In On Yarn mode, there is no need for a master or work, only the client needs to submit tasks. The past deployment methods have formed this mindset in me. Sorry to bother you with this issue.