search

clemlabprojects / ambari

Fork of Apache Ambari maintained by Clemlab Company
https://www.clemlab.com
Apache License 2.0
42 stars 17 forks source link

【feat】Could you integrate spark 3.4.1 ? #26

Closed zyclove closed 8 months ago

zyclove commented 1 year ago

Spark 3.3.x and 3.4.x have many new features and function enhancements, making development more efficient. At present, it has been optimized and improved many times, and has been used in the production environment for a long time.

It is very hopeful to be able to upgrade integration or multi-version management.

Many thanks for contributing to the community.

Best regards

lucasbak commented 1 year ago

Hi @zyclove,

It was not planned for ODP 1.0 nor 1.1, as it pulls lot of dependencies with new version. However as it is a requested feature, we are going to discuss about it internally and keep you update as soon as it is released. (ODP 1.2 with Hadoop 3.3 is going to be release next October). Best regards,

Clemlab Team

zyclove commented 1 year ago

Upgrading Hadoop 3.3 is a very good plan and I am looking forward to it. I also really hope to upgrade spark 3.4. May I ask whether ODP 1.1 has been verified on the arm architecture cluster?

Best regards

lucasbak commented 1 year ago

ODP 1.1 has not been verified on arm as core components such as zookeeper, hadoop, hbase and ranger need native librairies. However we started to study it. We focused on ubuntu operating system compatibility for ODP 1.2 release.

Best regards,

Clemlab Team

lucasbak commented 1 year ago

Hi @zyclove,

Its integrated in ODP 1.2. We have also built a version of ODP 1.1 with spark 3.4.1 for answer yo your need links to ODP 1.1 with spark 3.4.1

Best regards,

Clemlab Team

lucasbak commented 11 months ago

Hi @zyclove ,

Did you try Spark 3.4.1 ? so we can mark this issue as closed.

Best regards,

Clemlab Team