search

JahstreetOrg / spark-on-kubernetes-helm

Spark on Kubernetes infrastructure Helm charts repo
Apache License 2.0
199 stars 76 forks source link

Upgrade to Spark 3.2.0 #75

Closed Rustem closed 2 years ago

Rustem commented 2 years ago

Hi @jahstreet how are you doing so far ? First of all thank you for your initiatives. This is awesome. What is the current status of the project ? We are considering to leverage your work for our analytics platforms. We are thinking to use spark distribution 3.2.0 and hadoop 3.2. What would be your suggestion in such case?

jahstreet commented 2 years ago

Hi Rustem. Thank you for your feedback. I hope to get to the project in the coming month or 2 when would jump on the relevant data platform project as well ;)

What would be your suggestion in such case?

Basically you can use the existing build as a reference and patch it accordingly. There might be some dependency conflicts in Livy package resolving which might be the most complex step.

jahstreet commented 2 years ago

Closing this issue in favour of https://github.com/JahstreetOrg/spark-on-kubernetes-helm/issues/82 within which it will be resolved, lets continue there.

And thanks for the patience 🙏 .