search

banzaicloud / spark-metrics

Spark metrics related custom classes and sinks (e.g. Prometheus)
Apache License 2.0
176 stars 66 forks source link

Updating spark dependencies to 3.2 versions and updating prometheus sink constuctor #81

Closed michaelcbennett closed 2 years ago

michaelcbennett commented 2 years ago
Q A
Bug fix? yes
New feature? no
API breaks? no
Deprecations? no
Related tickets n/a
License Apache 2.0

What's in this PR?

Updated the library to support spark 3.2. Also updated the other dependencies to more recent versions.

Why?

The spark 3.2 metrics system changed the constructor for sinks to no longer look for a constructor with a security manager, so I added new constructor without the security manager (which wasn't used previously anyway). I left the older constructor in place to maintain backwards compatibility, but it just calls the new constructor.

Checklist

CLAassistant commented 2 years ago

CLA assistant check
All committers have signed the CLA.

kun-hackle commented 2 years ago

Hi, any update? We are using EMR 6.6.0+ w/ Spark 3.2.X.

michaelcbennett commented 2 years ago

We are currently using it with this PR on EMR 6.6 / Spark 3.2. I am working through the Contributor License Agreement with my employer, but hope to have to clearance soon.

kun-hackle commented 2 years ago

Thanks. Will try to build this branch for our production too.