search

mesosphere / spark-build

Used to build the mesosphere/spark docker image and the DC/OS Spark package
52 stars 34 forks source link

Removed 3rd-party StatsD sink and switched to Spark's built-in one #476

Closed akirillov closed 5 years ago

akirillov commented 5 years ago

What changes were proposed in this pull request?

Resolves DCOS-47406

Previously, DCOS Spark package relied on a 3rd-party StatsD Sink] for metrics reporting. Spark 2.3 ships with StatsD Sink included. This PR removes 3rd-party dependency and adds integration tests for metrics for Spark application running in both HOST mode and on top of overlay network.

How were these changes tested?

New integration test specifically for this change

Release Notes

akirillov commented 5 years ago

Thanks, @porridge. Kafka test flakiness is a known issue and will be addressed separately. As for one of CNI tests failing - logs show that it was unable to start any of executors. This issue should be addressed separately too. As for current PR - the changes didn't cause any issues, and newly added tests passed in all the builds. I'm about to merge this PR and triage CNI tests issues if/when they reappear.