search

palantir / k8s-spark-scheduler

A Kubernetes Scheduler Extender to provide gang scheduling support for Spark on Kubernetes
Apache License 2.0
175 stars 43 forks source link

Update k8s scheduler lib and report packing metrics #232

Closed pisarenko-net closed 1 year ago

pisarenko-net commented 1 year ago

PR implements new metrics for packing efficiency. Packing is now run with 2 functions: the configured one and "pack tightly". The second round of packing is only used for reporting in metrics.

This depends on a new (unreleased) version of k8s-spark-scheduler-lib so this can't be merged until a new lib is released and dependencies are updated.

pisarenko-net commented 1 year ago

Disclaimer: please review the metrics interactions carefully. I am not sure if anything else is required (like an explicit emit, or rate limiting). If there's a good primer on metrics please send me a link.

pisarenko-net commented 1 year ago

Thank you for the review Chris, I'm going to fix merge conflicts and incorporate feedback.