sematic-ai / sematic

An open-source ML pipeline development platform
Other
975 stars 59 forks source link

Support custom annotations for Ray pods #1131

Closed augray closed 1 month ago

augray commented 1 month ago

Some k8s infrastructure may require custom annotations and/or labels to be applied to pods in the cluster (ex: custom autoscalers). Sematic already supports control over this for runner and standalone job pods, but doesn't for pods created via the Ray integration. This PR adds that support. The integration works by adding helm-configurable annotations and labels. The integration works by allowing you to specify annotations or labels to apply to ALL pods created by Sematic's ray integration. You can specify different configurations for pods with/without GPUs but otherwise the configurations must be universal. These configurations must be specified at the helm level, and NOT by pipeline authors.

Testing

In addition to the unit test, manually applied a Ray cluster yaml created with these new configurations to confirm the resulting pods had the expected annotations and labels.