numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io/
Apache License 2.0
1.09k stars 111 forks source link

chore: propagate all the vertex template properties to vertex object #1777

Closed whynowy closed 3 months ago

whynowy commented 3 months ago

Currently, when creating vertex pods, we merge the vertex template definition in a pipeline object (spec.templates.vertex.xxx) with the vertex definitions (spec.vertices[.].xxx), to generate the final pod spec. And the vertex template definition will not be propagated to the Vertex object. There's nothing wrong, but it has some potential problems.

  1. If someone updates the vertex template (e.g. add an annotation to all the vertex pods), the new pod spec will not take effect unless the vertex pod is recreated (because there's no Vertex object change);
  2. Vertex object does not have finalized spec, and we could not use its spec to make runtime decision - e.g. to tell if there's an annotation on a vertex pod. @yhl25

This PR updated the logic to propagate the vertex template definition to the vertex object, and stop merging the template definition when generating pod spec from a vertex object. It makes no change to the final vertex pod spec.