search

numaproj / numaplane

Control Plane for Numaproj
Apache License 2.0
4 stars 3 forks source link

Kustomize Transformer Config #59

Closed afugazzotto closed 1 week ago

afugazzotto commented 1 week ago

Modifications

Created the Kustomize Transformer config file starting from the Numaflow equivalent file. Also, added some sample files for testing and documentation.

NOTE: if changes are made to the Numaflow Kustomize Transformer config file, we need to manually update the equivalent file on this repo. If we want to automate the task, I created this issue to do in the future: https://github.com/numaproj/numaplane/issues/58

Verification

Manually tested by using the kustomize binary and kubectl apply -k command on the config/kustomize/examples/transformer directory

juliev0 commented 1 week ago

Interesting, I didn't know about kustomize transformer configs.

Is this something that has been discussed with others regarding our process on the Numaportal side? Since we are supporting multiple versions of Numaflow at the same time, does it make sense to have this? (as opposed to maybe some other process where Numaportal first creates a Pipeline spec of a given version and then encapsulates it into a Rollout CRD)

I would imagine in Numaflow that there's some codegen automation that creates this file. I don't think I want to add something like this if we don't have all of the automation that would be needed to accompany it.

cc'ing @whynowy

afugazzotto commented 1 week ago

@juliev0 on the Numaflow side I did not see any automation in place to update this file. Last few changes to it seem to have been performed manually (https://github.com/numaproj/numaflow/commits/d72bd714b9f89236650c71152317f5acbb234785/docs/user-guide/reference/kustomize/numaflow-transformer-config.yaml). I created an issue https://github.com/numaproj/numaplane/issues/58 to add automation to generate this file on Numaplane side if necessary to have a more complete solution, but I'm not sure it is really needed at this time.

juliev0 commented 1 week ago

Last few changes to it seem to have been performed manually (https://github.com/numaproj/numaflow/commits/d72bd714b9f89236650c71152317f5acbb234785/docs/user-guide/reference/kustomize/numaflow-transformer-config.yaml).

Thank you for finding this. @whynowy What is the process currently for updating that file?

So, if a given user is using Numaflow version 1.0 and our file is updated to include all of the fields for Numaflow version 2.0 (perhaps we can assume Numaflow will only ever add new fields and not remove them? but I assume array indices could change (but maybe there aren't any in there?)) will there be any issues?

afugazzotto commented 1 week ago

Last few changes to it seem to have been performed manually (https://github.com/numaproj/numaflow/commits/d72bd714b9f89236650c71152317f5acbb234785/docs/user-guide/reference/kustomize/numaflow-transformer-config.yaml).

Thank you for finding this. @whynowy What is the process currently for updating that file?

So, if a given user is using Numaflow version 1.0 and our file is updated to include all of the fields for Numaflow version 2.0 (perhaps we can assume Numaflow will only ever add new fields and not remove them? but I assume array indices could change (but maybe there aren't any in there?)) will there be any issues?

If versioning is a concern, we could make versioned releases of this file generated based on the specified "version" (or commit hash) of the equivalent file on the Numaflow repo. Maybe we could create versions at certain points in time.

whynowy commented 6 days ago

There's no automation in Numaflow for this, and I don't think there will be frequent update on these. If there's any update, most of them would be adding new fields, I can not foresee a removing case.