Open abhi8893 opened 2 days ago
As far as I recall, no, because this would require to use different component not only node selectors and currently every node is translated to ContainerOp.
Thanks @em-pe. Yep deep dived into the code and makes sense. I would love to contribute to this package, as I loved the concept of translating the kedro nodes into vertex ai nodes, grouping logic etc. It seems this is EXACTLY what I want, however with a bit more flexibility. I am embarking on a VertexAI + CloudComposer + Kedro MLOPS journey, and would share my learnings, and hopefully contribute back :)
As I understand, through a tag based system, the plugin can assign different compute targets for each node as follows:
Is there support for using Dataproc serverless components for a pipeline / node as well in this plugin?