Add support for GKEStartPodOperator

[davidvanrooij]

It would be great to add support for creating pods through GKEStartPodOperator to provide better integration with Google Kubernetes Engine (GKE).

GKEStartPodOperator extends KubernetesPodOperator to provide authorization using Google Cloud credentials. There is no need to manage the kube_config file, as it will be generated automatically. All Kubernetes parameters (except config_file) are also valid for the GKEStartPodOperator.

The chief benefit of using GKEStartPodOperator is to easily create pods in a different cluster than the airflow instance which is less trivial with KubernetesPodOperator.

Happy to have a stab at creating a PR.

[jlaneve]

I think the right approach may be to do this with https://github.com/astronomer/astronomer-cosmos/issues/427. There's probably a few other operators we'd want to support, so generalizing this makes more sense IMO. Curious to get your thoughts though!

[davidvanrooij]

I like the approach of adding the ability to have a custom subclass since that is more generalizable. I had briefly attempted this by monkey-patching the package to drop in a different subclass but that very quickly became unmaintainable.

I'll keep an eye out on https://github.com/astronomer/astronomer-cosmos/issues/427 in the meantime. Let me know if I can assist in any way.

[jlaneve]

Agree, monkey patching isn't sustainable in the long-term and I'd rather just support this directly in Cosmos 🙂

Opening this issue is super helpful! And if you have any other thoughts/ideas on #427 definitely feel free to drop a comment on that issue

[tatiana]

@davidvanrooij #427 is available as part of Cosmos 1.2 - if you have a chance, please give it a try and let us know how it goes.

An alternative approach to run Cosmos in GCP is to wait for the PR #611 to be released and use ExecutionMode.VIRTUALENV (https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#virtualenv).

[davidvanrooij]

@tatiana Awesome, thanks for the update. I'll give that approach a try and get back to you.

[dosubot[bot]]

Hi, @davidvanrooij

I'm helping the Cosmos team manage their backlog and am marking this issue as stale. The issue you opened requests support for GKEStartPodOperator to enhance integration with Google Kubernetes Engine (GKE). There have been discussions in the comments about a more general approach to support multiple operators, as well as an alternative approach using Cosmos 1.2 and PR #611. You expressed gratitude and mentioned plans to try the suggested approach.

Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and contributions to the Cosmos repository!