Allow ConfigConnector to be enabled on GKE Autopilot clusters.

[toumorokoshi]

Describe the feature or resource

From https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/411#issuecomment-903166182, the new GKE autopilot clusters do not support enabling the Config Connector add-on.

This prevents the usage of Config Connector on autopilot clusters, which is a popular option to eliminate some managed of the underlying nodepools.

Importance

This may force users to use vanilla GKE clusters or install Config Connector manually, even if they want to use autopilot clusters.

[jcanseco]

More details: Unsupported cluster features for GKE Autopilot.

[andrewmclagan]

Config connector is hugely beneficial, is there any specific reason it cannot be used with Autopilot clusters?

[xiaobaitusi]

Hi @andrewmclagan, thanks for commenting on this thread.

Unfortunately, AFAIK, this is the technical decision made on the GKE side. We will reach out to the GKE team internally for better understanding. We would also suggest to open a support case to GKE directly to clarify priority and the need directly.

[xiaobaitusi]

Hi @andrewmclagan, could you also clarify your use case a little? If you want to have a managed instance of Config connector, we now recommend Config Controller that has quicker release cadence (than GKE addon) and build-in integration with Config Sync and Policy controller.

You can also manually install Config connector into autopilot clusters and see if that works.

After discussing with teams, we currently don't have a concrete plan to support Config Connector on GKE Autopilot clusters. Keep this issue open to capture more data points for this need.

[lostick]

@xiaobaitusi is the Config Connector Operator supported in Autopilot?

[jcanseco]

Hi @lostick, we are not completely sure at the moment since we haven't been able to test it yet. Autopilot is meant to be a simpler version of GKE that exposes less configurability and supports less features. There may be features that Config Connector relies on that Autopilot doesn't support (e.g. I do remember seeing Autopilot having limitations around webhooks, but I'm not sure if that's the case anymore).

Until we can be more confident about recommending Autopilot as an option, an alternative option we can recommend today is Config Controller. It is a managed offering of Config Connector that is managed and hosted by GCP.

[leeren]

@jcanseco What is the recommended way of declaratively managing Google Cloud infrastructure for projects using GKE Autopilot clusters?

[diviner524]

@leeren Great question. This depends on how you are organizing the configurations, and dividing between infrastructure and workload/application artifacts.

By default our recommendation is Config Controller. You can create a dedicated Config Controller instance to manage GCP infrastructure, so you don't need to install KCC in any GKE autopilot clusters.

Again it depends on your design/architecture, so our recommendation may not be a working solution for you. If you can share more details on your use case we can give better suggestions.

[leeren]

Thanks for the reply @diviner524. For config controller, is the idea that after setting up config controller, everything can be declaratively built & deployed via kpt / config-sync / config-controller?

Are there any good examples of this? Basically wondering if using Terraform is still worth having for initial project creation / config controller initialization / enabling pre-config controller APIs.

[jcanseco]

Yep that's correct @leeren. We unfortunately don't have good docs on this yet, but it's in our backlog.

The closest thing we have to an example is the Landing Zone blueprint. It uses kpt + Config Sync + Config Controller, but it's a fairly verbose page as you can see.

[trevex]

I am interested in running Config Connector as a cluster addon in GKE Autopilot and went ahead and tested the operator. Unfortunately it does not work without issues. The first problem I ran into is the cnrm-resource-stats-recorder. It uses hostNetwork and hostPort: https://github.com/GoogleCloudPlatform/k8s-config-connector/blob/master/operator/channels/packages/configconnector/1.92.0/cluster/workload-identity/0-cnrm-system.yaml#L1629

From my understanding that also means ConfigConnector operator will not work on GKE clusters using PSPs without manual fixes.

[dudo]

So the recommendation for Config Controller doesn't feel like it answers the question here. Config Connector allows me to define my database resource (for instance) right next to the deployment that's going to use it. That's incredible... just like Ingress managing a Load Balancer. Love it.

The Controller is another DSL to learn and manage, too, with its own git-ops pipeline. That's no good. I have my cluster managed already, and Controller just adds complexity where I'm looking to simplify.

In the case of databases, I'd rather manage them through the UI (or maybe terraform, depending on the number) than setup Controller.

Please please please enable ConfigConnector for Autopilot.

[jcanseco]

@trevex Thank you for that information! That is actually quite valuable.

I will say that we are trying to get rid of cnrm-resource-stats-recorder's reliance on hostNetwork and hostPort, so perhaps Config Connector will be runnable on GKE Autopilot when that happens, though we do not have any ETA right now.

@dudo Thank you, that is valuable information for us to know as well (the limitations of Config Controller from POV of users). I will surface this to our team so people are aware.

[diviner524]

@dudo Thank you for the feedback on the challenge of adopting Config Controller!

We are very interested to learn more on the limitations you mentioned:

1. You mentioned the preference of defining database resource right next to the deployment which uses it. Does this imply you want to share some information between the database resource and the deployment? If so, what exactly are the things that needs to be shared? database name? Passwords?
2. Are you using a different git-ops pipeline to manage your cluster than the option provided by Config Controller (Config Sync)? If so, will it help if we enable Config Controller to support your own git-ops pipeline, ex: Argo CD, Flux?

[emgeee]

I stumbled on this thread after trying to use ConfigConnector on an Autopilot cluster and I must echo @dudo's sentiment here. ConfigController appears to be a heavy handed tool designed with larger org use-cases in mind. It seems a bit counter-intuitive to recommend that for use with a tool (GKE autopilot) that's supposed to simplify things.

To give an example of my current situation -- I am trying to deploy a simple Python API to a GKE autopilot cluster. I am storing the deployment.yaml file in the same git repository as the code itself. Following best practices, I would like to use workload identities and therefor need to create a new GSA for my workload to assume while running. According to the official docs this should be doable via ConfigConnector and would present a very convenient way of defining resources in the same place.

Note: I do not personally care about git-ops workflows right now as I am a team of 1 and am fine manually applying changes to my clusters.

Because ConfigConnector is not supported on Autopilot clusters, I will either have to add the relevant resources via the web interface or cobble together some bash scripts to execute the appropriate gcloud commands.

[arikkfir]

Completely agree with both @dudo and @emgeee.

This forced us to revert back to GKE Standard mode.

It almost looks like Google intentionally directs people to use Config Controller instead of ConfigConnector because its a billable service:

• $0.1 24 30 (flat fee of ConfigController), plus:
• $0.1 24 30 (flat fee for ConfigController's underlying separate GKE cluster) plus:
• $?? for that GKE cluster's resources.

(Billing information source)

[diviner524]

@arikkfir Thanks for the feedback! We're aware of the requirement to run Config Connector on Autopilot clusters and we're currently working on publishing instructions for manual installation on Autopilot.

With regard to Config Controller, we'd like to encourage customers to consider using Config Controller for their installation, as it's a managed service that includes Config Connector as an out-of-the-box feature. While we do acknowledge that there are management and cluster fees associated with this option (listed as a disadvantage in our doc [1] 😄), we believe the benefits of a managed service can outweigh the costs.

Using Config Controller can save users a lot of effort when it comes to installation, upgrades, and monitoring of Config Connector. That said, we do recognize the limitations of this option, as mentioned by @dudo and @emgeee.

Please stay tuned for our next release, which will officially support manual installation of Config Connector on Autopilot. In the meantime, if you have any further questions or concerns, please don't hesitate to reach out to us.

[1] https://cloud.google.com/config-connector/docs/concepts/installation-types

[arikkfir]

Thanks @diviner524 - looking forward to Config Connector's next release then.

[clhodapp]

Requiring a manual installation still makes Config Connector on GKE Autopilot a second-class citizen versus Config Connector on GKE Standard. It will have a much larger maintenance burden. Hopefully this release serves as an enabling step to bringing the Config Connector add-on to GKE Autopilot.

[vszal]

Following up on @diviner524's post above, manual installation is now supported on GKE Autopilot: https://cloud.google.com/config-connector/docs/how-to/advanced-install

[AeroNotix]

Question remains, then, if config-connector itself is "supported" via a manual installation on GKE Autopilot - what's the hurdle to make it available as an addon?

Seems like the left hand doesn't know what the right hand is doing over at Google HQ.