Support for `wait_for_crds` in `google_gke_hub_feature`

brettcurtis commented 2 weeks ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Description

The plan fails when building MCS clusters and utilizing the k8s CRD: MultiClusterService in subsequent workflows since the CRDs are not fully installed yet.

│ Error: API did not recognize GroupVersionKind from manifest (CRD may not be installed)
│ 
│   with module.kubernetes_istio.kubernetes_manifest.istio_gateway_mcs[0],
│   on .terraform/modules/kubernetes_istio/regional/main.tf line 320, in resource "kubernetes_manifest" "istio_gateway_mcs":
│  320: resource "kubernetes_manifest" "istio_gateway_mcs" {
│ 
│ no matches for kind "MultiClusterService" in group "networking.gke.io"

It would be nice if this was managed at the resource vs. some W/A on the customer side.

New or Affected Resource(s)

google_gke_hub_feature

Potential Terraform Configuration

No response

References

No response

SarahFrench commented 1 week ago

This looks like an instance of this : https://github.com/hashicorp/terraform-provider-kubernetes/issues/1782

I'll transfer the issue to the hashicorp/terraform-provider-kubernetes repo

arybolovlev commented 1 week ago

Hi @brettcurtis,

This is expected behavior due to how Terraform and the kubernetes_manifest resource are designed. CRD that handles the resource that you want to create should be installed first.

brettcurtis commented 1 week ago

Thanks for looking. I was talking about the google_gke_hub_feature resource in the google provider. I'm asking for the feature request there if possible. Maybe we can transfer it back @SarahFrench ?

When you enable MCS:

resource "google_gke_hub_feature" "multi_cluster_service_discovery" {
  name     = "multiclusterservicediscovery"
  location = "global"
  project  = local.global_infra.project_id

}

I believe it installs some CDRs as a result (maybe they are part of multiclusteringress I have both). I was simply hoping we could wait for those to be deployed before moving forward. Also just to point out my manifest workflow runs after this workflow has completed successfully so it must take at least a couple minutes for the CRDs t be fully installed on the backend after the resource completes.

The kubernetes_ingress resource has something like what I'm looking for here, wait_for_laod_balancer.

arybolovlev commented 1 week ago

manifest workflow runs after this workflow has completed successfully

Does this all happen within a single Terraform run?

brettcurtis commented 1 week ago

No, different Terraform runs.

arybolovlev commented 1 week ago

Thank you! When you re-run the one that executes kubernetes_manifest, does it succeed?

brettcurtis commented 1 week ago

Yes it does, usually less than a minute later, so it's quick.

brettcurtis commented 1 week ago

I think I have a theory around this looking into the logs a bit more. The gke_hub_feature for "multiclusterservicediscovery" doesn't depend on cluster or any other hub resources. For example this through line 303

So I suspect it's just an async process that looks for clusters that should have the feature and applies the CRDs. I'm going to test if a explicit dependency will help "speed up" the process. The idea being it may create the CRDs on apply and completion.

EDIT I tested the explicit dep, and it didn't help.

hashicorp / terraform-provider-kubernetes