search

fluxcd / terraform-provider-flux

Terraform and OpenTofu provider for bootstrapping Flux
https://registry.terraform.io/providers/fluxcd/flux/latest
Apache License 2.0
330 stars 88 forks source link

Provider crashed with context canceled #476

Closed 0x46616c6b closed 3 weeks ago

0x46616c6b commented 11 months ago

I'm sad raising a issue again which I already found closed (#447): The provider 1.0.0-rc.3 crashes with the following message: plugin6.(*GRPCProvider).ApplyResourceChange: error="rpc error: code = Canceled desc = context canceled

In the same workspace a Kubernetes Cluster on Azure with AKS is deployed. I manually checked the connectivity with cURL and can reach the API and connect with the TLS certificates.

Also enabled as much logging I can TF_LOG=TRACE.

2023-05-23T19:53:01.391Z [TRACE] GRPCProvider.v6: ApplyResourceChange
2023-05-23T19:53:01.393Z [TRACE] provider.terraform-provider-flux_v1.0.0-rc.3: Received request: tf_proto_version=6.3 tf_provider_addr=registry.terraform.io/fluxcd/flux tf_resource_type=flux_bootstrap_git tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov6/tf6server/server.go:803 @module=sdk.proto tf_req_id=ce635d8e-fa36-5a2e-bb8f-9aca99bb9ce8 timestamp=2023-05-23T19:53:01.391Z
2023-05-23T19:53:01.393Z [TRACE] provider.terraform-provider-flux_v1.0.0-rc.3: Sending request downstream: tf_provider_addr=registry.terraform.io/fluxcd/flux tf_proto_version=6.3 tf_req_id=ce635d8e-fa36-5a2e-bb8f-9aca99bb9ce8 tf_resource_type=flux_bootstrap_git tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov6/internal/tf6serverlogging/downstream_request.go:17 @module=sdk.proto timestamp=2023-05-23T19:53:01.391Z
2023-05-23T19:53:01.393Z [TRACE] provider.terraform-provider-flux_v1.0.0-rc.3: Checking ResourceTypes lock: tf_provider_addr=registry.terraform.io/fluxcd/flux tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-framework@v1.2.0/internal/fwserver/server.go:339 @module=sdk.framework tf_req_id=ce635d8e-fa36-5a2e-bb8f-9aca99bb9ce8 tf_resource_type=flux_bootstrap_git timestamp=2023-05-23T19:53:01.391Z
2023-05-23T19:53:01.393Z [TRACE] provider.terraform-provider-flux_v1.0.0-rc.3: Checking ResourceSchemas lock: @caller=github.com/hashicorp/terraform-plugin-framework@v1.2.0/internal/fwserver/server.go:413 tf_rpc=ApplyResourceChange tf_req_id=ce635d8e-fa36-5a2e-bb8f-9aca99bb9ce8 tf_resource_type=flux_bootstrap_git @module=sdk.framework tf_provider_addr=registry.terraform.io/fluxcd/flux timestamp=2023-05-23T19:53:01.391Z
2023-05-23T19:53:01.393Z [TRACE] provider.terraform-provider-flux_v1.0.0-rc.3: ApplyResourceChange received no PriorState, running CreateResource: @caller=github.com/hashicorp/terraform-plugin-framework@v1.2.0/internal/fwserver/server_applyresourcechange.go:42 tf_provider_addr=registry.terraform.io/fluxcd/flux @module=sdk.framework tf_req_id=ce635d8e-fa36-5a2e-bb8f-9aca99bb9ce8 tf_resource_type=flux_bootstrap_git tf_rpc=ApplyResourceChange timestamp=2023-05-23T19:53:01.392Z
2023-05-23T19:53:01.393Z [TRACE] provider.terraform-provider-flux_v1.0.0-rc.3: Resource implements ResourceWithConfigure: tf_resource_type=flux_bootstrap_git tf_provider_addr=registry.terraform.io/fluxcd/flux tf_req_id=ce635d8e-fa36-5a2e-bb8f-9aca99bb9ce8 tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-framework@v1.2.0/internal/fwserver/server_createresource.go:43 @module=sdk.framework timestamp=2023-05-23T19:53:01.392Z
2023-05-23T19:53:01.393Z [DEBUG] provider.terraform-provider-flux_v1.0.0-rc.3: Calling provider defined Resource Configure: @caller=github.com/hashicorp/terraform-plugin-framework@v1.2.0/internal/fwserver/server_createresource.go:50 @module=sdk.framework tf_req_id=ce635d8e-fa36-5a2e-bb8f-9aca99bb9ce8 tf_resource_type=flux_bootstrap_git tf_rpc=ApplyResourceChange tf_provider_addr=registry.terraform.io/fluxcd/flux timestamp=2023-05-23T19:53:01.392Z
2023-05-23T19:53:01.393Z [DEBUG] provider.terraform-provider-flux_v1.0.0-rc.3: Called provider defined Resource Configure: @module=sdk.framework tf_provider_addr=registry.terraform.io/fluxcd/flux tf_resource_type=flux_bootstrap_git tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-framework@v1.2.0/internal/fwserver/server_createresource.go:52 tf_req_id=ce635d8e-fa36-5a2e-bb8f-9aca99bb9ce8 timestamp=2023-05-23T19:53:01.392Z
2023-05-23T19:53:01.393Z [DEBUG] provider.terraform-provider-flux_v1.0.0-rc.3: Calling provider defined Resource Create: tf_provider_addr=registry.terraform.io/fluxcd/flux tf_req_id=ce635d8e-fa36-5a2e-bb8f-9aca99bb9ce8 tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-framework@v1.2.0/internal/fwserver/server_createresource.go:96 @module=sdk.framework tf_resource_type=flux_bootstrap_git timestamp=2023-05-23T19:53:01.392Z
2023-05-23T19:53:01.393Z [INFO]  provider.terraform-provider-flux_v1.0.0-rc.3: create timeout configuration not found, using provided default: @caller=github.com/hashicorp/terraform-plugin-framework-timeouts@v0.3.1/resource/timeouts/timeouts.go:105 tf_req_id=ce635d8e-fa36-5a2e-bb8f-9aca99bb9ce8 tf_resource_type=flux_bootstrap_git tf_rpc=ApplyResourceChange @module=flux tf_provider_addr=registry.terraform.io/fluxcd/flux timestamp=2023-05-23T19:53:01.392Z
2023-05-23T19:53:01.412Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:06.170Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:06.413Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:11.172Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:11.414Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:16.172Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:16.415Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:21.239Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:21.474Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:26.340Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:26.583Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:31.443Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:31.959Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:36.672Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:37.232Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:41.782Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:42.376Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:46.885Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:47.524Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:52.011Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:52.760Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:53:57.431Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:53:58.091Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:54:12.060Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/fluxcd/flux\"] (close)"
2023-05-23T19:54:12.782Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" is waiting for "flux_bootstrap_git.this"
2023-05-23T19:54:12.822Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/fluxcd/flux/1.0.0-rc.3/linux_amd64/terraform-provider-flux_v1.0.0-rc.3 pid=105 error="signal: killed"
2023-05-23T19:54:12.825Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Canceled desc = context canceled"
2023-05-23T19:54:12.828Z [ERROR] plugin6.(*GRPCProvider).ApplyResourceChange: error="rpc error: code = Canceled desc = context canceled"
2023-05-23T19:54:12.834Z [TRACE] maybeTainted: flux_bootstrap_git.this encountered an error during creation, so it is now marked as tainted
2023-05-23T19:54:12.837Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for flux_bootstrap_git.this
2023-05-23T19:54:12.838Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for flux_bootstrap_git.this
2023-05-23T19:54:12.838Z [TRACE] evalApplyProvisioners: flux_bootstrap_git.this is tainted, so skipping provisioning
2023-05-23T19:54:12.839Z [TRACE] maybeTainted: flux_bootstrap_git.this was already tainted, so nothing to do
2023-05-23T19:54:12.839Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for flux_bootstrap_git.this
2023-05-23T19:54:12.839Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for flux_bootstrap_git.this
2023-05-23T19:54:12.843Z [ERROR] vertex "flux_bootstrap_git.this" error: Request cancelled
2023-05-23T19:54:12.843Z [TRACE] vertex "flux_bootstrap_git.this": visit complete, with errors
2023-05-23T19:54:12.844Z [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/fluxcd/flux\"] (close)" errored, so skipping
2023-05-23T19:54:12.844Z [TRACE] dag/walk: upstream of "root" errored, so skipping
2023-05-23T19:54:12.845Z [DEBUG] cloud/state: state read serial is: 19; serial is: 19
2023-05-23T19:54:12.845Z [DEBUG] cloud/state: state read lineage is: c952e7c2-bcca-5d3b-8a38-8622941e4e02; lineage is: c952e7c2-bcca-5d3b-8a38-8622941e4e02
2023-05-23T19:54:13.355Z [DEBUG] provider: plugin exited

Configuration

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.54.0"
    }

    azuread = {
      source  = "hashicorp/azuread"
      version = "2.38.0"
    }

    flux = {
      source  = "fluxcd/flux"
      version = "1.0.0-rc.3"
    }

    github = {
      source  = "integrations/github"
      version = "5.24.0"
    }

    tls = {
      source  = "hashicorp/tls"
      version = "4.0.4"
    }
  }

  backend "remote" {
    hostname     = "<redacted>"
    organization = "<redacted>"

    workspaces {
      name = "<redacted>"
    }
  }
}
provider "azurerm" {
  features {}
}

provider "azuread" {}

provider "flux" {
  kubernetes = {
    host                   = module.aks.admin_host
    client_certificate     = base64decode(module.aks.admin_client_certificate)
    client_key             = base64decode(module.aks.admin_client_key)
    cluster_ca_certificate = base64decode(module.aks.admin_cluster_ca_certificate)
  }

  git = {
    url = "ssh://git@github.com/org/repo.git"
    ssh = {
      username    = "git"
      private_key = tls_private_key.this.private_key_pem
    }
  }
}

provider "github" {
  app_auth {
    id              = var.github_app_id
    installation_id = var.github_app_installation_id
    pem_file        = var.github_app_pem_file
  }
}

data "azurerm_client_config" "current" {}
resource "tls_private_key" "this" {
  algorithm = "ED25519"
}

resource "github_repository_deploy_key" "this" {
  repository = var.repository
  title      = "Flux Key for Cluster"
  key        = tls_private_key.this.public_key_openssh
  read_only  = true
}

resource "azurerm_resource_group" "this" {
  name     = "cluster"
  location = var.location
}

resource "azurerm_virtual_network" "this" {
  name                = "cluster-vnet"
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name

  address_space = ["10.10.0.0/16"]
}

resource "azurerm_subnet" "this" {
  name                 = "cluster-subnet"
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.this.name

  address_prefixes = ["10.10.0.0/24"]
}

module "aks" {
  source  = "Azure/aks/azurerm"
  version = "6.8.0"

  prefix                    = "cluster"
  resource_group_name       = azurerm_resource_group.this.name
  kubernetes_version        = "1.25"
  automatic_channel_upgrade = "patch"

  agents_size               = "Standard_D16s_v5"
  agents_count              = 2
  agents_pool_name          = "nodepool"
  agents_availability_zones = ["1", "2"]
  # Used when agent_size is changed
  temporary_name_for_rotation = "temp"

  node_resource_group = "cluster-nodepool"

  enable_auto_scaling    = false
  enable_host_encryption = true

  admin_username                  = null
  azure_policy_enabled            = true
  log_analytics_workspace_enabled = false

  network_plugin = "azure"
  network_policy = "azure"
  vnet_subnet_id = azurerm_subnet.this.id

  rbac_aad                          = true
  rbac_aad_managed                  = true
  role_based_access_control_enabled = true

  depends_on = [azurerm_resource_group.this]
}

resource "flux_bootstrap_git" "this" {
  path = "path/to/cluster"

  depends_on = [module.aks]
}
mcortinas commented 11 months ago

Wow! Same for me! My environment is running on top of GKE on GCP, v1.24.11-gke.1000 I opened this other issue before read this ..... https://github.com/fluxcd/terraform-provider-flux/issues/487

swade1987 commented 1 month ago

Hello @0x46616c6b ,

I hope you're doing well! I'm the newest contributor to this repository, and I'm currently in the process of issue grooming to ensure that all concerns are addressed promptly and efficiently.

I noticed this issue you reported and wanted to check in with you to see if it's still affecting your work. Your feedback is invaluable to us, and any additional insights or updates you can share would be greatly appreciated to help us understand and solve the problem more effectively.

If this issue has been resolved, could you please share how it was fixed? This information could be incredibly helpful to others in the community facing similar problems. It would also allow us to close this issue with a clear resolution. In case the issue is still open and troubling you, let's work together to find a solution. Your satisfaction and the smooth functioning of our project are our top priorities.

Thank you for your time and contributions to our community. Looking forward to your response!

Best regards,

Steve

Can I propose that you upgrade to the latest version of the provider and use the flux_bootstrap_git resource. If you still have issues please let me know.

0x46616c6b commented 3 weeks ago

Hey, i don't use the provider anymore and can't help to reproduce this issue again. Sorry.