search

hashicorp / terraform-k8s

Terraform Cloud Operator for Kubernetes
https://learn.hashicorp.com/tutorials/terraform/kubernetes-operator?in=terraform/kubernetes
Mozilla Public License 2.0
454 stars 71 forks source link

Introduce better logging #96

Open rverma-dev opened 3 years ago

rverma-dev commented 3 years ago

Following the guide here

Terraform workspaces created but no run configuration.

Exception:

2021-05-29T00:38:51.072Z    INFO    terraform-k8s   Checking outputs    {"Organization": "nslhub", "WorkspaceID": "ws-fkYap2hAd8ogTkxp", "RunID": ""}
2021-05-29T00:38:51.094Z    INFO    terraform-k8s   Updated outputs {"Organization": "nslhub", "WorkspaceID": "ws-fkYap2hAd8ogTkxp"}
2021-05-29T00:38:51.094Z    INFO    terraform-k8s   Updating secrets    {"name": "greetings-outputs"}
2021-05-29T00:38:51.094Z    DEBUG   controller-runtime.manager.events   Normal  {"object": {"kind":"Workspace","namespace":"ci","name":"greetings","uid":"220cacfe-7f6c-4717-9258-ad02aec80e87","apiVersion":"app.terraform.io/v1alpha1","resourceVersion":"16909471"}, "reason": "WorkspaceEvent", "message": "Updated outputs for run "}
2021-05-29T00:38:51.564Z    INFO    terraform-k8s   Starting module backed run  {"Organization": "nslhub", "Name": "greetings", "Namespace": "ci"}
2021-05-29T00:38:51.775Z    ERROR   controller  Reconciler error    {"controller": "workspace-controller", "name": "greetings", "namespace": "ci", "error": "resource not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:246
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
    /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90

This makes it very tough to understand the issue. Also observed 

Status:
  Config Version ID:  
  Run ID:             
  Run Status:         
  Workspace ID:       ws-fkYap2hAd8ogTkxp
rverma-dev commented 3 years ago

The issue starts resurfacing again, 

2021-06-12T07:59:02.740Z    ERROR   terraform-k8s   Failed to update output status  {"error": "workspaces.app.terraform.io \"s3-logs-ap-south-1\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/hashicorp/terraform-k8s/workspacehelper.(*WorkspaceHelper).reconcileWorkspace
    /workspace/workspacehelper/workspace_helper.go:142
github.com/hashicorp/terraform-k8s/workspacehelper.(*WorkspaceHelper).Reconcile
    /workspace/workspacehelper/workspace_helper.go:442
github.com/hashicorp/terraform-k8s/controllers.(*WorkspaceReconciler).Reconcile
    /workspace/controllers/workspace_controller.go:63
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
    /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90

On further debugging its failing here https://github.com/hashicorp/terraform-k8s/blob/973ac76005280a5aff9656fcb39a436dcda481c4/workspacehelper/workspace_helper.go#L141

There is no way to identify the cause of this. It seems it is failing when updating the workspace id in workspace resource.

mottetm commented 2 years ago

@rverma-nsl Did you ever figure out what was the root cause of your first issue? Facing the same but can't figure out what the issue is...