temperamental "Error: 0" in `terraform plan` caused by unhandled 502 from rollbar

dylan-shipwell commented 2 years ago

502 from rollbar, creates "Error 0"

[DEBUG] provider.terraform-provider-rollbar_v1.6.0: {
  "level": "error",
  "StatusCode": 502,
  "Status": "502 Bad Gateway",
  "ErrorResult": {
    "Err": 0,
    "Message": ""
  },
  "time": "2022-09-08T21:13:43Z"
}

$ terraform plan
│ Error: 0.
│.
│   with rollbar_project.main,
│   on terraform.tf line 17, in resource "rollbar_project" "main":
│   17: resource "rollbar_project" "main" {

relevant portions of TF_LOG=TRACE:

``` # ... [DEBUG] provider.terraform-provider-rollbar_v1.6.0: {"level":"error","StatusCode":502,"Status":"502 Bad Gateway","ErrorResult":{"Err":0,"Message":""},"time":"2022-09-08T21:13:43Z"} [DEBUG] provider.terraform-provider-rollbar_v1.6.0: {"level":"error","error":"0 ","time":"2022-09-08T21:13:43Z","message":"Error listing teams"} [DEBUG] provider.terraform-provider-rollbar_v1.6.0: {"level":"error","error":"0 ","time":"2022-09-08T21:13:43Z","message":"Error listing custom teams"} [DEBUG] provider.terraform-provider-rollbar_v1.6.0: {"level":"error","project_id":575948,"error":"0 ","time":"2022-09-08T21:13:43Z"} [DEBUG] provider.terraform-provider-rollbar_v1.6.0: {"level":"error","projectID":575948,"error":"0 ","time":"2022-09-08T21:13:43Z"} [TRACE] provider.terraform-provider-rollbar_v1.6.0: Called downstream: tf_proto_version=5 tf_provider_addr=provider tf_rpc=ReadResource @module=sdk.proto tf_req_id=62xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx99 tf_resource_type=rollbar_project @caller=github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/tf5server/server.go:558 timestamp=2022-09-08T21:13:43.353Z [TRACE] provider.terraform-provider-rollbar_v1.6.0: Served request: tf_req_id=62xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx99 tf_resource_type=rollbar_project tf_provider_addr=provider tf_rpc=ReadResource @caller=github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/tf5server/server.go:564 @module=sdk.proto tf_proto_version=5 timestamp=2022-09-08T21:13:43.353Z [ERROR] vertex "rollbar_project.main" error: 0 [TRACE] vertex "rollbar_project.main": visit complete, with errors [TRACE] vertex "rollbar_project.main": dynamic subgraph encountered errors: 0 [ERROR] vertex "rollbar_project.main" error: 0 [TRACE] vertex "rollbar_project.main": visit complete, with errors [TRACE] vertex "rollbar_project.main (expand)": dynamic subgraph encountered errors: 0 [ERROR] vertex "rollbar_project.main (expand)" error: 0 [TRACE] vertex "rollbar_project.main (expand)": visit complete, with errors [TRACE] dag/walk: upstream of "output.id (expand)" errored, so skipping [TRACE] dag/walk: upstream of "rollbar_project_access_token.notifications (expand)" errored, so skipping [TRACE] dag/walk: upstream of "rollbar_project_access_token.post_server_item (expand)" errored, so skipping [TRACE] dag/walk: upstream of "output.notifications_api_key (expand)" errored, so skipping [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/rollbar/rollbar\"].notifications_provider" errored, so skipping [TRACE] dag/walk: upstream of "output.post_server_item_token (expand)" errored, so skipping [TRACE] dag/walk: upstream of "rollbar_notification.pd_resolve_item (expand)" errored, so skipping [TRACE] dag/walk: upstream of "rollbar_notification.pd_reactivate_item (expand)" errored, so skipping [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/rollbar/rollbar\"] (close)" errored, so skipping [TRACE] dag/walk: upstream of "rollbar_notification.pd_high_occurrence (expand)" errored, so skipping [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/rollbar/rollbar\"].notifications_provider (close)" errored, so skipping [TRACE] dag/walk: upstream of "(close)" errored, so skipping # ... ╷ │ Error: 0. │. │ with rollbar_project.main, │ on terraform.tf line 17, in resource "rollbar_project" "main": │ 17: resource "rollbar_project" "main" { │. ╵ ```

suggestions:

consider producing a descriptive diagnostic summary text,

so relevant emitted DEBUG log lines and relevant terraform plan output produces meaningful messages.
consider implemented reasonable http request retry control for tempermental server errors,

to reduce the impact of rollbar instability on Operators (especially those of us with way too many states in tf to refresh, where a failure to generate a state refresh is a major time sink)

troubleshooting

for what it's worth, I this symptom reproduced more reliably when TF_LOG=TRACE. for what it's worth, adding TF_LOG=TRACE AND a null resource depends on (erroring vertex) reduced the reproducibility to similar levels as when TF_LOG was unset or set to DEBUG. Both of these observations are likely related to temporal rollbar api stability, not terraform quirks.

resource "null_resource" "tfbug_vertex_right" {
  depends_on = [rollbar_project.main]
}

awendt commented 2 years ago

From what we've seen, the Resty client is not configured to retry: https://github.com/rollbar/terraform-provider-rollbar/blob/db1f5862ed0bceddb14241259fad15dcd0c83dbb/client/client.go#L45-L72

github-actions[bot] commented 2 years ago

:tada: This issue has been resolved in version 1.8.0 :tada:

The release is available on:

GitHub release
v1.8.0

Your semantic-release bot :package::rocket:

rollbar / terraform-provider-rollbar