search

rollbar / terraform-provider-rollbar

Terraform provider for Rollbar
https://rollbar.com
MIT License
28 stars 13 forks source link

Plugin still crashing after upgrade to 1.10.0 #346

Closed JJaakson closed 1 year ago

JJaakson commented 1 year ago

Similarly to #340, we also encountered concurrent map writes fatal errors in our terraform plan. Fix implemented with 1.10.0 doesn't seem to fix the problem. Logs (ran it in DEBUG mode, but since we have thousands of objects, there are a lot of logs), overall seems to be problem with access_tokens API:

...
2023-02-01T13:36:48.216+0200 [DEBUG] ReferenceTransformer: "module.{...}.rollbar_project_access_token.additional_tokens[1]" references: []
2023-02-01T13:36:48.216+0200 [DEBUG] provider.terraform-provider-rollbar_v1.10.0: {"level":"info","projectID":....,"time":"2023-02-01T13:36:48+02:00","message":"Reading Rollbar project resource"}
2023-02-01T13:36:48.217+0200 [DEBUG] provider.terraform-provider-rollbar_v1.10.0: fatal error: concurrent map iteration and map write{"level":"debug","projectID":...,"time":"2023-02-01T13:36:48+02:00","message":"Reading project from API"}
2023-02-01T13:36:48.217+0200 [DEBUG] provider.terraform-provider-rollbar_v1.10.0: 
...
╷
│ Error: Request cancelled
│ 
│   with module......rollbar_project.project,
│   on .terraform/modules/.../main.tf line 2, in resource "rollbar_project" "project":
│    2: resource "rollbar_project" "project" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.....rollbar_project.project,
│   on .terraform/modules/.../main.tf line 2, in resource "rollbar_project" "project":
│    2: resource "rollbar_project" "project" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵

Stack trace from the terraform-provider-rollbar_v1.10.0 plugin:

fatal error: concurrent map iteration and map write{"level":"debug","projectID":...,"time":"2023-02-01T13:36:48+02:00","message":"Reading project from API"}

goroutine 176248 [running]:
runtime.throw({0x1053ef5ec, 0x26})
    runtime/panic.go:1198 +0x54 fp=0x140008725b0 sp=0x14000872580 pc=0x104d80ec4
runtime.mapiternext(0x14000872728)
    runtime/map.go:858 +0x538 fp=0x14000872630 sp=0x140008725b0 pc=0x104d5af68
github.com/go-resty/resty/v2.parseRequestHeader(0x140004d21c0, 0x14001375b00)
    github.com/go-resty/resty/v2@v2.5.0/middleware.go:100 +0x78 fp=0x14000872790 sp=0x14000872630 pc=0x105370838
github.com/go-resty/resty/v2.(*Client).execute(0x140004d21c0, 0x14001375b00)
    github.com/go-resty/resty/v2@v2.5.0/client.go:850 +0x1a4 fp=0x14000872950 sp=0x14000872790 pc=0x10536ee44
github.com/go-resty/resty/v2.(*Request).Execute.func1()
    github.com/go-resty/resty/v2@v2.5.0/request.go:725 +0xb0 fp=0x140008729d0 sp=0x14000872950 pc=0x105377620
github.com/go-resty/resty/v2.Backoff(0x14000872c20, {0x14000872c00, 0x4, 0x4})
    github.com/go-resty/resty/v2@v2.5.0/retry.go:93 +0x138 fp=0x14000872b00 sp=0x140008729d0 pc=0x105378bc8
github.com/go-resty/resty/v2.(*Request).Execute(0x14001375b00, {0x1053d2f70, 0x3}, {0x14000938900, 0x3b})
    github.com/go-resty/resty/v2@v2.5.0/request.go:719 +0x424 fp=0x14000872c70 sp=0x14000872b00 pc=0x105377484
github.com/go-resty/resty/v2.(*Request).Get(...)
    github.com/go-resty/resty/v2@v2.5.0/request.go:645
github.com/rollbar/terraform-provider-rollbar/client.(*RollbarAPIClient).ListTeamProjectIDs(0x140004a2348, 0x6bf35)
    github.com/rollbar/terraform-provider-rollbar/client/team.go:326 +0x6b8 fp=0x14000873150 sp=0x14000872c70 pc=0x1053b1158
github.com/rollbar/terraform-provider-rollbar/client.(*RollbarAPIClient).FindProjectTeamIDs(0x140004a2348, 0x922d0)
    github.com/rollbar/terraform-provider-rollbar/client/project.go:222 +0x3d4 fp=0x140008734a0 sp=0x14000873150 pc=0x1053a6334
github.com/rollbar/terraform-provider-rollbar/rollbar.resourceProjectRead({0x105652ce8, 0x140001835c0}, 0x1400063db80, {0x10556cb80, 0x14000495050})
    github.com/rollbar/terraform-provider-rollbar/rollbar/resource_project.go:187 +0x594 fp=0x140008737f0 sp=0x140008734a0 pc=0x1053c4984
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x140001c8540, {0x105652c78, 0x1400079d1c0}, 0x1400063db80, {0x10556cb80, 0x14000495050})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:358 +0x118 fp=0x14000873860 sp=0x140008737f0 pc=0x1053382b8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0x140001c8540, {0x105652c78, 0x1400079d1c0}, 0x14000696a90, {0x10556cb80, 0x14000495050})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:635 +0x388 fp=0x14000873910 sp=0x14000873860 pc=0x105339cc8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0x140000c33f8, {0x105652c78, 0x1400079d1c0}, 0x1400079d240)
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/grpc_provider.go:576 +0x5f4 fp=0x14000873a50 sp=0x14000873910 pc=0x10532ef84
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0x14000127080, {0x105652d20, 0x1400054ea50}, 0x14000182000)
    github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/tf5server/server.go:553 +0x3ac fp=0x14000873b40 sp=0x14000873a50 pc=0x10521213c
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x10560b8a0, 0x14000127080}, {0x105652d20, 0x1400054ea50}, 0x14000679f20, 0x0)
    github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x1c0 fp=0x14000873bb0 sp=0x14000873b40 pc=0x105201070
google.golang.org/grpc.(*Server).processUnaryRPC(0x140001baa80, {0x10565eb38, 0x140004f0000}, 0x1400132a700, 0x140003a8630, 0x105b11ed0, 0x0)
    google.golang.org/grpc@v1.36.0/server.go:1217 +0xc38 fp=0x14000873e30 sp=0x14000873bb0 pc=0x105164708
google.golang.org/grpc.(*Server).handleStream(0x140001baa80, {0x10565eb38, 0x140004f0000}, 0x1400132a700, 0x0)
    google.golang.org/grpc@v1.36.0/server.go:1540 +0xa34 fp=0x14000873f60 sp=0x14000873e30 pc=0x105168604
google.golang.org/grpc.(*Server).serveStreams.func1.2(0x14000484520, 0x140001baa80, {0x10565eb38, 0x140004f0000}, 0x1400132a700)
    google.golang.org/grpc@v1.36.0/server.go:878 +0x94 fp=0x14000873fb0 sp=0x14000873f60 pc=0x1051625f4
runtime.goexit()
    runtime/asm_arm64.s:1133 +0x4 fp=0x14000873fb0 sp=0x14000873fb0 pc=0x104db3804
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.36.0/server.go:876 +0x1f0

goroutine 1 [select, 9 minutes]:
github.com/hashicorp/go-plugin.Serve(0x1400014da18)
    github.com/hashicorp/go-plugin@v1.4.1/server.go:469 +0x12dc
github.com/hashicorp/terraform-plugin-sdk/v2/plugin.Serve(0x140000aaa20)
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/plugin/serve.go:147 +0x98c
main.main()
    github.com/rollbar/terraform-provider-rollbar/main.go:54 +0x4b4

goroutine 19 [select, 9 minutes]:
github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv(0x140003a8360)
    github.com/hashicorp/go-plugin@v1.4.1/grpc_broker.go:121 +0x6c
github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0x140000d1860)
    github.com/hashicorp/go-plugin@v1.4.1/grpc_broker.go:411 +0x70
created by github.com/hashicorp/go-plugin.(*GRPCServer).Init
    github.com/hashicorp/go-plugin@v1.4.1/grpc_server.go:85 +0x4f8

goroutine 20 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x12cb32950, 0x72)
    runtime/netpoll.go:303 +0xb4
internal/poll.(*pollDesc).wait(0x140000aaf78, 0x72, 0x1)
    internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
    internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140000aaf60, {0x14000210000, 0x1000, 0x1000})
    internal/poll/fd_unix.go:167 +0x1dc
os.(*File).read(...)
    os/file_posix.go:32
os.(*File).Read(0x140000a8a60, {0x14000210000, 0x1000, 0x1000})
    os/file.go:119 +0x74
bufio.(*Reader).Read(0x14000080f50, {0x14000212000, 0x400, 0x400})
    bufio/bufio.go:227 +0x20c
github.com/hashicorp/go-plugin.copyChan({0x105667b50, 0x14000126c80}, 0x1400009cba0, {0x105642380, 0x140000a8a60})
    github.com/hashicorp/go-plugin@v1.4.1/grpc_stdio.go:181 +0x144
created by github.com/hashicorp/go-plugin.newGRPCStdioServer
    github.com/hashicorp/go-plugin@v1.4.1/grpc_stdio.go:37 +0x94

goroutine 21 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x12cb32770, 0x72)
    runtime/netpoll.go:303 +0xb4
internal/poll.(*pollDesc).wait(0x140000ab038, 0x72, 0x1)
    internal/poll/fd_poll_runtime.go:84 +0x38
internal/poll.(*pollDesc).waitRead(...)
    internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140000ab020, {0x14000211000, 0x1000, 0x1000})
    internal/poll/fd_unix.go:167 +0x1dc
os.(*File).read(...)
    os/file_posix.go:32
os.(*File).Read(0x140000a8a70, {0x14000211000, 0x1000, 0x1000})
    os/file.go:119 +0x74
bufio.(*Reader).Read(0x1400005df50, {0x14000212400, 0x400, 0x400})
    bufio/bufio.go:227 +0x20c
github.com/hashicorp/go-plugin.copyChan({0x105667b50, 0x14000126c80}, 0x1400009cc00, {0x105642380, 0x140000a8a70})
    github.com/hashicorp/go-plugin@v1.4.1/grpc_stdio.go:181 +0x144
created by github.com/hashicorp/go-plugin.newGRPCStdioServer

Error: The terraform-provider-rollbar_v1.10.0 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

2023-02-01T13:36:55.652+0200 [DEBUG] provider: plugin exited
ghost commented 1 year ago

Hey @JJaakson, Thanks for reporting this issue, and thank you for sharing the detailed logs too. We're investigating this issue now. I'll get back to you whenever we have an update.

ghost commented 1 year ago

Hey @JJaakson, We found out what causes the race condition. The fix is expected to be done this week.

bwmetcalf commented 1 year ago

When can we expect a new release? Thanks.

github-actions[bot] commented 1 year ago

:tada: This issue has been resolved in version 1.11.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: