search

rollbar / terraform-provider-rollbar

Terraform provider for Rollbar
https://rollbar.com
MIT License
28 stars 13 forks source link

Plugin crashing after upgrade to 1.9.0 #340

Closed bwmetcalf closed 1 year ago

bwmetcalf commented 1 year ago

This is likely related to the changes to address issue #330. After upgrading to 1.9.0, the provider occasionally crashes with the following:

│    1: data "rollbar_project" "default" {
│ 
│ The plugin.(*GRPCProvider).ReadDataSource request was cancelled.
╵
Stack trace from the terraform-provider-rollbar_v1.9.0 plugin:
fatal error: concurrent map writes
fatal error: concurrent map writes
goroutine 161 [running]:
runtime.throw({0xc060e4, 0x10})
    runtime/panic.go:1198 +0x71 fp=0xc0004bb6d8 sp=0xc0004bb6a8 pc=0x433871
runtime.mapassign_faststr(0xb9b9c0, 0xc0004904b0, {0xc0003558a0, 0x1e})
    runtime/map_faststr.go:294 +0x38b fp=0xc0004bb740 sp=0xc0004bb6d8 pc=0x412c8b
net/textproto.MIMEHeader.Set(...)
    net/textproto/header.go:22
net/http.Header.Set(...)
    net/http/header.go:38
github.com/go-resty/resty/v2.(*Client).SetHeader(...)
    github.com/go-resty/resty/v2@v2.5.0/client.go:173
github.com/rollbar/terraform-provider-rollbar/rollbar.setDataSourceHeader({0xc01c94, 0xf}, 0xbfbc54)
    github.com/rollbar/terraform-provider-rollbar/rollbar/common.go:46 +0xa5 fp=0xc0004bb790 sp=0xc0004bb740 pc=0xa7ebe5
github.com/rollbar/terraform-provider-rollbar/rollbar.dataSourceProjectRead(0x68, {0xb2d080, 0xc000490690})
    github.com/rollbar/terraform-provider-rollbar/rollbar/data_source_project.go:77 +0xc8 fp=0xc0004bb8a0 sp=0xc0004bb790 pc=0xa7ed28
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xbfa7e5, {0xd11158, 0xc0001f1700}, 0xc0004159b0, {0xb2d080, 0xc000490690})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:346 +0x178 fp=0xc0004bb918 sp=0xc0004bb8a0 pc=0x9f9fd8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ReadDataApply(0xc00021eee0, {0xd11158, 0xc0001f1700}, 0xc000458680, {0xb2d080, 0xc000490690})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:569 +0xf7 fp=0xc0004bb988 sp=0xc0004bb918 pc=0x9fb697
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadDataSource(0xc000204798, {0xd11158, 0xc0001f1700}, 0xc000183720)
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/grpc_provider.go:1133 +0x3a8 fp=0xc0004bba68 sp=0xc0004bb988 pc=0x9f4f68
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadDataSource(0xc000214d80, {0xd11200, 0xc000415260}, 0xc00041e550)
    github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/tf5server/server.go:478 +0x37b fp=0xc0004bbb70 sp=0xc0004bba68 pc=0x8d913b
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadDataSource_Handler({0xbcf440, 0xc000214d80}, {0xd11200, 0xc000415260}, 0xc0001eb9e0, 0x0)
    github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:416 +0x170 fp=0xc0004bbbc8 sp=0xc0004bbb70 pc=0x8ca150
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000230700, {0xd1e398, 0xc00010a480}, 0xc000165100, 0xc000353170, 0x11fdeb0, 0x0)
    google.golang.org/grpc@v1.36.0/server.go:1217 +0xc8f fp=0xc0004bbe48 sp=0xc0004bbbc8 pc=0x83400f
google.golang.org/grpc.(*Server).handleStream(0xc000230700, {0xd1e398, 0xc00010a480}, 0xc000165100, 0x0)
    google.golang.org/grpc@v1.36.0/server.go:1540 +0xa2a fp=0xc0004bbf68 sp=0xc0004bbe48 pc=0x837bea
google.golang.org/grpc.(*Server).serveStreams.func1.2()
    google.golang.org/grpc@v1.36.0/server.go:878 +0x98 fp=0xc0004bbfe0 sp=0xc0004bbf68 pc=0x831cb8
runtime.goexit()
    runtime/asm_amd64.s:1581 +0x1 fp=0xc0004bbfe8 sp=0xc0004bbfe0 pc=0x463621
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.36.0/server.go:876 +0x294
goroutine 1 [select]:
github.com/hashicorp/go-plugin.Serve(0xc000137a28)
    github.com/hashicorp/go-plugin@v1.4.1/server.go:469 +0x1493
github.com/hashicorp/terraform-plugin-sdk/v2/plugin.Serve(0xc000200360)
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/plugin/serve.go:147 +0x905
main.main()
    github.com/rollbar/terraform-provider-rollbar/main.go:54 +0x607
goroutine 17 [select]:
github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv(0xc29ff2)
    github.com/hashicorp/go-plugin@v1.4.1/grpc_broker.go:121 +0x6d
github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc00021d0e0)
    github.com/hashicorp/go-plugin@v1.4.1/grpc_broker.go:411 +0x48
created by github.com/hashicorp/go-plugin.(*GRPCServer).Init
    github.com/hashicorp/go-plugin@v1.4.1/grpc_server.go:85 +0x4a9
goroutine 18 [IO wait]:
internal/poll.runtime_pollWait(0x7fb50fc31ea8, 0x72)
    runtime/netpoll.go:303 +0x85
internal/poll.(*pollDesc).wait(0xc000200900, 0xc00011d000, 0x1)
    internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000200900, {0xc00011d000, 0x1000, 0x1000})
    internal/poll/fd_unix.go:167 +0x25a
os.(*File).read(...)
    os/file_posix.go:32
os.(*File).Read(0xc0002082d8, {0xc00011d000, 0x400, 0xafaae0})
    os/file.go:119 +0x5e
bufio.(*Reader).Read(0xc000062740, {0xc0000e0400, 0x400, 0x0})
    bufio/bufio.go:227 +0x1b4
github.com/hashicorp/go-plugin.copyChan({0xd27f10, 0xc000214980}, 0x0, {0xcffde0, 0xc0002082d8})
    github.com/hashicorp/go-plugin@v1.4.1/grpc_stdio.go:181 +0x1f6
created by github.com/hashicorp/go-plugin.newGRPCStdioServer
    github.com/hashicorp/go-plugin@v1.4.1/grpc_stdio.go:37 +0xfa
goroutine 19 [IO wait]:
internal/poll.runtime_pollWait(0x7fb50fc31cc8, 0x72)
    runtime/netpoll.go:303 +0x85
internal/poll.(*pollDesc).wait(0xc0002009c0, 0xc0001b4000, 0x1)
    internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0002009c0, {0xc0001b4000, 0x1000, 0x1000})
    internal/poll/fd_unix.go:167 +0x25a
os.(*File).read(...)
    os/file_posix.go:32
os.(*File).Read(0xc0002082e8, {0xc0001b4000, 0x400, 0xafaae0})
    os/file.go:119 +0x5e
bufio.(*Reader).Read(0xc000062f40, {0xc0000e0800, 0x400, 0x0})
    bufio/bufio.go:227 +0x1b4
github.com/hashicorp/go-plugin.copyChan({0xd27f10, 0xc000214980}, 0x0, {0xcffde0, 0xc0002082e8})
    github.com/hashicorp/go-plugin@v1.4.1/grpc_stdio.go:181 +0x1f6
created by github.com/hashicorp/go-plugin.newGRPCStdioServer
    github.com/hashicorp/go-plugin@v1.4.1/grpc_stdio.go:38 +0x188
goroutine 21 [syscall]:
os/signal.signal_recv()
    runtime/sigqueue.go:169 +0x98
os/signal.loop()
    os/signal/signal_unix.go:24 +0x19
created by os/signal.Notify.func1.1
Error: The terraform-provider-rollbar_v1.9.0 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

We suspect this is due to the retry implemented for #330 hitting a limit resulting in the crash. The retry logic implemented for the 502s we were encountering seems to be working for the most part, but we sometimes hit the above.

Please let us know if more information is needed.

ghost commented 1 year ago

@bwmetcalf Thanks for reporting this issue. We'll investigate, and I'll get back to you.

bwmetcalf commented 1 year ago

Any update on this? This morning has been particularly bad for hitting 502s against the rollbar API.

ghost commented 1 year ago

We are working on the fix, but it's a complex problem. Unfortunately, the issue is in the provider's thread handling; thus, we'll need to be super cautious with the fix. I'll keep you updated.

ghost commented 1 year ago

@bwmetcalf We plan to release a new version with this fix by the end of next week.

bwmetcalf commented 1 year ago

@bwmetcalf We plan to release a new version with this fix by the end of next week.

Great! Thank you!