search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.84k stars 9.19k forks source link

r/aws_kinesis_firehose_delivery_stream: Sweeper hangs, blocking nightly CI #26192

Closed ewbankkit closed 2 years ago

ewbankkit commented 2 years ago

Community Note

From some recent nightly CI runs:

[10:22:01] :     [Step 4/4] 2022/08/09 10:22:01 [DEBUG] Sweeper (aws_cloudwatch_log_group) has dependency (aws_glue_job), running..
[10:22:01] :     [Step 4/4] 2022/08/09 10:22:01 [DEBUG] Running Sweeper (aws_glue_job) in region (us-west-2)
[10:22:01] :     [Step 4/4] 2022/08/09 10:22:01 [INFO] No Glue Jobs to sweep
[10:22:01] :     [Step 4/4] 2022/08/09 10:22:01 [DEBUG] Completed Sweeper (aws_glue_job) in region (us-west-2) in 42.109435ms
[10:22:01] :     [Step 4/4] 2022/08/09 10:22:01 [DEBUG] Sweeper (aws_cloudwatch_log_group) has dependency (aws_kinesis_analytics_application), running..
[10:22:01] :     [Step 4/4] 2022/08/09 10:22:01 [DEBUG] Sweeper (aws_kinesis_analytics_application) already ran in region (us-west-2)
[10:22:01] :     [Step 4/4] 2022/08/09 10:22:01 [DEBUG] Sweeper (aws_cloudwatch_log_group) has dependency (aws_kinesis_firehose_delivery_stream), running..
[10:22:01] :     [Step 4/4] 2022/08/09 10:22:01 [DEBUG] Running Sweeper (aws_kinesis_firehose_delivery_stream) in region (us-west-2)
[11:52:40] :     [Step 4/4] About to upload artifacts to S3
[11:52:40] :     [Step 4/4] Artifact S3 upload complete
[11:52:40] :     [Step 4/4] Build canceled

and

[15:30:20] :     [Step 4/4] 2022/07/20 15:30:20 [DEBUG] Completed Sweeper (aws_qldb_ledger) in region (us-west-2) in 106.607057ms
[15:30:20] :     [Step 4/4] 2022/07/20 15:30:20 [DEBUG] Running Sweeper (aws_cloudfront_realtime_log_config) in region (us-west-2)
[15:30:20] :     [Step 4/4] 2022/07/20 15:30:20 [DEBUG] Completed Sweeper (aws_cloudfront_realtime_log_config) in region (us-west-2) in 305.843037ms
[15:30:20] :     [Step 4/4] 2022/07/20 15:30:20 [DEBUG] Running Sweeper (aws_schemas_registry) in region (us-west-2)
[15:30:21] :     [Step 4/4] 2022/07/20 15:30:21 [DEBUG] Completed Sweeper (aws_schemas_registry) in region (us-west-2) in 592.925963ms
[15:30:21] :     [Step 4/4] 2022/07/20 15:30:21 [DEBUG] Running Sweeper (aws_kinesis_firehose_delivery_stream) in region (us-west-2)
[17:16:24] :     [Step 4/4] request expired, resigning
[17:23:04] :     [Step 4/4] SIGQUIT: quit
[17:23:12] :     [Step 4/4] PC=0x46c520 m=0 sigcode=0
[17:23:20] :     [Step 4/4] 
[17:23:21] :     [Step 4/4] goroutine 0 [idle]:
[17:23:25] :     [Step 4/4] runtime.epollwait()
[17:23:30] :     [Step 4/4]     /opt/goenv/versions/1.17.12/src/runtime/sys_linux_amd64.s:666 +0x20
[17:23:33] :     [Step 4/4] runtime.netpoll(0xc000066000)
[17:23:35] :     [Step 4/4]     /opt/goenv/versions/1.17.12/src/runtime/netpoll_epoll.go:127 +0xdc
[17:23:36] :     [Step 4/4] runtime.findrunnable()
[17:23:36] :     [Step 4/4]     /opt/goenv/versions/1.17.12/src/runtime/proc.go:2947 +0x593
[17:23:36] :     [Step 4/4] runtime.schedule()
[17:23:36] :     [Step 4/4]     /opt/goenv/versions/1.17.12/src/runtime/proc.go:3367 +0x239
[17:23:36] :     [Step 4/4] runtime.park_m(0xc0000016c0)
[17:23:36] :     [Step 4/4]     /opt/goenv/versions/1.17.12/src/runtime/proc.go:3516 +0x14d
[17:23:36] :     [Step 4/4] runtime.mcall()
[17:23:36] :     [Step 4/4]     /opt/goenv/versions/1.17.12/src/runtime/asm_amd64.s:307 +0x43
[17:23:36] :     [Step 4/4] 
[17:23:36] :     [Step 4/4] goroutine 1 [runnable]:
[17:23:36] :     [Step 4/4] github.com/aws/aws-sdk-go/aws/signer/v4.(*signingCtx).buildCanonicalHeaders(0xc00116d3f0, {0x9d9c280, 0xc00072f1e8}, 0xc352d66fc0)
[17:23:36] :     [Step 4/4]     /var/lib/teamcity/go/pkg/mod/github.com/aws/aws-sdk-go@v1.44.57/aws/signer/v4/v4.go:626 +0x285
[17:23:36] :     [Step 4/4] github.com/aws/aws-sdk-go/aws/signer/v4.(*signingCtx).build(0xc00116d3f0, 0x0)
[17:23:36] :     [Step 4/4]     /var/lib/teamcity/go/pkg/mod/github.com/aws/aws-sdk-go@v1.44.57/aws/signer/v4/v4.go:533 +0x186
[17:23:36] :     [Step 4/4] github.com/aws/aws-sdk-go/aws/signer/v4.Signer.signWithBody({0xc00051c280, 0x0, {0x9d7f8c0, 0xc0000c8028}, 0x0, 0x0, 0x1, 0x8fa89b0, 0x0}, 0xc351802200, ...)
[17:23:36] :     [Step 4/4]     /var/lib/teamcity/go/pkg/mod/github.com/aws/aws-sdk-go@v1.44.57/aws/signer/v4/v4.go:351 +0x39a
[17:23:36] :     [Step 4/4] github.com/aws/aws-sdk-go/aws/signer/v4.SignSDKRequestWithCurrentTime(0xc352f04f00, 0x8fa89b0, {0x0, 0x0, 0xf8e60b0})
[17:23:36] :     [Step 4/4]     /var/lib/teamcity/go/pkg/mod/github.com/aws/aws-sdk-go@v1.44.57/aws/signer/v4/v4.go:482 +0x325
[17:23:36] :     [Step 4/4] github.com/aws/aws-sdk-go/aws/signer/v4.SignSDKRequest(0xc352f04f00)
[17:23:36] :     [Step 4/4]     /var/lib/teamcity/go/pkg/mod/github.com/aws/aws-sdk-go@v1.44.57/aws/signer/v4/v4.go:429 +0x27
[17:23:36] :     [Step 4/4] github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc352f05108, 0xc352f04f00)
[17:23:36] :     [Step 4/4]     /var/lib/teamcity/go/pkg/mod/github.com/aws/aws-sdk-go@v1.44.57/aws/request/handlers.go:267 +0x9a
[17:23:36] :     [Step 4/4] github.com/aws/aws-sdk-go/aws/request.(*Request).Sign(0xc352f04f00)
[17:23:36] :     [Step 4/4]     /var/lib/teamcity/go/pkg/mod/github.com/aws/aws-sdk-go@v1.44.57/aws/request/request.go:445 +0x97
[17:23:37] :     [Step 4/4] github.com/aws/aws-sdk-go/aws/corehandlers.glob..func2(*** Test killed: ran too long (4h1m0s).
[17:23:37] :     [Step 4/4] FAIL    github.com/hashicorp/terraform-provider-aws/internal/sweep  14504.133s
[17:23:37] :     [Step 4/4] FAIL
[17:23:37]W:     [Step 4/4] Process exited with code 1
[17:23:51]E:     [Step 4/4] Process exited with code 1 (Step: Pre-Sweeper (Command Line))
[17:23:37]E:     [Step 4/4] Step Pre-Sweeper (Command Line) failed
[17:23:37] : Completed the Golang tests processing.
522 tests processed.

Running the aws_kinesis_firehose_delivery_stream sweeper manually:

% make sweep SWEEPARGS=-sweep-run=aws_kinesis_firehose_delivery_stream SWEEP=us-west-2
# make sweep SWEEPARGS=-sweep-run=aws_example_thing
WARNING: This will destroy infrastructure. Use only in development accounts.
go test ./internal/sweep -v -tags=sweep -sweep=us-west-2 -sweep-run=aws_kinesis_firehose_delivery_stream -timeout 60m
2022/08/09 12:14:06 [DEBUG] Running Sweepers for region (us-west-2):
2022/08/09 12:14:06 [DEBUG] Running Sweeper (aws_kinesis_firehose_delivery_stream) in region (us-west-2)
2022/08/09 12:14:06 [INFO] Retrieved credentials from "EnvConfigCredentials"
2022/08/09 12:14:06 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2022/08/09 12:14:06 [DEBUG] Trying to get account information via sts:GetCallerIdentity
^CFAIL  github.com/hashicorp/terraform-provider-aws/internal/sweep  106.473s
github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.