Error: 429 error without the "X-Redlock-Status" header - returned HTML:

[billdoering]

Describe the bug

I receive the above error inconsistently during Planning Runs for prismacloud_integration.security_hub Resources. Sometimes I get 8 errors referencing an AWS Account ID, sometimes I get 7, 10 or none ! If I use the GUI to "Refresh State" sometimes it completes successfully and then right after another Refresh State will fail with errors.

Expected behavior

I expect to be able to continue to add AWS Account IDs to the variable file so that new prismacloud_integration.security_hub Resources are created for each added Account. This has worked for the past 3 months. Now I get the "Error 429" (see below).

I have confirmed via a TF client that the number of prismacloud_integration.security_hub Resources matches the TFC State file. I see no discrepancies or potential drift. Using "Refresh State" in the GUI will occasionally successfully COMPLETE a Plan. But then if I was to Refresh State again it may Fail. Any new Plan fails.

Current behavior

After a change in the Repo (adding an AWS Account to the variables file) or Triggering a New Run from the GUI to Refresh State I get the following errors intermittently that reference AWS Account IDs. These Account IDs do not seem to be directly related to the issue as some of these Accounts have been working since Dec. 2022. The order and number of the Accounts identified in the "Error: 429" lines change with each Run. Some times there are 6 Account IDs referenced, sometimes 10 or some number inbetween.

Error: 429 error without the "X-Redlock-Status" header - returned HTML: with prismacloud_integration.security_hub["777800573549"] on main.tf line 1, in resource "prismacloud_integration" "security_hub": resource "prismacloud_integration" "security_hub" {

Error: 429 error without the "X-Redlock-Status" header - returned HTML: with prismacloud_integration.security_hub["332465446158"] on main.tf line 1, in resource "prismacloud_integration" "security_hub": resource "prismacloud_integration" "security_hub" {

...

Possible solution

I am not sure if this is a BUG in the Provider or some new functionality that may have somehow impacted my state. I had been using the following versioning when you were still on the 1.2.x Provider code

  #version = ">= 1.2.0"

After things stopped working and I realized you had 1.3.0 out I changed that to

  version = "= 1.2.11"

BUT, maybe something from 1.3.x already did something unexpectedly ?

Steps to reproduce

The simplest way to reproduce is to do a Refresh State from the GUI. I am willing to post all the code if that is the next step. Adding one account string to the variable seemed to start the problem which as I have mentioned I have been doing over the last 3 months

variable "aws_accounts_to_integrate" { type = map(string) default = { "804665673296": "ucsb-ccid - 804665673296" "195006774330": "Audit - 195006774330" "143771954461": "ucsb-identity - 143771954461" "585393162772": "Log archive - 585393162772"

"794820895137": "ucsb-network - 794820895137"

---

resource "prismacloud_integration" "security_hub" {

for_each = var.aws_accounts_to_integrate

name = each.value

integration_type = "aws_security_hub"
description = "Made by Terraform to create security hub integration in Prisma Cloud/Settings/Integrations"
enabled = true
integration_config {
    account_id = each.key
    regions {
        api_identifier = "us-west-2"
        name = "AWS Oregon"
        cloud_type = "aws"
        }
    }

}

Screenshots

Context

I am slowly adding our 51 AWS Accounts in order to "onboard" them such that they will all have the Security Hub Integration added to their Account as well as the Notification setup so that the Accounts get Prisma Cloud Alerts as Security Hub Findings.

Your Environment

TFC 1.3.6 Workspace: ucsb-prismacloud Set Provider to version = "= 1.2.11" BUT this was after the problem occurred and as alluded to above I may have 1.3.0 Provide used for some Runs before I noticed a new major version was out ....

[welcome-to-palo-alto-networks[bot]]

:tada: Thanks for opening your first issue here! Welcome to the community!

[billdoering]

I have cleaned up my post about the error and hopefully made things cleaner, easier to read.

Is there anything else I can do to get someone to help me evaluate if this is an issue with the Provider ???

Thanks for your help

[nikitapanw]

Helloe @billdoering , we have released new version of terraform provider, could you please try using the new version and update us regarding the response. We have provided fix for that. Thanks

[ftbrecordspan]

Hi @billdoering, his should be resolved in v1.3.4. Please re-open as necessary.

[billdoering]

ftbrecordspan,

Looks like I still experience the same issue, same error. I changed the Provider version to =1.3.4 and I got the same error as previously documented. I ran a Refresh State from TCF GUI and that errored with the same error as well.

I checked to see what the latest revision was and noticed it was now 1.3.7. After reading the changelog it didn't seem like there were any relevant changes after the integration rate limit issue, BUT I decided to try it anyway.

So, I changed the Provider to =1.3.7 and it Planned and Finished. yeah, or so I thought.

So, I went to ADD an Integration to an Account and it errored out again. Then I commented out that Account and it Planned and Finished again.

I ran the Refresh State from the TFC GUI and it errored out again. Same error as always.

I do not know what to do next. Any input would be appreciated. Thanks Bill

[billdoering]

@ftbrecordspan - I can not find a way to re-open a Ticket that has been closed by a Collaborator.

[ftbrecordspan]

@billdoering v1.3.4 included optimization of the integration resource in an effort to avoid these errors stemming from rate limiting by the upstream API. I'm sorry to hear that these changes didn't alleviate the issue for you.

I've re-opened this issue and other solutions will be evaluated.

[billdoering]

@ftbrecordspan as a workaround I was thinking I could add a delay to my for_each loop that might avoid the rate limiting?

I thought that would be simple but it doesn't appear so. Any suggestions on a way to do this ?

thanks Bill

[UshaPriya2001]

Hello @billdoering , Due to API rate limit constraint, you can not onboard 20 or more prismacloud_integration resources at once. Try onboarding 20 or less resources in one go. We will provide support to fix API rate limit with terraform.

[AnushreeHS]

@billdoering Thanks for opening the issue, this is released with terraform version = "1.3.9" Closing the issue Thanks

[billdoering]

Thank you !!!! This worked great for me now. Appreciate you quick work