Error reading operation responses from API when provisioning google_access_context_manager_service_perimeter_* resources

SarahFrench commented 1 month ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.8.0 on darwin_arm64

provider registry.terraform.io/hashicorp/google v6.5.0

Affected Resource(s)

google_access_context_manager_service_perimeter_ingress_policy
google_access_context_manager_service_perimeter_dry_run_ingress_policy
google_access_context_manager_service_perimeter_dry_run_egress_policy

Terraform Configuration

Here is (incomplete) configuration I was passed as part of the customer issue: access-context-manager-bug.zip

Debug Output

I don't have any as I cannot reproduce this issue due to lacking sufficient testing resources.

Expected Behavior

Configuration should apply without error, or with an error that a user can act on. The current error only makes sense to provider developers and it's unclear if it's a result of bad Terraform configuration/eventual consistency or a bug in the provider.

Actual Behavior

The resource is successfully created in GCP, but then the error below is encountered, and this stops the resource's data being saved in state:

│ Error: Error decoding response from operation, could not find nested object
│ 
│   with module.service-perimeter.module.service-perimeter.google_access_context_manager_service_perimeter_ingress_policy.ingress_policy["ingress_policy_prd_rule_00040"],
│   on .terraform/modules/service-perimeter/module/main.tf line 96, in resource "google_access_context_manager_service_perimeter_ingress_policy" "ingress_policy":
│   96: resource "google_access_context_manager_service_perimeter_ingress_policy" "ingress_policy" {

This error originates from the provider here: LINK TO CODE

That error is triggered when flattenNestedAccessContextManagerServicePerimeterIngressPolicy returns a nil value for opRes and there is no error returned, which is only possible from these places.

Steps to reproduce

I am unable to reproduce this bug because I don't have access to sufficient testing resources- i.e. an organization with associated licenses I can test with. Currently I hit the same issue as shown here https://github.com/hashicorp/terraform-provider-google/issues/7752.

The configuration I've provided under Terraform Configuration is all the information I was passed as part of the original customer issue.

Important Factoids

No response

References

No response

SarahFrench commented 1 month ago

I've filed this issue on behalf of a customer, and I am unable to reproduce the error due to lacking access to sufficient testing resources. This issue is the first step to let them reach out to GCP customer support, so I will not be answering questions about the bug here.

ggtisc commented 1 month ago

Hi @SarahFrench!

For system security reasons I can't download .zip files. Could you share the code in a git repository out of a zip file or through this site just pasting the code?

SarahFrench commented 1 month ago

That's fair, here's a copy of the config in a gist: https://gist.github.com/SarahFrench/aae1556850003bb9c7ece7479c0c04d7

ggtisc commented 1 month ago

Thank you @SarahFrench!

I'm checking out this issue, but there is still some missing data that I need to replicate this issue. Could you please provide me with an example of how you are introducing the data for the variables and for_each loop ? It doesn't need to be sensitive information, just examples that can be used to replicate this scenario.

SarahFrench commented 1 month ago

I'm afraid I don't have that information from the customer - I'll try and get more information to put here. The main reason I opened a GitHub issue was because it was a prerequisite for the customer to reach out to GCP support, so I could only post what information I had.

ggtisc commented 1 month ago

No worries, I'm going to do a few tests with as much data as I can replace and I'll let you know if I find any bugs. In case it cannot be replicated, I would need complete information please.

ggtisc commented 1 month ago

After many tries I ran into the following error with the license while trying to reproduce:

Error: Error creating AccessLevel: googleapi: Error 400: Org owning policy must have license
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.DebugInfo",
│     "detail": "[ORIGINAL ERROR] generic::failed_precondition: com.google.identity.cloud.attributecalculator.exceptions.PublicCanonicalCodeException: Org owning policy must have license Code: FAILED_PRECONDITION [google.rpc.error_details_ext] { message: \"Org owning policy must have license\" }"
│   }
│ ]
│ 
│   with google_access_context_manager_access_level.al_19767,
│   on main.tf line 29, in resource "google_access_context_manager_access_level" "al_19767":
│   29: resource "google_access_context_manager_access_level" "al_19767" {

Could you let me know if you need me to use a Google-owned org to get a working reproduction?

SarahFrench commented 1 month ago

Ah yeah, I've been navigating that problem too 😓

The organization that you use for testing needs a BeyondCorp Enterprise license attached. I've been talking to @roaks3 about this, and he's been looking into what resolved the same issue when it was first encountered by the Google Terraform team. tl;dr yes I believe you'd need to use a Google-owned org that's been allow listed to avoid needing a paid license, or arrange for a license to be added to your test org.

roaks3 commented 2 weeks ago

@SarahFrench checking back in on this. We have an environment that I can try to reproduce in if needed, but I don't know if we have enough information here to reproduce. I'm also wondering if the support ticket has had any movement.

I can confirm from our nightly test debug logs that the operation does include the expected fields in the expected structure (so this issue doesn't indicate a server-side change), so my guess is that there is something specific to this config that is causing the fields to be missing. Without knowing the values that were used (to some degree), it's hard to determine where to look. If the customer is able, a debug log would help quite a bit to look at API request/response pairs.

SarahFrench commented 2 weeks ago

Thanks @roaks3 - I've shared this with HashiCorp support to ask whether I should share customer debug logs with you privately and/or whether the user has supplied debug logs with their GCP support ticket. Once I hear more I'll update here.

Charlesleonius commented 1 week ago

Hi all, I've been making some changes to the way the flattening functions work for these resources in an effort to combat permadiffs due to server-side re-ordering of certain fields. Seems like this issue occurred before these changes. Is the customer still seeing this issue?

hashicorp / terraform-provider-google