azurerm_pim_eligible_role_assignment attempts to recreate after circa 45 days - error "already exists"

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.8.2

AzureRM Provider Version

3.99.0

Affected Resource(s)/Data Source(s)

azurerm_pim_eligible_role_assignment

Terraform Configuration Files

resource "time_static" "now" {}

resource "azurerm_pim_eligible_role_assignment" "contributor" {
  scope              = data.azurerm_subscription.current.id
  role_definition_id = "${data.azurerm_subscription.current.id}${data.azurerm_role_definition.contributor.id}"
  principal_id       = var.contributor_pim_group

  schedule {
    start_date_time = time_static.now.rfc3339
    expiration {
      duration_hours = 0
    }
  }
}

Debug Output/Panic Output

Error: A resource with the ID "/subscriptions/xxxxxxx|/subscriptions/xxxxxxx/providers/Microsoft.Authorization/roleDefinitions//b24988ac-6180-42a0-ab88-20f7382dd24c|yyyyyyy" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_pim_eligible_role_assignment" for more information.
│ 
│   with azurerm_pim_eligible_role_assignment.contributor,
│   on main.tf line 549, in resource "azurerm_pim_eligible_role_assignment" "contributor":
│  549: resource "azurerm_pim_eligible_role_assignment" "contributor" {
│ 
│ A resource with the ID
│ "/subscriptions/xxxxxxx|/subscriptions/xxxxxxx/providers/Microsoft.Authorization/roleDefinitions//b24988ac-6180-42a0-ab88-20f7382dd24c|yyyyyyy8"
│ already exists - to be managed via Terraform this resource needs to be
│ imported into the State. Please see the resource documentation for
│ "azurerm_pim_eligible_role_assignment" for more information.

Expected Behaviour

Terraform should not loose the existing PIM eligible role assignment from it state and should not try to recreate it despite the role assignment still exists in Azure. The state of Terraform must be always the as the state in Azure it self.

Actual Behaviour

After a period of time, circa 45 days Terraform removes the roles assignment form the TF state and therefore wants to recreate the role assignment. But the create fails then obviously because the role assignment is still there in Azure.

Steps to Reproduce

Update PIM settings to allow permanent role assignments.
Create a PIM Role assignment without expiry.
Wait circa 45 days, reapply terraform config.
Terraform remove the role assignment from its state and wants to re-create it. -> Error.

Important Factoids

No response

References

It was presumed it is fixed in: https://github.com/hashicorp/terraform-provider-azurerm/issues/24118 but it is actually not. The root cause seems to be still exist.

https://github.com/hashicorp/terraform-provider-azurerm/issues/23111 seems to have the same root cause. Which make importing a an existing azurerm_pim_eligible_role_assignment resource impossible which makes this issue even more annoying. Because normally such errors could be fixed with a terraform import but this is not possibile in this case because Terraform thinks the ressource does no exist in Azure (wich is not true and a bug in the azurerm provider).

I did quite some troubleshooting around this issue yesterday which leads me to the conclusion that the information which is saved in the terraform state is based on the wrong Azure resource. The relevant code was already reference from @TeamDman in https://github.com/hashicorp/terraform-provider-azurerm/issues/23111 :

https://github.com/hashicorp/terraform-provider-azurerm/blob/dcf32586e80a12a20b5f979a934dcf1dc60eecf5/internal/services/authorization/pim_eligible_role_assignment_resource.go#L284-L304

There are basically two Azure resources for PIM eligible role assignments:

roleEligibilityScheduleRequests: API / resource to send request which makes Azure to create or delete the corresponding roleEligibilitySchedules object / resource. But roleEligibilityScheduleRequests does not represent the actual PIM role assignment resource or any other long-lived Azure resource at all. It is more for logging/auditing purpose and is therefore also cleaned up by azure after some time (presumably 45 days).
roleEligibilitySchedules: This is the actual Azure resource of the PIM eligible role assignment which keeps existent and which is also shown in the Azure Portal as the PIM role assignment. This is the only resource which Terraform should reference in it its state. Because this it the actual representation of azurerm_pim_eligible_role_assignment in Azure.

If following the official documentation https://learn.microsoft.com/en-us/rest/api/authorization/privileged-role-eligibility-rest-sample it is also obvious that you can get existing assignments only through roleEligibilitySchedules roleEligibilityScheduleInstances -> List eligible assignments. Only creating and deleting assignment is then made trough roleEligibilityScheduleRequests -> Grant eligible assignment, Remove eligible assignment. The resulting form the roleEligibilityScheduleRequest is then creating or deleting a roleEligibilitySchedule.

The problem seems to be now that the azurerm_pim_eligible_role_assignment reference the roleEligibilityScheduleRequests resource in it state. Which is definitely wrong and leads to the problem which causes the re-creation and which makes importing existing assignments impossible.

In the Terraform trace log it is also quite obvious why terraform loses track of the existing role assignment. It is basically also because roleEligibilityScheduleRequests instead of roleEligibilitySchedules is reference: First the correct request is made which also works correctly and actually finds the role assignment:

[DEBUG] provider.terraform-provider-azurerm_v3.99.0_x5: AzureRM Response for https://management.azure.com/subscriptions/.../providers/Microsoft.Authorization/roleEligibilitySchedules?%24filter=%28principalId+eq+%27f0ba8720-80af-4631-aa38-11a17aaebc89%27%29&api-version=2020-10-01: 
HTTP/2.0 200 OK
Content-Length: 7748

{"value":[{"properties":...

Then a second GET request is made for roleEligibilityScheduleRequests. And that is where the trouble begins:

[DEBUG] provider.terraform-provider-azurerm_v3.99.0_x5: AzureRM Response for https://management.azure.com/subscriptions/.../providers/Microsoft.Authorization/roleEligibilityScheduleRequests/727bbac1-c3d8-e5a2-1565-c0798d7b24d7?api-version=2020-10-01: 
HTTP/2.0 404 Not Found
Content-Length: 0

So when the roleEligibilityScheduleRequests ressource get cleaned up on Azure side and is there fore not found anymore the resource is removed from the state:

[INFO]  provider.terraform-provider-azurerm_v3.99.0_x5: [DEBUG] Role Management Policy: (Principal Id "f0ba8721-99af-4631-aa38-56a17acebc89" / Scope "/subscriptions/..." / Role Definition Id "/subscriptions/.../providers/Microsoft.Authorization/roleDefinitions/ba92f5b4-2d11-453d-a403-e96b0029c9fe") was not found - removing from state:

But this should not happend as long as the actual roleEligibilitySchedules resource still exists.

So, in short the azurerm_pim_eligible_role_assignment must use only information from the roleEligibilitySchedules to create the object in the terraform state and to check if the resource exists during the state refreshes. The roleEligibilityScheduleRequests should never references anywhere in the Terraform state because this resource is not persistent and is therefore not save to use to check if the PIM eligible roles assignment is present or not. If that would be the case, I think, https://github.com/hashicorp/terraform-provider-azurerm/issues/23111 would also be fixed.

@manicminer Then this is probably also fixed with #25956 ?

@J0F3 Thanks! Yes this will be fixed with that PR 🙂

I believe this is actually a duplicate of #24118, which was closed prematurely so I understand why this issue was opened.

@manicminer thank you! Yes I (re)created this issue because https://github.com/hashicorp/terraform-provider-azurerm/issues/24118 was closed because the problem was supposedly solved. But it showed now that it was actually not.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-azurerm