importing azurerm_pim_eligible_role_assignment fails with resource not existant

[tim-krehan]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

Terraform fails while importing the azurerm_pim_eligible_role_assignment. The assignment is still present in Azure but not in the state. To manage it, I tried importing it but it cannot find the assignment in Azure. The import fails with Cannot import non-existent remote object The apply fails with A resource with the ID "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test-001|/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Authorization/roleDefinitions/00482a5a-887f-4fb3-b363-3b7fe8e74483|11111111-1111-1111-1111-111111111111" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_pim_eligible_role_assignment" for more information.

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.5.6

AzureRM Provider Version

3.71.0

Affected Resource(s)/Data Source(s)

azurerm_pim_eligible_role_assignment

Terraform Configuration Files

resource "azurerm_pim_active_role_assignment" "pim_active" {
  scope              = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test-001"
  role_definition_id = "/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Authorization/roleDefinitions/00482a5a-887f-4fb3-b363-3b7fe8e74483"
  principal_id       = "11111111-1111-1111-1111-111111111111"

  justification = "Sample Justification"

  schedule {
    start_date_time = "2023-07-13T11:40:26.0000000+02:00"
    expiration {
      duration_days = 180
    }
  }
}

import {
  to = azurerm_pim_active_role_assignment.pim_active
  id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-test-001|/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Authorization/roleDefinitions/00482a5a-887f-4fb3-b363-3b7fe8e74483|11111111-1111-1111-1111-111111111111"
}

Debug Output/Panic Output

╷
│ Error: Cannot import non-existent remote object
│
│ While attempting to import an existing object to "azurerm_pim_active_role_assignment.pim_active",
│ the provider detected that no object exists with the given id. Only pre-existing objects can be imported; check that the id is correct and that it is associated with the
│ provider's configured region or endpoint, or use "terraform apply" to create a new remote object for this resource.
╵

Expected Behaviour

The allready present assignment will get imported

Actual Behaviour

The Import throws the error, that the object in Azure is not existant, while it definitly is. Even the apply fails to create it, stating that the object is allready present

Steps to Reproduce

1. terraform apply

Important Factoids

No response

References

No response

[xuzhang3]

@tim-krehan Looks you want to import the azurerm_pim_eligible_role_assignment but the configuration is azurerm_pim_active_role_assignment. Can retry with the same resource type?

[tim-krehan]

hmmm i had that issue with both active and eligible. The thing is: i can't reproduce it right now 😢

Seems that it was a one-time thing?

I think this issue can be closed - I will reference it, once I come upon that issue again.

[audunsolemdal]

I am using the correct resource type and having the same issue. The azurerm_pim_eligible_role_assignment has been buggy from the start in my case, a lot of times seemingly out of nowhere stating that it needs to be created, even though it has already been created via terraform

I know imports have worked previously for me, not sure where the issue comes from.

[xuzhang3]

@audunsolemdal There appears to be a delay in the response of this resource.. The resource cannot deleted within the first 5 mins etc. Can you retry later?

[unique-dominik]

I believe this issue should be reopened. I can reproduce it right now…

terraform apply -target='azurerm_pim_eligible_role_assignment.this["user1"]'
--> yes
Error: A resource with the ID "/subscriptions/<s>|/subscriptions/<s>/providers/Microsoft.Authorization/roleDefinitions/324395f4-284c-2fba-f19b-4ee363d92e04|<user1>" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_pim_eligible_role_assignment" for more information.

Then:

terraform import 'azurerm_pim_eligible_role_assignment.this["user1"]' "/subscriptions/<s>|/subscriptions/<s>/providers/Microsoft.Authorization/roleDefinitions/324395f4-284c-2fba-f19b-4ee363d92e04|<user>"

-->

Prepared azurerm_pim_eligible_role_assignment for import

--> 

Error: Cannot import non-existent remote object
│
│ While attempting to import an existing object to "azurerm_pim_eligible_role_assignment.this[\"user1"]", the provider detected that no object exists with the given id. Only pre-existing objects can be imported; check that the id is correct and that it is
│ associated with the provider's configured region or endpoint, or use "terraform apply" to create a new remote object for this resource.

I can play this chess game back and forward ♟️

In the UI, the assignment exists. To me it currently occurs that it is just orphaned completely.

Terraform v1.6.0
on darwin_arm64
+ provider registry.terraform.io/hashicorp/azuread v2.43.0
+ provider registry.terraform.io/hashicorp/azurerm v3.75.0
+ provider registry.terraform.io/hashicorp/time v0.9.1

[audunsolemdal]

I agree that this is still a problem

[braun-daniel]

I could reproduce this issue as well.

Terraform v1.6.2
on darwin_arm64
+ provider registry.terraform.io/hashicorp/azurerm v3.77.0

[MohnJadden]

I'm having the same issue when using an index key, opened #23657 in case it's something specific with index key imports. One thing I also just noticed is that when destroying azurerm_pim_eligible_role_assignment resources, the assignment is never actually removed. If you're re-trying the creation after destruction, the "resource already exists" situation will recur and thus lead you to the import issue once more.

The only sure fix has been clicking through the portal and removing the assignment. I just opened issue #23672 in hopes of getting it fixed.

The azurerm_pim_eligible_role_assignment resource provider has been buggy since it launched, unfortunately this is just more of the same.

[unique-dominik]

It does not really help to resolve anyones issues but I'd like to also support the maintainers in sharing two discoveries I made over the recent weeks.

Scheduling changes

The whole eligibility setup thing does not base on pure CRUD like objects it seems. It bases on "requests" (orders) you make and the system "processing" those. The resulting resources are read-only. It can be seen in the REST Docs that you always make a request of a kind and then you wait for a result that "pops up somewhere" else.

E.g. revoking an eligibility needs a "request" of type AdminRemove and then, after some time, the change will be persisted. I had to do that like two weeks ago to clean our terraform state as back then (see next point) the Management Group scoped PIM permissions had no UI!

An example for history:

curl --location --request PUT 'https://management.azure.com/providers/Microsoft.Management/managementGroups/uqe/providers/Microsoft.Authorization/roleEligibilityScheduleRequests/<random-uuid-from-an-uuid-tool>?api-version=2020-10-01' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <token>' \
--data '{
  "Properties": {
    "RoleDefinitionId": "/providers/Microsoft.Authorization/roleDefinitions/<role>",
    "PrincipalId": "<principal>",
    "RequestType": "AdminRemove"
    }
}'

Afterwards, you can plan and apply again as the resulting "eligibility assignment" will be revoked.

This brings me to point 2.

New Azure UI

Since some days, you can do what I did via maddening REST calls via the UI! 🔔

It does not help much for the terraform issues but at least it's easier to clean the state 🤣

[Scarlettliuyc]

hi @tim-krehan good day. I'm not sure whether you are the one from hashcrop, we still have this kind of issue. When use "terraform plan" it shows the PIM doesn't exist. And use the import existing resource, there the errors: Error: Cannot import non-existent remote object │ │ While attempting to import an existing object to │ "azurerm_pim_eligible_role_assignment.pim-495f1d9d-ab33-2196-479b-8aff97b3df0b", │ the provider detected that no object exists with the given id. Only │ pre-existing objects can be imported; check that the id is correct and that │ it is associated with the provider's configured region or endpoint, or use │ "terraform apply" to create a new remote object for this resource.

however the object add definition is correct. We want to confirm any progress for this issue

[tim-krehan]

@Scarlettliuyc Sorry to disappoint you, I am just an user that is affected by this.

[smokedlinq]

@xuzhang3 FYI I just tried this on 3.85.0 release and it still fails. If I try to do the apply, it says it already exists. I add the import and it says it doesn't exist.

[spectrum048k]

@xuzhang3 I have also test version 3.85 and have observed same behaviour as @smokedlinq - apply still fails saying resources exists and import fails saying it doesn't. Can this issue please be re-opened?

[unique-dominik]

I was so sick of this that I "developed" this in the meantime:

import { AuthorizationManagementClient } from "@azure/arm-authorization";
import { DefaultAzureCredential } from "@azure/identity";

import { v4 as uuidv4 } from 'uuid';

const tfOutputSanitized = `
/subscriptions/xxxxxxx-b530-xxxx-9b08-xxxxxxxxx|/subscriptions/xxxxxx-xxxx-4277-9b08-xxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/xxxxxx-89b4-xxxx-9698-xxxxxxxxx|xxxxxx-f49b-xxxx-bd59-xxxxxxxx
`;

/**
 * This script is the automation of docs/docs/enterprise/single-tenant/troubleshooting.md#terraform-state-forgot-resources
 * I now had to fix it the third time, so I decided to automate it.
 * It would really be nice if either Microsoft or Hashicorp would fix this issue and until then we do this rain dance every 4-6 weeks.
 * npm run repair:subscriptions
 * 
 * The terraform apply output that is erroneous must be a bit revised to be used here, but it can nearly be pasted 1:1 and then the script will fix it.
 */

async function putRoleEligibilityScheduleRequests() {

  const subscriptionsToFixToday: {subscription: string, roleDefinitionId: string, principalId: string}[] = [];

  const lines = tfOutputSanitized.split("\n");
  for (const line of lines) {
    const parts = line.split("|");
    if (parts.length === 3) {
      const subscription = parts[0];
      const roleDefinitionId = parts[1];
      const principalId = parts[2];
      subscriptionsToFixToday.push({subscription, roleDefinitionId, principalId});
    }
  }

  console.log(`Parsed ${subscriptionsToFixToday.length} incidents to fix.`);

  console.log(`Revoking eligible role assignments for ${subscriptionsToFixToday.length} subscriptions.`)
  // Setup env and get a token for ARM
  const subscriptionId = process.env["AUTHORIZATION_SUBSCRIPTION_ID"] || "00000000-0000-0000-0000-000000000000";
  const credential = new DefaultAzureCredential();
  const client = new AuthorizationManagementClient(credential, subscriptionId);

  for (const assignment of subscriptionsToFixToday) {
    const roleEligibilityScheduleRequestName = uuidv4();
    const scope =`providers/Microsoft.Subscription${assignment.subscription}`;
    const parameters = {
      principalId: assignment.principalId,
      requestType: "AdminRemove",
      roleDefinitionId: assignment.roleDefinitionId,
    };
    const result = await client.roleEligibilityScheduleRequests.create(
      scope,
      roleEligibilityScheduleRequestName,
      parameters
    );
    console.log(`Assignment [${result.status?.toUpperCase()}] for scope ${result.expandedProperties?.scope?.displayName} for role ${result.expandedProperties?.roleDefinition?.displayName} for principal ${result.expandedProperties?.principal?.displayName}.`);
  }
}

putRoleEligibilityScheduleRequests();

{
  "name": "repair-kit",
  ...
  "scripts": {
    "repair:subscriptions": "npx ts-node src/subscriptions.ts"
  },
  ...
  "devDependencies": {
    "@azure/arm-authorization": "^9.0.0",
    "@azure/identity": "^4.0.0",
    "@types/node": "^20.10.4",
    "@types/uuid": "^9.0.7",
    "uuid": "^9.0.1"
  }
}

Where tfOutputSanitized is basically the tf output complaining and using multiple-time multi-cursor-edit to strip it just to the resource id.

Sorry but this is the best I got until now to get out of the misery.

[xuzhang3]

@smokedlinq can you share your TF script and the TRACE logs? I do not have the same env as you do so I can only generate a potentials scenario in my mind. One way you can do is check the role assignment in the PIM role assignment in the PIM role assignment page. The details I want to know is which scope you want to assign the role and the sub/upper resources role assign details. This role should be at least assigned to the different resource in different scope.

[unique-dominik]

@smokedlinq can you share your TF script and the TRACE logs? I do not have the same env as you do so I can only generate a potentials scenario in my mind. One way you can do is check the role assignment in the PIM role assignment in the PIM role assignment page. The details I want to know is which scope you want to assign the role and the sub/upper resources role assign details. This role should be at least assigned to the different resource in different scope.

He @xuzhang3, they are not. They are 100% the same as the last time we/I ran apply. I can reproduce what @smokedlinq reports like every other month. That is why I wrote this script. Let me know what I shall provide to debug next time but the scopes and the role_id's are 100% the same, we rerun the same tf code that applied 4 weeks ago without change and the error appears.

The 🔑 is to wait until something @ARM/Entra changes, then the tf state breaks until fixes as above. Check also my message https://github.com/hashicorp/terraform-provider-azurerm/issues/23111#issuecomment-1757304348 where I basically did what you wanted to see 💟

[xuzhang3]

@unique-dominik @spectrum048k I cannot reproduce this error in v3.85 Can you try the script for testing? The script should work fine in v3.85 but fails in v3.84

#terraform {
#  required_providers {
#    azurerm = {
#      source  = "hashicorp/azurerm"
#      version = "=3.85.0"
#    }
#  }
#}
provider "azurerm" {
  features {
  }
}

data "azurerm_client_config" "example" {}

resource "time_static" "example" {}

resource "azurerm_pim_eligible_role_assignment" "pim1" {
  scope              = "/subscriptions/<Sub ID>/resourceGroups/<Resource Group Name>"
  role_definition_id = "<Role definition ID>"
  principal_id       = data.azurerm_client_config.example.object_id

  justification = "test"

  schedule {
    start_date_time = time_static.example.rfc3339
    expiration {
      duration_days = 180
    }
  }
}

resource "azurerm_pim_eligible_role_assignment" "pim2" {
  scope              = "/subscriptions/Sub ID"
  role_definition_id = "<Role definition ID>"
  principal_id       = data.azurerm_client_config.example.object_id

  justification = "test"

  schedule {
    start_date_time = time_static.example.rfc3339
    expiration {
      duration_days = 180
    }
  }
}

[unique-dominik]

Will try again next time yes. I can also not repro it right now, it needs some days time to break from the other end 😢 I will post back once it happens again, what detail would you need then 👁️

[MohnJadden]

I too am still getting the OP error in 3.85. If the PIM eligibility exists in Azure, Terraform says it has to be imported. If I try to import it, I get the import error as well.

If the eligibility doesn't exist in Azure, it creates without incident. If the eligibility is created by Terraform using the test that @xuzhang3 provided, it updates it, but it still doesn't provide for indefinite/permanent eligibility as described in #23366 and #22766 . It looks like the PR for #23295 has not been merged yet, which might help.

[spectrum048k]

@xuzhang3 In my testing I have noticed it takes circa 45 days for the pim assignment to 'fail' in the Azure backend before the issue presents itself in terraform.

[unique-dominik]

@spectrum048k Sounds about right, I guessed 4-6 weeks above 👍

[unique-dominik]

Today I have it again for 17 azurerm_pim_eligible_role_assignment, is there anything else I can check @xuzhang3?

I use rocket and will replace always the first and last UUID tokens with X to anonymise one of our groups.

Plan

Terraform says, it wants to create the assignment:

# module.rocket.azurerm_pim_eligible_role_assignment.manager will be created
  + resource "azurerm_pim_eligible_role_assignment" "manager" {
      + id                 = (known after apply)
      + justification      = "Eligible Managers to the [rocket] tenant via the Management Group."
      + principal_id       = "XXXXXXXX-9a72-4803-a7a5-XXXXXXXXXXXX"
      + principal_type     = (known after apply)
      + role_definition_id = "/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX"
      + scope              = "/providers/Microsoft.Management/managementGroups/lz-rocket"

      + schedule {
          + start_date_time = "2023-11-07T12:08:49Z"

          + expiration {
              + duration_days  = 180
              + duration_hours = (known after apply)
              + end_date_time  = (known after apply)
            }
        }

      + ticket {
          + number = "XXXXXXXX"
          + system = "JIRA"
        }
    }

Apply

The apply though fails:

╷
│ Error: A resource with the ID "/providers/Microsoft.Management/managementGroups/lz-rocket|/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX|XXXXXXXX-9a72-4803-a7a5-XXXXXXXXXXXX" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_pim_eligible_role_assignment" for more information.
│
│   with module.rocket.azurerm_pim_eligible_role_assignment.manager,
│   on ../../modules/management-group/mgmt-grp_managers.tf line 43, in resource "azurerm_pim_eligible_role_assignment" "manager":
│   43: resource "azurerm_pim_eligible_role_assignment" "manager" {
│
│ A resource with the ID
│ "/providers/Microsoft.Management/managementGroups/lz-rocket|/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX|XXXXXXXX-9a72-4803-a7a5-XXXXXXXXXXXX"
│ already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for
│ "azurerm_pim_eligible_role_assignment" for more information.
╵

ARM

I tried now to grab some infos from ARM to maybe help:

For the management group in question, I got all roleEligibilitySchedules:

https://management.azure.com/providers/Microsoft.Management/managementGroups/lz-rocket/providers/Microsoft.Authorization/roleEligibilitySchedules?api-version=2020-10-01

{
  "value": [
    {
      "properties": {
        "roleEligibilityScheduleRequestId": "/providers/Microsoft.Management/managementGroups/lz-rocket/providers/Microsoft.Authorization/roleEligibilityScheduleRequests/XXXXXXXX-856b-e9c6-ed6e-XXXXXXXXXXXX",
        "scope": "/providers/Microsoft.Management/managementGroups/lz-rocket",
        "roleDefinitionId": "/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX",
        "principalId": "4a7e9459-9a72-4803-a7a5-26a6aaa76c1a",
        "principalType": "Group",
        "status": "Provisioned",
        "startDateTime": "2023-11-07T12:09:29.17Z",
        "endDateTime": "2024-05-05T12:09:29.13Z",
        "memberType": "Direct",
        "createdOn": "2023-11-07T12:09:29.17Z",
        "updatedOn": "2023-11-07T12:09:29.17Z",
        "expandedProperties": {
          "principal": {
            "id": "4a7e9459-9a72-4803-a7a5-26a6aaa76c1a",
            "displayName": "[Rocket] Managers",
            "type": "Group"
          },
          "roleDefinition": {
            "id": "/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX",
            "displayName": "[Rocket] Manager",
            "type": "CustomRole"
          },
          "scope": {
            "id": "/providers/Microsoft.Management/managementGroups/lz-rocket",
            "displayName": "Rocket",
            "type": "managementgroup"
          }
        }
      },
      "name": "XXXXXXXX-856b-e9c6-ed6e-XXXXXXXXXXXX",
      "id": "/providers/Microsoft.Management/managementGroups/lz-rocket/providers/Microsoft.Authorization/roleEligibilitySchedules/XXXXXXXX-856b-e9c6-ed6e-XXXXXXXXXXXX",
      "type": "Microsoft.Authorization/roleEligibilitySchedules"
    },
    ...
  ]
}

So, yes, indeed it already exists. Somehow the provider does not see it on the GET.

I also got the specific roleEligibilitySchedule

https://management.azure.com/providers/Microsoft.Management/managementGroups/lz-rocket/providers/Microsoft.Authorization/roleEligibilitySchedules/XXXXXXXX-856b-e9c6-ed6e-XXXXXXXXXXXX?api-version=2020-10-01

which returns the same object as above.

I used the https://learn.microsoft.com/en-us/rest/api/authorization/role-eligibility-schedules/get via the "Try it now" Web Version of it.

So I wonder what is the difference between how the provider checks for the existence of a roleEligibilitySchedule and the REST &| Graph API?

Important to note: This script always "fixes" it for some weeks, by just removing and adding the assignment again… 😢 It can also be done by ClickOps removing all affected assignments.

State inspection

Actually, I also now checked this (terraform show -json), and 🔴 I can not find any information in the state about the resource module.rocket.azurerm_pim_eligible_role_assignment.manager.

@xuzhang3 The state seems just to have forgotten the resource. Sadly somehow I faile(ed) to enable snapshotting or versioning so I can not evidence the change at this time. The same kind of resources (that I repaired last week) are in the state…

It can be that when I did run apply to try if it goes this this time that it already removed the "non-found" resources. Boils down again, why does it not find them if they are clearly there.

Reimport

The import fails as well:

❯ terraform import "module.rocket.azurerm_pim_eligible_role_assignment.manager" "/providers/Microsoft.Management/managementGroups/lz-rocket|/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX|XXXXXXXX-9a72-4803-a7a5-XXXXXXXXXXXX"
...
module.rocket.azurerm_pim_eligible_role_assignment.manager: Importing from ID "/providers/Microsoft.Management/managementGroups/lz-rocket|/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX|XXXXXXXX-9a72-4803-a7a5-XXXXXXXXXXXX"...
module.rocket.azurerm_pim_eligible_role_assignment.manager: Import prepared!
  Prepared azurerm_pim_eligible_role_assignment for import
module.rocket.azurerm_pim_eligible_role_assignment.manager: Refreshing state... [id=/providers/Microsoft.Management/managementGroups/lz-rocket|/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX|XXXXXXXX-9a72-4803-a7a5-XXXXXXXXXXXX]

...

│ Error: Cannot import non-existent remote object
│
│ While attempting to import an existing object to "module.rocket.azurerm_pim_eligible_role_assignment.manager", the provider detected that no object exists
│ with the given id. Only pre-existing objects can be imported; check that the id is correct and that it is associated with the provider's configured region
│ or endpoint, or use "terraform apply" to create a new remote object for this resource.

I assume the "GET" or "READ" for this works the same as the refresh so I fear that really something is wrong when getting the schedule 😢 Somewhere here maybe?

P.S.: Checking the provider

After checking a bit the provider code I also checked the roleEligibilityScheduleInstance

GET https://management.azure.com/providers/Microsoft.Management/managementGroups/lz-rocket/providers/Microsoft.Authorization/roleEligibilityScheduleInstances?api-version=2020-10-01

[{
  "properties": {
    "roleEligibilityScheduleId": "/providers/Microsoft.Management/managementGroups/lz-rocket/providers/Microsoft.Authorization/roleEligibilitySchedules/XXXXXXXX-856b-e9c6-ed6e-XXXXXXXXXXXX",
    "scope": "/providers/Microsoft.Management/managementGroups/lz-rocket",
    "roleDefinitionId": "/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX",
    "principalId": "4a7e9459-9a72-4803-a7a5-26a6aaa76c1a",
    "principalType": "Group",
    "status": "Provisioned",
    "startDateTime": "2023-11-07T12:09:29.17Z",
    "endDateTime": "2024-05-05T12:09:29.13Z",
    "memberType": "Direct",
    "createdOn": "2023-11-07T12:09:29.17Z",
    "expandedProperties": {
      "principal": {
        "id": "4a7e9459-9a72-4803-a7a5-26a6aaa76c1a",
        "displayName": "[Rocket] Managers",
        "type": "Group"
      },
      "roleDefinition": {
        "id": "/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX",
        "displayName": "[Rocket] Manager",
        "type": "CustomRole"
      },
      "scope": {
        "id": "/providers/Microsoft.Management/managementGroups/lz-rocket",
        "displayName": "Rocket",
        "type": "managementgroup"
      }
    }
  },
  "name": "XXXXXXXX-3c3d-46ea-b855-XXXXXXXXXXXX",
  "id": "/providers/Microsoft.Management/managementGroups/lz-rocket/providers/Microsoft.Authorization/roleEligibilityScheduleInstances/XXXXXXXX-3c3d-46ea-b855-XXXXXXXXXXXX",
  "type": "Microsoft.Authorization/roleEligibilityScheduleInstances"
}...]

Also the instance gets found.

Debug log

Now, as my last resort as non-go novice, I decided to enable the debug logs and give it one more try:

I found 1 thing that confirms a theory of mine above:

2023-12-27T10:57:42.057+0100 [INFO]  provider.terraform-provider-azurerm_v3.84.0_x5: [DEBUG] Role Management Policy: (Principal Id "XXXXXXXX-9a72-4803-a7a5-XXXXXXXXXXXX" / Scope "/providers/Microsoft.Management/managementGroups/lz-rocket" / Role Definition Id "/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-7241-2b75-0e65-XXXXXXXXXXXX") was not found - removing from state: timestamp="2023-12-27T10:57:42.054+0100"

So indeed, once an assignment is not found, it is just removed from the state 💡

Now interesting is this line:

https://management.azure.com/providers/Microsoft.Management/managementGroups/lz-rocket/providers/Microsoft.Authorization/roleEligibilityScheduleRequests/XXXXXXXX-856b-e9c6-ed6e-XXXXXXXXXXXX?api-version=2020-10-01

which asks for roleEligibilityScheduleRequests and not for roleEligibilityScheduleInstances or roleEligibilitySchedules. So, let me check that as well in the Explorer:

And YES, it is indeed missing!

GET https://management.azure.com/providers/Microsoft.Management/managementGroups/lz-rocket/providers/Microsoft.Authorization/roleEligibilityScheduleRequests?api-version=2020-10-01

[only the surveyors object]

SUMMARY

I believe the Read()/GET bases on a wrong entity. Maybe the first 15 minutes of a create the request should be checked but after that, the schedule or its instance should be taken as reference.

What people do when renewing the assignment is just recreating a new Role Eligibility Schedule Request for a schedule.

The technical questions is when does a "request" get removed? It appears after 4-6 weeks and then the provider has an issue. Or like how often does Microsoft "vacuum" completed requests…

@stephybun I believe this deserves to be reopened ❤️

[MohnJadden]

@unique-dominik This is valuable stuff - maybe if it doesn't get reopened, it merits getting opened in a new issue and referenced back to this one.

[unique-dominik]

@MohnJadden I will do that lets say in 24-48 hours to give people time to come back from NYE 🚀 Thank you for the motivation 🍻

[troyel]

+1 on this one! Very inconvenient error to suddenly happen after xx-days. Thanks for the research and explanations @unique-dominik

[troyel]

@xuzhang3 would you perhaps be able to have a look at this issue? That would be super helpful! As I see it now the resource azurerm_pim_eligible_role_assignment is broken and can't be used before this issue is fixed. Many thanks!

[unique-dominik]

@MohnJadden Shall we open a new issue? I somehow doubt a bit that the interaction there is bigger WDYT?

[srnebu]

After upgrading to 3.85 and 3.86 I am still not able to import this type of ressources. Currently I have to remove those not working configuration from my terraform script.

[srnebu]

any updates?

[unique-dominik]

I am a bit lost as there is no reaction on any linked issue 😢 The root cause is now clear. I wish I could code go.

[stephybun]

@xuzhang3 could you please take a look or provide an update, it appears #24077 hasn't fixed the issue entirely cc. @mybayern1974

[srnebu]

any news?

[daisy-timms-xlab]

We are also seeing this behaviour with azurerm version:3.87.0, terraform version: 1.5.7

[xuzhang3]

@stephybun we have locate some of the issues and trying to fix it.

[unique-dominik]

@xuzhang3 Any update? I look currently at:

Even though nothing changed 🙈

[daisy-timms-xlab]

Hi - we're having to do a lot of unpleasant workarounds in tf to handle this issue at the moment. Can we please have an ETA to help us decide whether we should switch to using the azapi or hold out for a fix?

[MohnJadden]

It looks like the fix for this is in a PR that still needs review before merging. If 3.90 dropped last week, maybe we'd see it merged for 3.91 in a few weeks? Who knows.

If I were you I'd try for azapi - and if you do use azapi for PIM roles like this please do post some links to sample code, I'd love to have a working PIM solution in TF somehow.

[spectrum048k]

Here's the related PR, still not merged it seems: https://github.com/hashicorp/terraform-provider-azurerm/pull/24524

[xuzhang3]

@unique-dominik #24524 is still not merged, this PR helps resolve these issues

[unique-dominik]

@xuzhang3 I know I wonder why not it seems you created it weeks ago 😭

[unique-dominik]

It starts to become sad. @MohnJadden Can you do anything about it 😞

[MohnJadden]

@unique-dominik I wish I could, but I don't work for Hashicorp - I'm just a guy who needs this functionality and am waiting on @manicminer to approve the PR so we can get the fix in.

[jakubslonxlab]

@xuzhang3 @manicminer Do you know when this PR - 24524 will be merged? This is still affecting our workloads...

[srnebu]

@xuzhang3 @manicminer had to remove some configs today again to keep my script working. Really waiting for this fix.

[xuzhang3]

@jakubslonxlab @srnebu I don't have merge permissions so I don't know when this fix will be available.

[kristeey]

Still experiencing this error for a azurerm_pim_active_role_assignment resource when trying to import it with hashicorp/azurerm version 3.94.0.

[...] the provider detected that no object exists with the given id. [...]

[xuzhang3]

@kristeey What is the status of the role assignment you are trying to import? Active or not active and does this role assignment created by current SPN?

[kristeey]

@xuzhang3 The status is Assigned and the assignment is created by the current SPN.

[kristeey]

Since this was a blocker I deleted the role assignment and reapplied it again.

[spectrum048k]

@xuzhang3 can you provide guidance on how to use this fix, i.e. do we need to import all the assignments missing from state? or do we have to delete the assignment and re-apply as @kristeey did?