search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.6k stars 4.64k forks source link

Timeout with error (but successful) when deploying custom rbac role #19603

Closed eric-mark closed 1 year ago

eric-mark commented 1 year ago

Is there an existing issue for this?

Community Note

Terraform Version

1.3.6

AzureRM Provider Version

3.34.0

Affected Resource(s)/Data Source(s)

azurerm_role_definition

Terraform Configuration Files

resource "azurerm_role_definition" "rbac_custom_sentinel_def" {
  name        = "Microsoft Sentinel Custom Role ${var.env}"
  scope       = "/subscriptions/${var.az_subscription_id_map[var.env]}/resourceGroups/${var.region_name_map[var.env]}${var.env_name_map[var.env]}-${var.rg_name}"
  description = "Microsoft Sentinel Contributor role with deny permissions to delete cases & incidents ${var.env}"

  permissions {
    actions = [
      "Microsoft.Authorization/*/read",
      "Microsoft.Insights/alertRules/*",
      "Microsoft.Insights/myworkbooks/read",
      "Microsoft.Insights/workbooks/*",
      "Microsoft.OperationalInsights/querypacks/*",
      "Microsoft.OperationalInsights/workspaces/*/read",
      "Microsoft.OperationalInsights/workspaces/analytics/query/action",
      "Microsoft.OperationalInsights/workspaces/dataSources/read",
      "Microsoft.OperationalInsights/workspaces/savedSearches/*",
      "Microsoft.operationalinsights/workspaces/tables/write",
      "Microsoft.OperationalInsights/workspaces/query/read",
      "Microsoft.OperationalInsights/workspaces/query/*/read",
      "Microsoft.OperationsManagement/solutions/read",
      "Microsoft.Resources/deployments/*",
      "Microsoft.Resources/subscriptions/resourceGroups/read",
      "Microsoft.SecurityInsights/*",
      "Microsoft.Support/*"
    ]
    not_actions = [
      "Microsoft.SecurityInsights/cases/*/Delete",
      "Microsoft.SecurityInsights/incidents/*/Delete",
      "Microsoft.SecurityInsights/ConfidentialWatchlists/*",
      "Microsoft.OperationalInsights/workspaces/query/ConfidentialWatchlist/*"
    ]
    data_actions     = []
    not_data_actions = []
  }

  assignable_scopes = [
    "/subscriptions/${var.az_subscription_id_map[var.env]}/resourceGroups/${var.region_name_map[var.env]}${var.env_name_map[var.env]}-${var.rg_name}"
  ]

Debug Output/Panic Output

{"properties":{"roleName":"Microsoft Sentinel Custom Role","type":"CustomRole","description":"Microsoft Sentinel Contributor role with deny permissions to delete cases & incidents sbx","assignableScopes":["/subscriptions/XXXXXX-XXXXXXXXX-XXXXXXXXX-XXXXXXXXXX/resourceGroups/cdb-siemsnt01.rg"],"permissions":[{"actions":["Microsoft.Authorization/*/read","Microsoft.Insights/alertRules/*","Microsoft.Insights/myworkbooks/read","Microsoft.Insights/workbooks/*","Microsoft.OperationalInsights/querypacks/*","Microsoft.OperationalInsights/workspaces/*/read","Microsoft.OperationalInsights/workspaces/analytics/query/action","Microsoft.OperationalInsights/workspaces/dataSources/read","Microsoft.OperationalInsights/workspaces/savedSearches/*","Microsoft.OperationalInsights/workspaces/query/read","Microsoft.OperationalInsights/workspaces/query/*/read","Microsoft.OperationsManagement/solutions/read","Microsoft.Resources/deployments/*","Microsoft.Resources/subscriptions/resourceGroups/read","Microsoft.SecurityInsights/*","Microsoft.Support/*"],"notActions":["Microsoft.SecurityInsights/cases/*/Delete","Microsoft.SecurityInsights/incidents/*/Delete","Microsoft.SecurityInsights/ConfidentialWatchlists/*","Microsoft.OperationalInsights/workspaces/query/ConfidentialWatchlist/*"],"dataActions":[],"notDataActions":[]}],"createdOn":"2022-08-10T18:56:33.7839463Z","updatedOn":"2022-12-06T23:36:05.7652010Z","createdBy":"XXXXXX-XXXXXXXX-XXXXXXX-XXXXXXXX","updatedBy":"XXXXXX-XXXXXXXX-XXXXXXX-XXXXXXXX"},"id":"/subscriptions/XXXXXX-XXXXXXXXX-XXXXXXXXX-XXXXXXXXXX/providers/Microsoft.Authorization/roleDefinitions/XXXXXXX-XXXXXXXXXXX-XXXXXXXX-XXXXXXXXXXX","type":"Microsoft.Authorization/roleDefinitions","name":"XXXXXXX-XXXXXXXXXXX-XXXXXXXX-XXXXXXXXXXX"}: timestamp=2022-12-06T15:45:22.984-0800
2022-12-06T15:45:23.002-0800 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2022-12-06T15:45:23.006-0800 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.34.0/linux_amd64/terraform-provider-azurerm_v3.34.0_x5 pid=4122
2022-12-06T15:45:23.006-0800 [DEBUG] provider: plugin exited
2022-12-06T15:45:23.014-0800 [DEBUG] POST https://gitlab.com/api/v4/projects/XXXXXXXXXX/terraform/state/rbac_custom_roles_sbx?ID=XXXXXX-XXXXXX-XXXXX-XXXXX
2022-12-06T15:45:23.808-0800 [DEBUG] DELETE https://gitlab.com/api/v4/projects/XXXXXXXXXX/terraform/state/rbac_custom_roles_sbx/lock

Expected Behaviour

We would expect this to completed without the EOF errors within a reasonable time. Running this command through the web portal takes seconds, but through the AzureRM provider via Terraform it takes anywhere to 6.5 to 9 minutes.

Actual Behaviour

The AzureRM call is taking 6.5 to 9 minutes to complete and it returning with EOF errors. We have also noticed this behaviour in azurerm_role_assignment too.

Steps to Reproduce

TF_LOG_PATH="./tf-debug.log" TF_LOG=DEBUG tf apply

Important Factoids

n/a

References

We have support case with Microsoft #2211290010003278 and they also recommended that we should open a GitHub issue because they also believe there may be an issue with the AzureRM Provider.

ziyeqf commented 1 year ago

Hi @eric-mark, thanks for reporting.

For the EOF error, Could you please check the log file if there are any logs look like stack trace? I assume it is caused by a bug in Provider while these description may be not enough to diagnosis.

For the time it takes more than Portal: I have noticed you are using remote state feature of Terraform. Could you please try with local state file? Then we can narrow down the possible cause of why it takes more time.

eric-mark commented 1 year ago

Hi @ziyeqf,

I do not see any logs that look like stack trace. Here is a bit more logs showing that it's stuck in the loop. We noticed from the portal that the job is completed, but via AzureRM provider, it's just stuck in the loop until it times out.

2022-12-09T09:49:14.054-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Request: GET //subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview HTTP/1.1 Host: management.azure.com User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 authorization/2020-04-01-preview HashiCorp Terraform/1.3.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-XXXXX-XXXXX-XXXXX-XXXXX X-Ms-Correlation-Request-Id: 471639a8-9cd6-3026-5de2-cd9d5ee3e9fb Accept-Encoding: gzip: timestamp=2022-12-09T09:49:14.054-0800 2022-12-09T09:49:14.167-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Response for https://management.azure.com//subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview: HTTP/2.0 200 OK Cache-Control: no-cache Content-Type: application/json; charset=utf-8 Date: Fri, 09 Dec 2022 17:48:55 GMT Expires: -1 Pragma: no-cache Set-Cookie: x-ms-gateway-slice=Production; path=/; secure; samesite=none; httponly Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Ms-Correlation-Request-Id: 471639a8-9cd6-3026-5de2-cd9d5ee3e9fb X-Ms-Ratelimit-Remaining-Subscription-Reads: 11903 X-Ms-Request-Id: a1fd0793-d00d-4c46-9e99-629bd273b74f X-Ms-Routing-Request-Id: WESTUS2:20221209T174855Z:bd72799f-e5ec-4f3d-b062-9c0da7e81f9b

{"properties":{"roleName":"Microsoft Sentinel Custom Role","type":"CustomRole","description":"Microsoft Sentinel Operator role with workspace, watchlist and threat intelligence permissions sbx","assignableScopes":["/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg"],"permissions":[{"actions":["Microsoft.Authorization//read","Microsoft.Insights/alertRules/","Microsoft.Insights/myworkbooks/read","Microsoft.Insights/workbooks/","Microsoft.OperationalInsights/querypacks/","Microsoft.OperationalInsights/workspaces//read","Microsoft.OperationalInsights/workspaces/analytics/query/action","Microsoft.OperationalInsights/workspaces/dataSources/read","Microsoft.OperationalInsights/workspaces/savedSearches/","Microsoft.OperationalInsights/workspaces/query/read","Microsoft.OperationalInsights/workspaces/query//read","Microsoft.OperationsManagement/solutions/read","Microsoft.Resources/deployments/","Microsoft.Resources/subscriptions/resourceGroups/read","Microsoft.SecurityInsights//read","Microsoft.SecurityInsights/automationRules/","Microsoft.SecurityInsights/cases/","Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action","Microsoft.SecurityInsights/incidents/","Microsoft.SecurityInsights/threatIntelligence/","Microsoft.SecurityInsights/Watchlists/","Microsoft.Support/"],"notActions":["Microsoft.SecurityInsights/cases//Delete","Microsoft.SecurityInsights/incidents//Delete","Microsoft.SecurityInsights/ConfidentialWatchlists/","Microsoft.OperationalInsights/workspaces/query/ConfidentialWatchlist/*"],"dataActions":[],"notDataActions":[]}],"createdOn":"2022-08-23T23:32:48.6190540Z","updatedOn":"2022-12-09T17:42:40.8377336Z","createdBy":"XXXXX-XXXXX-XXXXX-XXXXX","updatedBy":"XXXXX-XXXXX-XXXXX-XXXXX"},"id":"/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX","type":"Microsoft.Authorization/roleDefinitions","name":"XXXXX-XXXXX-XXXXX-XXXXX"}: timestamp=2022-12-09T09:49:14.167-0800 2022-12-09T09:49:24.178-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Request: GET //subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview HTTP/1.1 Host: management.azure.com User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 authorization/2020-04-01-preview HashiCorp Terraform/1.3.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-XXXXX-XXXXX-XXXXX-XXXXX X-Ms-Correlation-Request-Id: 471639a8-9cd6-3026-5de2-cd9d5ee3e9fb Accept-Encoding: gzip: timestamp=2022-12-09T09:49:24.177-0800 2022-12-09T09:49:24.301-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Response for https://management.azure.com//subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview: HTTP/2.0 200 OK Cache-Control: no-cache Content-Type: application/json; charset=utf-8 Date: Fri, 09 Dec 2022 17:49:05 GMT Expires: -1 Pragma: no-cache Set-Cookie: x-ms-gateway-slice=Production; path=/; secure; samesite=none; httponly Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Ms-Correlation-Request-Id: 471639a8-9cd6-3026-5de2-cd9d5ee3e9fb X-Ms-Ratelimit-Remaining-Subscription-Reads: 11902 X-Ms-Request-Id: 664d9245-1bd0-47fa-83d1-9461b0f26297 X-Ms-Routing-Request-Id: WESTUS2:20221209T174905Z:08160483-29c5-457d-af26-5a84b816defb

{"properties":{"roleName":"Microsoft Sentinel Custom Role","type":"CustomRole","description":"Microsoft Sentinel Operator role with workspace, watchlist and threat intelligence permissions sbx","assignableScopes":["/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg"],"permissions":[{"actions":["Microsoft.Authorization//read","Microsoft.Insights/alertRules/","Microsoft.Insights/myworkbooks/read","Microsoft.Insights/workbooks/","Microsoft.OperationalInsights/querypacks/","Microsoft.OperationalInsights/workspaces//read","Microsoft.OperationalInsights/workspaces/analytics/query/action","Microsoft.OperationalInsights/workspaces/dataSources/read","Microsoft.OperationalInsights/workspaces/savedSearches/","Microsoft.OperationalInsights/workspaces/query/read","Microsoft.OperationalInsights/workspaces/query//read","Microsoft.OperationsManagement/solutions/read","Microsoft.Resources/deployments/","Microsoft.Resources/subscriptions/resourceGroups/read","Microsoft.SecurityInsights//read","Microsoft.SecurityInsights/automationRules/","Microsoft.SecurityInsights/cases/","Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action","Microsoft.SecurityInsights/incidents/","Microsoft.SecurityInsights/threatIntelligence/","Microsoft.SecurityInsights/Watchlists/","Microsoft.Support/"],"notActions":["Microsoft.SecurityInsights/cases//Delete","Microsoft.SecurityInsights/incidents//Delete","Microsoft.SecurityInsights/ConfidentialWatchlists/","Microsoft.OperationalInsights/workspaces/query/ConfidentialWatchlist/*"],"dataActions":[],"notDataActions":[]}],"createdOn":"2022-08-23T23:32:48.6190540Z","updatedOn":"2022-12-09T17:42:40.8377336Z","createdBy":"XXXXX-XXXXX-XXXXX-XXXXX","updatedBy":"XXXXX-XXXXX-XXXXX-XXXXX"},"id":"/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX","type":"Microsoft.Authorization/roleDefinitions","name":"XXXXX-XXXXX-XXXXX-XXXXX"}: timestamp=2022-12-09T09:49:24.301-0800 2022-12-09T09:49:34.311-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Request: GET //subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview HTTP/1.1 Host: management.azure.com User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 authorization/2020-04-01-preview HashiCorp Terraform/1.3.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-XXXXX-XXXXX-XXXXX-XXXXX X-Ms-Correlation-Request-Id: 471639a8-9cd6-3026-5de2-cd9d5ee3e9fb Accept-Encoding: gzip: timestamp=2022-12-09T09:49:34.311-0800 2022-12-09T09:49:34.525-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Response for https://management.azure.com//subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview: HTTP/2.0 200 OK Cache-Control: no-cache Content-Type: application/json; charset=utf-8 Date: Fri, 09 Dec 2022 17:49:15 GMT Expires: -1 Pragma: no-cache Set-Cookie: x-ms-gateway-slice=Production; path=/; secure; samesite=none; httponly Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Ms-Correlation-Request-Id: 471639a8-9cd6-3026-5de2-cd9d5ee3e9fb X-Ms-Ratelimit-Remaining-Subscription-Reads: 11901 X-Ms-Request-Id: 8441a0d2-1175-4f64-a883-62e30fa3963d X-Ms-Routing-Request-Id: WESTUS2:20221209T174915Z:637d7566-5f54-4118-9a04-57a7b2ef4eed

{"properties":{"roleName":"Microsoft Sentinel Custom Role","type":"CustomRole","description":"Microsoft Sentinel Operator role with workspace, watchlist and threat intelligence permissions sbx","assignableScopes":["/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg"],"permissions":[{"actions":["Microsoft.Authorization//read","Microsoft.Insights/alertRules/","Microsoft.Insights/myworkbooks/read","Microsoft.Insights/workbooks/","Microsoft.OperationalInsights/querypacks/","Microsoft.OperationalInsights/workspaces//read","Microsoft.OperationalInsights/workspaces/analytics/query/action","Microsoft.OperationalInsights/workspaces/dataSources/read","Microsoft.OperationalInsights/workspaces/savedSearches/","Microsoft.OperationalInsights/workspaces/query/read","Microsoft.OperationalInsights/workspaces/query//read","Microsoft.OperationsManagement/solutions/read","Microsoft.Resources/deployments/","Microsoft.Resources/subscriptions/resourceGroups/read","Microsoft.SecurityInsights//read","Microsoft.SecurityInsights/automationRules/","Microsoft.SecurityInsights/cases/","Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action","Microsoft.SecurityInsights/incidents/","Microsoft.SecurityInsights/threatIntelligence/","Microsoft.SecurityInsights/Watchlists/","Microsoft.Support/"],"notActions":["Microsoft.SecurityInsights/cases//Delete","Microsoft.SecurityInsights/incidents//Delete","Microsoft.SecurityInsights/ConfidentialWatchlists/","Microsoft.OperationalInsights/workspaces/query/ConfidentialWatchlist/*"],"dataActions":[],"notDataActions":[]}],"createdOn":"2022-08-23T23:32:48.6190540Z","updatedOn":"2022-12-09T17:42:40.8377336Z","createdBy":"XXXXX-XXXXX-XXXXX-XXXXX","updatedBy":"XXXXX-XXXXX-XXXXX-XXXXX"},"id":"/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX","type":"Microsoft.Authorization/roleDefinitions","name":"XXXXX-XXXXX-XXXXX-XXXXX"}: timestamp=2022-12-09T09:49:34.524-0800 2022-12-09T09:49:34.525-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Request: GET //subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview HTTP/1.1 Host: management.azure.com User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 authorization/2020-04-01-preview HashiCorp Terraform/1.3.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-XXXXX-XXXXX-XXXXX-XXXXX X-Ms-Correlation-Request-Id: 471639a8-9cd6-3026-5de2-cd9d5ee3e9fb Accept-Encoding: gzip: timestamp=2022-12-09T09:49:34.525-0800 2022-12-09T09:49:34.634-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Response for https://management.azure.com//subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview: HTTP/2.0 200 OK Cache-Control: no-cache Content-Type: application/json; charset=utf-8 Date: Fri, 09 Dec 2022 17:49:15 GMT Expires: -1 Pragma: no-cache Set-Cookie: x-ms-gateway-slice=Production; path=/; secure; samesite=none; httponly Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Ms-Correlation-Request-Id: 471639a8-9cd6-3026-5de2-cd9d5ee3e9fb X-Ms-Ratelimit-Remaining-Subscription-Reads: 11900 X-Ms-Request-Id: 0e9e530b-9153-4b60-987b-ec5bb06f1e75 X-Ms-Routing-Request-Id: WESTUS2:20221209T174915Z:da666331-bd1e-4e94-b6e3-02397a70e3f2

{"properties":{"roleName":"Microsoft Sentinel Custom Role","type":"CustomRole","description":"Microsoft Sentinel Operator role with workspace, watchlist and threat intelligence permissions sbx","assignableScopes":["/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg"],"permissions":[{"actions":["Microsoft.Authorization//read","Microsoft.Insights/alertRules/","Microsoft.Insights/myworkbooks/read","Microsoft.Insights/workbooks/","Microsoft.OperationalInsights/querypacks/","Microsoft.OperationalInsights/workspaces//read","Microsoft.OperationalInsights/workspaces/analytics/query/action","Microsoft.OperationalInsights/workspaces/dataSources/read","Microsoft.OperationalInsights/workspaces/savedSearches/","Microsoft.OperationalInsights/workspaces/query/read","Microsoft.OperationalInsights/workspaces/query//read","Microsoft.OperationsManagement/solutions/read","Microsoft.Resources/deployments/","Microsoft.Resources/subscriptions/resourceGroups/read","Microsoft.SecurityInsights//read","Microsoft.SecurityInsights/automationRules/","Microsoft.SecurityInsights/cases/","Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action","Microsoft.SecurityInsights/incidents/","Microsoft.SecurityInsights/threatIntelligence/","Microsoft.SecurityInsights/Watchlists/","Microsoft.Support/"],"notActions":["Microsoft.SecurityInsights/cases//Delete","Microsoft.SecurityInsights/incidents//Delete","Microsoft.SecurityInsights/ConfidentialWatchlists/","Microsoft.OperationalInsights/workspaces/query/ConfidentialWatchlist/*"],"dataActions":[],"notDataActions":[]}],"createdOn":"2022-08-23T23:32:48.6190540Z","updatedOn":"2022-12-09T17:42:40.8377336Z","createdBy":"XXXXX-XXXXX-XXXXX-XXXXX","updatedBy":"XXXXX-XXXXX-XXXXX-XXXXX"},"id":"/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX","type":"Microsoft.Authorization/roleDefinitions","name":"XXXXX-XXXXX-XXXXX-XXXXX"}: timestamp=2022-12-09T09:49:34.633-0800 2022-12-09T09:49:34.639-0800 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF" 2022-12-09T09:49:34.642-0800 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.35.0/linux_amd64/terraform-provider-azurerm_v3.35.0_x5 pid=924 2022-12-09T09:49:34.642-0800 [DEBUG] provider: plugin exited 2022-12-09T09:49:34.644-0800 [DEBUG] POST https://gitlab.com/api/v4/projects/XXXXXX/terraform/state/rbac_custom_roles_sbx?ID=XXXXX-XXXXX-XXXXX-XXXXX 2022-12-09T09:49:35.279-0800 [DEBUG] DELETE https://gitlab.com/api/v4/projects/XXXXXX/terraform/state/rbac_custom_roles_sbx/lock

eric-mark commented 1 year ago

Hi @ziyeqf,

Here is the same operation ran using local state which took 8.5 minutes to deploy:

{"properties":{"roleName":"Microsoft Sentinel Custom Role sbx","type":"CustomRole","description":"Microsoft Sentinel Operator role with workspace, watchlist and threat intelligence permissions sbx","assignableScopes":["/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg"],"permissions":[{"actions":["Microsoft.Authorization//read","Microsoft.Insights/alertRules/","Microsoft.Insights/myworkbooks/read","Microsoft.Insights/workbooks/","Microsoft.OperationalInsights/querypacks/","Microsoft.OperationalInsights/workspaces//read","Microsoft.OperationalInsights/workspaces/analytics/query/action","Microsoft.OperationalInsights/workspaces/dataSources/read","Microsoft.OperationalInsights/workspaces/savedSearches/","Microsoft.operationalinsights/workspaces/tables/write","Microsoft.OperationalInsights/workspaces/query/read","Microsoft.OperationalInsights/workspaces/query//read","Microsoft.OperationsManagement/solutions/read","Microsoft.Resources/deployments/","Microsoft.Resources/subscriptions/resourceGroups/read","Microsoft.SecurityInsights//read","Microsoft.SecurityInsights/automationRules/","Microsoft.SecurityInsights/cases/","Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action","Microsoft.SecurityInsights/incidents/","Microsoft.SecurityInsights/threatIntelligence/","Microsoft.SecurityInsights/Watchlists/","Microsoft.Support/"],"notActions":["Microsoft.SecurityInsights/cases//Delete","Microsoft.SecurityInsights/incidents//Delete","Microsoft.SecurityInsights/ConfidentialWatchlists/","Microsoft.OperationalInsights/workspaces/query/ConfidentialWatchlist/*"],"dataActions":[],"notDataActions":[]}],"createdOn":"2022-08-23T23:32:48.6190540Z","updatedOn":"2022-12-09T18:23:22.7141332Z","createdBy":"XXXXX-XXXXX-XXXXX-XXXXX","updatedBy":"XXXXX-XXXXX-XXXXX-XXXXX"},"id":"/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX","type":"Microsoft.Authorization/roleDefinitions","name":"XXXXX-XXXXX-XXXXX-XXXXX"}: timestamp=2022-12-09T10:32:28.910-0800 2022-12-09T10:32:38.921-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Request: GET //subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview HTTP/1.1 Host: management.azure.com User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 authorization/2020-04-01-preview HashiCorp Terraform/1.3.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-XXXXX-XXXXX-XXXXX-XXXXX X-Ms-Correlation-Request-Id: 615c26d2-249b-5e7b-249f-5459f71e9288 Accept-Encoding: gzip: timestamp=2022-12-09T10:32:38.921-0800 2022-12-09T10:32:39.034-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Response for https://management.azure.com//subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview: HTTP/2.0 200 OK Cache-Control: no-cache Content-Type: application/json; charset=utf-8 Date: Fri, 09 Dec 2022 18:32:19 GMT Expires: -1 Pragma: no-cache Set-Cookie: x-ms-gateway-slice=Production; path=/; secure; samesite=none; httponly Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Ms-Correlation-Request-Id: 615c26d2-249b-5e7b-249f-5459f71e9288 X-Ms-Ratelimit-Remaining-Subscription-Reads: 11952 X-Ms-Request-Id: 84d581e5-cbd0-4884-b032-967b1fa7b153 X-Ms-Routing-Request-Id: WESTUS2:20221209T183220Z:d0498695-83d6-4466-971c-51b6b547081c

{"properties":{"roleName":"Microsoft Sentinel Custom Role sbx","type":"CustomRole","description":"Microsoft Sentinel Operator role with workspace, watchlist and threat intelligence permissions sbx","assignableScopes":["/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg"],"permissions":[{"actions":["Microsoft.Authorization//read","Microsoft.Insights/alertRules/","Microsoft.Insights/myworkbooks/read","Microsoft.Insights/workbooks/","Microsoft.OperationalInsights/querypacks/","Microsoft.OperationalInsights/workspaces//read","Microsoft.OperationalInsights/workspaces/analytics/query/action","Microsoft.OperationalInsights/workspaces/dataSources/read","Microsoft.OperationalInsights/workspaces/savedSearches/","Microsoft.operationalinsights/workspaces/tables/write","Microsoft.OperationalInsights/workspaces/query/read","Microsoft.OperationalInsights/workspaces/query//read","Microsoft.OperationsManagement/solutions/read","Microsoft.Resources/deployments/","Microsoft.Resources/subscriptions/resourceGroups/read","Microsoft.SecurityInsights//read","Microsoft.SecurityInsights/automationRules/","Microsoft.SecurityInsights/cases/","Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action","Microsoft.SecurityInsights/incidents/","Microsoft.SecurityInsights/threatIntelligence/","Microsoft.SecurityInsights/Watchlists/","Microsoft.Support/"],"notActions":["Microsoft.SecurityInsights/cases//Delete","Microsoft.SecurityInsights/incidents//Delete","Microsoft.SecurityInsights/ConfidentialWatchlists/","Microsoft.OperationalInsights/workspaces/query/ConfidentialWatchlist/*"],"dataActions":[],"notDataActions":[]}],"createdOn":"2022-08-23T23:32:48.6190540Z","updatedOn":"2022-12-09T18:23:22.7141332Z","createdBy":"XXXXX-XXXXX-XXXXX-XXXXX","updatedBy":"XXXXX-XXXXX-XXXXX-XXXXX"},"id":"/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX","type":"Microsoft.Authorization/roleDefinitions","name":"XXXXX-XXXXX-XXXXX-XXXXX"}: timestamp=2022-12-09T10:32:39.034-0800 2022-12-09T10:32:39.035-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Request: GET //subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview HTTP/1.1 Host: management.azure.com User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 authorization/2020-04-01-preview HashiCorp Terraform/1.3.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-XXXXX-XXXXX-XXXXX-XXXXX X-Ms-Correlation-Request-Id: 615c26d2-249b-5e7b-249f-5459f71e9288 Accept-Encoding: gzip: timestamp=2022-12-09T10:32:39.034-0800 2022-12-09T10:32:39.131-0800 [DEBUG] provider.terraform-provider-azurerm_v3.35.0_x5: AzureRM Response for https://management.azure.com//subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX?api-version=2018-01-01-preview: HTTP/2.0 200 OK Cache-Control: no-cache Content-Type: application/json; charset=utf-8 Date: Fri, 09 Dec 2022 18:32:19 GMT Expires: -1 Pragma: no-cache Set-Cookie: x-ms-gateway-slice=Production; path=/; secure; samesite=none; httponly Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Ms-Correlation-Request-Id: 615c26d2-249b-5e7b-249f-5459f71e9288 X-Ms-Ratelimit-Remaining-Subscription-Reads: 11951 X-Ms-Request-Id: 3f56ac40-73cd-4f07-9044-3f26f1ac8b86 X-Ms-Routing-Request-Id: WESTUS2:20221209T183220Z:eb4156fb-174a-48b0-aa84-9d28729041ab

{"properties":{"roleName":"Microsoft Sentinel Custom Role sbx","type":"CustomRole","description":"Microsoft Sentinel Operator role with workspace, watchlist and threat intelligence permissions sbx","assignableScopes":["/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/resourceGroups/cdb-siemsnt01.rg"],"permissions":[{"actions":["Microsoft.Authorization//read","Microsoft.Insights/alertRules/","Microsoft.Insights/myworkbooks/read","Microsoft.Insights/workbooks/","Microsoft.OperationalInsights/querypacks/","Microsoft.OperationalInsights/workspaces//read","Microsoft.OperationalInsights/workspaces/analytics/query/action","Microsoft.OperationalInsights/workspaces/dataSources/read","Microsoft.OperationalInsights/workspaces/savedSearches/","Microsoft.operationalinsights/workspaces/tables/write","Microsoft.OperationalInsights/workspaces/query/read","Microsoft.OperationalInsights/workspaces/query//read","Microsoft.OperationsManagement/solutions/read","Microsoft.Resources/deployments/","Microsoft.Resources/subscriptions/resourceGroups/read","Microsoft.SecurityInsights//read","Microsoft.SecurityInsights/automationRules/","Microsoft.SecurityInsights/cases/","Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action","Microsoft.SecurityInsights/incidents/","Microsoft.SecurityInsights/threatIntelligence/","Microsoft.SecurityInsights/Watchlists/","Microsoft.Support/"],"notActions":["Microsoft.SecurityInsights/cases//Delete","Microsoft.SecurityInsights/incidents//Delete","Microsoft.SecurityInsights/ConfidentialWatchlists/","Microsoft.OperationalInsights/workspaces/query/ConfidentialWatchlist/*"],"dataActions":[],"notDataActions":[]}],"createdOn":"2022-08-23T23:32:48.6190540Z","updatedOn":"2022-12-09T18:23:22.7141332Z","createdBy":"XXXXX-XXXXX-XXXXX-XXXXX","updatedBy":"XXXXX-XXXXX-XXXXX-XXXXX"},"id":"/subscriptions/XXXXX-XXXXX-XXXXX-XXXXX/providers/Microsoft.Authorization/roleDefinitions/XXXXX-XXXXX-XXXXX-XXXXX","type":"Microsoft.Authorization/roleDefinitions","name":"XXXXX-XXXXX-XXXXX-XXXXX"}: timestamp=2022-12-09T10:32:39.130-0800 2022-12-09T10:32:39.148-0800 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF" 2022-12-09T10:32:39.150-0800 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.35.0/linux_amd64/terraform-provider-azurerm_v3.35.0_x5 pid=1656 2022-12-09T10:32:39.150-0800 [DEBUG] provider: plugin exited

ziyeqf commented 1 year ago

Hi @eric-mark, thanks for these logs. I assume this happended when updating an existing resource, could you please confirm?

In TF provider we need to make sure it has achieved consistency then we can tell it has been updated successfully, it does need more time than Portal.

For the "EOF" error, it seems to be the way how provider exit, that's not an error. A way to veritfy is to create a simple config with only one resource group, and we can see it will succeed with the same EOF error in the log file.

For any further question please leave comments, thank you!

eric-mark commented 1 year ago

Hi @ziyeqf,

In this scenario, yes, it was updating an existing resource. We would either add or remove a permission from our custom rbac role. The major issue we have found that this is taking 8.5 minutes on average to complete. If we do this through the portal, it would take seconds to complete. We also noticed that when we deploy this change via AzureRM provider, we can navigate to the portal and it will show the job completed, but the AzureRM provider seems to be stuck in a loop for 8.5 minute before it says that the job is complete.

If we compare the AzureRM provider against AWS, AWS by far is 5 times faster to deploy our resources compared to AzureRM.

ziyeqf commented 1 year ago

Hi @eric-mark,

My explanation is the "update operation" has done, but we need to keep reading (seems to be a loop) to make sure what we read is as same as what we set. In case when deploying other resources depedent on this and get an outdated value. it takes at least 120s (12 * 10s) to veritfy, and why it took far more than 2 mintues is it did get an outdated value, then another 120 senconds, until it is stable.

On Portal it shows the "update operation" status and yes it has been finished. the verification cost most time.

eric-mark commented 1 year ago

Is there a way to improve this time delay? We don't have these kind of delay issues using the AWS provider when we build in AWS.

ziyeqf commented 1 year ago

Hi @eric-mark, there is an update has been done and it should be faster now (will cost about 120s).

eric-mark commented 1 year ago

Hi @ziyeqf

Just tested this our and it took 172s which is much better than before. Thank you.

github-actions[bot] commented 5 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.