azuredevops_variable_group error : 401 Unauthorized, also in 0.3.0

[azurepilot]

I am using the example from the azuredevops provider documentation to create a variable group which is linked to KV.

This issue was already reported for 0.2.0 see and claimed fixed, but there seems to be still a bug (at least in the docu)

When I use a PAT with the limited permission set mentioned in the documentation, I get: │ Error: Expanding variable group resource data: Failed to get the Azure Key vault secrets. Request returned status: 401 Unauthorized │ │ with azuredevops_variable_group.target, │ on terraform.tf line 92, in resource "azuredevops_variable_group" "target": │ 92: resource azuredevops_variable_group target {

However, if I use full scope PAT, it is nicely working.... As there are 29 scopes with up to 5 levels, it is awkard to test, which additional permissions are needed to make it working. As the error message hiding a lot of details, how can I provide additional debug details?

[xuzhang3]

@azurepilot Not all the custom permissions configurable in portal, some of the permission can be set with PAT API. See https://github.com/microsoft/terraform-provider-azuredevops/issues/712#issuecomment-1432544380 for more detals.

[pedrolsazevedo]

@azurepilot Not all the custom permissions configurable in portal, some of the permission can be set with PAT API. See #712 (comment) for more detals.

Ok, that we don't see all the permissions through the portal, but is there any documentation that tells which permissions are needed for this?

[kolosovpetro]

Any progress here? I have same issue

[xuzhang3]

@kolosovpetro As far as I know, PAT requires permissions: Build (Read); Pipeline Resources (Use and manage); Project and Team (Read, write, & manage); Service Connections (Read, query, & manage); Variable Groups (Read, create, & manage); Work Items (Read)

[kolosovpetro]

@xuzhang3 yep, PAT works fine, my issue comes when I try to authenticate azure devops provider using service principle (by client id + client secret), so I strictly follow the guide

https://registry.terraform.io/providers/microsoft/azuredevops/latest/docs/guides/authenticating_service_principal_using_a_client_secret

and still 401 having all setup including SP added to azure devops org, permissions configured, project permissions configured too etc

PS similar auth approach via client_id + secret works fine for azurerm provider

[xuzhang3]

@kolosovpetro What permissions have you assigned to the SPN in ADO? Usually project admin is enough

[kolosovpetro]

@xuzhang3 yes, service principle is within project administrator role, SP itself is contributor on subsctiption level if it matters

[erorubrz]

I am having the same issue since some time. I also am using a PAT and also tested with full access. Nothing works. I will try downgrading the provider and come back with more feedback.

Edit: I downgraded to 0.5.0 step-wise and no success. no idea why it comes up.

[xuzhang3]

@xuzhang3 yes, service principle is within project administrator role, SP itself is contributor on subsctiption level if it matters

I can not see your permission configurations, can you check the permissions of group Project Administrators and ensure all the require permission are granted?

[xuzhang3]

@erorubrz which resource are you trying to manage? Custom permission is not fully support in the web configuration page and you may need add the specific permission by the PAT API.

[erorubrz]

@erorubrz which resource are you trying to manage? Custom permission is not fully support in the web configuration page and you may need add the specific permission by the PAT API.

Hi, I managed to redo the testing yesterday and it seems to work fine now. Probably an issue on the AzDO part. Full Access worked fine yesterday and granular permissions work as well. Thank you for reaching out.

[kolosovpetro]

Any updates here? I still encounter 401 trying to authorize to AzureDevOps using SPN + client secret, below is the test project I have created

• https://github.com/kolosovpetro/terraform-azure-devops-provider

Setup of Demo project

Let's consider how SPN + client secret authorization works according to the provider documentation

• https://registry.terraform.io/providers/microsoft/azuredevops/latest/docs/guides/authenticating_service_principal_using_a_client_secret

So that there are three steps required

1. Create a service principal in Azure portal or using Azure PowerShell. Ignore steps about application roles and certificates.
2. Generate a client secret for the service principal
3. Add the service principal to your Azure DevOps Organization.

Which are satisfied by the demo project https://github.com/kolosovpetro/terraform-azure-devops-provider, more precisely

1. Create and add service principal to organization
2. Add service principal as project administrator
3. Configure provider (path to secret is a path to txt file that contains plain text SPN secret)

Terraform plan

The error I still get is 401 unauthorized, however configuration looks fine, complete log

PS E:\RiderProjects\03_TERRAFORM_PROJECTS\terraform-azure-devops-provider> terraform init -backend-config="azure.sas.conf" -reconfigure -upgrade
2024-05-31T14:35:50.772+0200 [INFO]  Terraform version: 1.8.4
2024-05-31T14:35:50.772+0200 [INFO]  Go runtime version: go1.22.1
2024-05-31T14:35:50.772+0200 [INFO]  CLI args: []string{"C:\\ProgramData\\chocolatey\\lib\\terraform\\tools\\terraform.exe", "init", "-backend-config=azure.sas.conf", "-reconfigure", "-upgrade"}
2024-05-31T14:35:50.776+0200 [INFO]  CLI command args: []string{"init", "-backend-config=azure.sas.conf", "-reconfigure", "-upgrade"}

Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding microsoft/azuredevops versions matching ">= 1.1.1"...
- Using previously-installed microsoft/azuredevops v1.1.1

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
PS E:\RiderProjects\03_TERRAFORM_PROJECTS\terraform-azure-devops-provider> terraform plan -out main.tfplan
2024-05-31T14:35:58.147+0200 [INFO]  Terraform version: 1.8.4
2024-05-31T14:35:58.148+0200 [INFO]  Go runtime version: go1.22.1
2024-05-31T14:35:58.148+0200 [INFO]  CLI args: []string{"C:\\ProgramData\\chocolatey\\lib\\terraform\\tools\\terraform.exe", "plan", "-out", "main.tfplan"}
2024-05-31T14:35:58.152+0200 [INFO]  CLI command args: []string{"plan", "-out", "main.tfplan"}
2024-05-31T14:35:58.165+0200 [INFO]  backend/local: starting Plan operation
Acquiring state lock. This may take a few moments...
2024-05-31T14:35:59.342+0200 [INFO]  provider: configuring client automatic mTLS
2024-05-31T14:35:59.365+0200 [INFO]  provider.terraform-provider-azuredevops_v1.1.1: configuring server automatic mTLS: timestamp="2024-05-31T14:35:59.364+0200"
2024-05-31T14:35:59.398+0200 [INFO]  provider: configuring client automatic mTLS
2024-05-31T14:35:59.414+0200 [INFO]  provider.terraform-provider-azuredevops_v1.1.1: configuring server automatic mTLS: timestamp="2024-05-31T14:35:59.413+0200"
2024-05-31T14:35:59.449+0200 [INFO]  backend/local: plan calling Plan
2024-05-31T14:35:59.450+0200 [INFO]  provider: configuring client automatic mTLS
2024-05-31T14:35:59.465+0200 [INFO]  provider.terraform-provider-azuredevops_v1.1.1: configuring server automatic mTLS: timestamp="2024-05-31T14:35:59.465+0200"
2024-05-31T14:35:59.498+0200 [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/microsoft/azuredevops\"]" changed the config value, but that value is unused
2024-05-31T14:35:59.498+0200 [INFO]  provider.terraform-provider-azuredevops_v1.1.1: 2024/05/31 14:35:59 [DEBUG] AzureRM Client User Agent: terraform-provider-azuredevops/v1.1.1: timestamp="2024-05-31T14:35:59.498+0200"
2024-05-31T14:35:59.875+0200 [INFO]  provider.terraform-provider-azuredevops_v1.1.1: 2024/05/31 14:35:59 getAzdoClient(): core.NewClient failed.: timestamp="2024-05-31T14:35:59.875+0200"
2024-05-31T14:35:59.875+0200 [ERROR] provider.terraform-provider-azuredevops_v1.1.1: Response contains error diagnostic: tf_provider_addr=registry.terraform.io/microsoft/azuredevops tf_req_id=a800ff47-75c6-2740-0ea0-e16221402c68 @caller=github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto diagnostic_detail="" diagnostic_severity=ERROR tf_proto_version=5.3 tf_rpc=Configure diagnostic_summary="Request returned status: 401 Unauthorized" timestamp="2024-05-31T14:35:59.875+0200"
2024-05-31T14:35:59.875+0200 [ERROR] vertex "provider[\"registry.terraform.io/microsoft/azuredevops\"]" error: Request returned status: 401 Unauthorized
2024-05-31T14:35:59.875+0200 [WARN]  Planning encountered errors, so plan is not applyable
2024-05-31T14:35:59.875+0200 [INFO]  backend/local: plan operation completed
2024-05-31T14:35:59.875+0200 [INFO]  backend/local: writing plan output to: main.tfplan

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Request returned status: 401 Unauthorized
│
│   with provider["registry.terraform.io/microsoft/azuredevops"],
│   on versions.tf line 12, in provider "azuredevops":
│   12: provider "azuredevops" {
│

Final thoughts

I have also tried to setup powershell session scoped varaibles for organization url, client ID etc, still no luck. Can it be that global env variable conflicts with the one defined in provider block inside terraform configuration?