terraform-provider-azuredevops_v1.1.1 plugin crashed when running from Azure Devops Pipeline

LukeE-NEC commented 3 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and Azure DevOps Provider) Version

Terraform v1.9.1 azuredevops v1.1.1

Affected Resource(s)

azuredevops_git_repository

Terraform Configuration Files (Truncated)

./terraform-repository/main.tf

resource "azuredevops_git_repository" "this" {
  for_each   = toset(var.repositories)
  project_id = data.azuredevops_project.cloudops.id
  name       = each.value
  initialization {
    init_type = "Clean"
  }
}

This is not the entire module but is what appears to be causing the error.

./main.tf

module "terraform-repository" {
  source = "./terraform-repository"
  repositories = [
    "lte-terraform-test-1",
    "lte-terraform-test-2",
  ]
}

Debug Output

https://gist.github.com/LukeE-NEC/46e5473d52cc9a32ff3d1ad9e946d9ed

Panic Output

https://gist.github.com/LukeE-NEC/46e5473d52cc9a32ff3d1ad9e946d9ed

Expected Behavior

We are creating a module which will take a list of repository names, and will create them in our Devops project along with some standard files/settings. This runs successfully when applied locally using a PAT.

However we would like to use an Azure Devops pipeline to run this. This issue occurs when trying to run a plan in the pipeline.

The state file is handled via a service connection with Workload Identity Federation to Azure Resource Manager. Based on the documentation it looks like all we needed to add in the provider.tf is this:

provider "azuredevops" {
  org_service_url = "https://dev.azure.com/OUR-ORG-NAME"
  use_oidc        = true
}

We have tried with and without the tenant/client id, and also tested with the use_msi options, but the same issue occurs each time. This is the first time we have tried to configure this so may be user error, however we cannot ascertain what is going wrong from the errors.

Actual Behavior

Init and Validate steps are successful but the plan fails with the error attached above.

Steps to Reproduce

terraform plan in Azure Pipeline

slzwgoosen commented 3 months ago

We are facing the same issue with an existing pipeline that deployed the Azure DevOps repositories successfully previously. Now we are getting the same 'panic: runtime error: invalid memory address or nil pointer dereference' when rerunning our pipelines. Setting the TF_LOG to Debug doesn't show much. Usually with other providers like azure ad, the logs show the API requests.

I tested locally it worked fine but then I deleted the repository and tried to run terraform plan again. Thats when I received the same error. This doesn't seem in-line with how terraform works, i.e. if there is a mismatch between current and desired then it should plan to create the resources, not fail.

Question from our side would be, how can we see more details in the logs? i.e. the request/response to Azure DevOps API Or how to debug further.

JeffBrownTech commented 3 months ago

Seeing same behavior on two different self-hosted Windows build agents during terraform plan step. Upgraded azuredevops provider from 1.0.1 to 1.1.1 with no luck. It is erroring on a data source for an existing service connection.

Happy to provide any additional output or details as needed to troubleshoot issue. Strangely enough, it works fine when running locally, only issue is in a pipeline.

Stack trace from the terraform-provider-azuredevops_v1.0.1 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x1880b71]

goroutine 96 [running]:
github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint.flattenServiceEndpointAzureRM(0xee423853399bfbec?, 0xc0000e0b40, {0xc00024c510?, 0x196c180?})
    github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_azurerm.go:426 +0x51
github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint.dataSourceServiceEndpointAzureRMRead(0x0?, {0x193ed60?, 0xc00015af20?})
    github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint/data_serviceendpoint_azurerm.go:26 +0x196
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x1c805c0?, {0x1c805c0?, 0xc00050b0b0?}, 0xd?, {0x193ed60?, 0xc00015af20?})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.23.0/helper/schema/resource.go:712 +0x178
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ReadDataApply(0xc0004ba9a0, {0x1c805c0, 0xc00050b0b0}, 0xc00010eb00, {0x193ed60, 0xc00015af20})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.23.0/helper/schema/resource.go:943 +0x145
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadDataSource(0xc0002ab500, {0x1c805c0?, 0xc00050af30?}, 0xc0004cc320)
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.23.0/helper/schema/grpc_provider.go:1179 +0x38f
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadDataSource(0xc00036ebe0, {0x1c805c0?, 0xc00050a8a0?}, 0xc00007c1e0)
    github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/tf5server/server.go:658 +0x3ef
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadDataSource_Handler({0x1abc9a0?, 0xc00036ebe0}, {0x1c805c0, 0xc00050a8a0}, 0xc0004e0000, 0x0)
    github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:421 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003c8000, {0x1c83320, 0xc0004d6340}, 0xc000234480, 0xc0004c8420, 0x2213250, 0x0)
    google.golang.org/grpc@v1.56.3/server.go:1335 +0xde3
google.golang.org/grpc.(*Server).handleStream(0xc0003c8000, {0x1c83320, 0xc0004d6340}, 0xc000234480, 0x0)
    google.golang.org/grpc@v1.56.3/server.go:1712 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc@v1.56.3/server.go:947 +0xca
created by google.golang.org/grpc.(*Server).serveStreams.func1
    google.golang.org/grpc@v1.56.3/server.go:958 +0x15c

Error: The terraform-provider-azuredevops_v1.0.1 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

chirangaalwis commented 3 months ago

Any update on an effort to fix this issue or any workaround? We are experiencing the same issue and so far, haven't found a way to resolve it so far.

catt231 commented 3 months ago

We are seeing this same running in devops - works locally - any ideas?

ThorbjoernEilers commented 2 months ago

It could be due to a service connection being manually deleted and still in the terraform state file. It worked for me after removing my github service connection from my statefile -> https://github.com/microsoft/terraform-provider-azuredevops/issues/1132#issuecomment-2298167102

LukeE-NEC commented 2 months ago

After updating to version 1.3.0 of the azuredevops module which was released today, I am no longer getting the issue. I did not get around to checking what ThorbjoernEilers suggested but this could be an option if not updating the module. Can someone else confirm updating has resolved the issue for them? If so I can close this off.

xuzhang3 commented 3 weeks ago

This is a permission, can reproduce this issue when I remove the read permission on the token used to manage the resource.

xuzhang3 commented 3 weeks ago

This issue has been fixed in v1.2.0 (#1087). Currently, if the ADO provider cannot find the repository, the repository will be deleted from the state. The tricky thing here is that we don't know whether this repository has been deleted or PAT/SPN/OIDC/identity does not have permissions. You need to check PAT/SPN/OIDC/identity permissions. The easiest way is to create a new repository with the same name. If this repository exists (and you have permission to create it), the service will reject the request.

microsoft / terraform-provider-azuredevops