tmeckel
commented
2 years ago @andreassiegel would it be feasible for you to call the REST API on the problematic Service Endpoint and provide the output (secrets redacted of course 😁) here?
Closed andreassiegel closed 1 week ago
tmeckel
commented
2 years ago @andreassiegel would it be feasible for you to call the REST API on the problematic Service Endpoint and provide the output (secrets redacted of course 😁) here?
andreassiegel
commented
2 years ago This is the response I'm getting for the GHE service connection (personal information removed and IDs shortened):
{
"data": {
},
"id": "047265e7-496d-441c-b80e-...",
"name": "devops-git",
"type": "githubenterprise",
"url": "https://git.customer.com/",
"createdBy": {
...
<all the information about my colleague who can still work with the Terraform script>
...
},
"description": "Service connection to access Github Enterprise, managed by Terraform (devops-base-setup)",
"authorization": {
"parameters": {
"apitoken": null
},
"scheme": "Token"
},
"isShared": false,
"isReady": true,
"owner": "Library",
"serviceEndpointProjectReferences": [
{
"projectReference": {
"id": "1e40a925-51b0-49a2-9d71-...",
"name": "projectname"
},
"name": "devops-git",
"description": "Service connection to access Github Enterprise, managed by Terraform (devops-base-setup)"
}
]
}Meanwhile, we figured out that someone has changed the administrator group in Azure DevOps so that I apparently lost access to several resources. I was only able to get the response above after I was manually added as an administrator again. 💥
Without the administrator access, I only get the following response:
{
"data": {
},
"id": "047265e7-496d-441c-b80e-...",
"name": "devops-git",
"isShared": false,
"isReady": true,
"serviceEndpointProjectReferences": null
}I guess, the null value in that response is causing the error I was seen during terraform plan.
xuzhang3
commented
2 years ago @andreassiegel Can you retry with the right permission granted? Similar to #491 , by grant the wrong permissions, service won't return the required property.
guhcampos
commented
2 years ago @xuzhang3 can you specify what are the required permissions and possibly document them?
I am having the exact same problem when running my terraform code in Azure Pipelines. The Build Service account for my pipeline has Edit project-level Information, View project-level Information and Manage project properties Allowed in the destination project, I figured that should be enough.
I can run the code myself without errors, the crash only happens when using the restricted service account above.
resource "azuredevops_serviceendpoint_kubernetes" "kubernetes_service_connection" {
for_each = data.azuredevops_project.projects_for_integration
project_id = each.value.project_id
service_endpoint_name = var.gke_cluster_name
apiserver_url = var.gke_cluster_endpoint
authorization_type = "ServiceAccount"
service_account {
token = base64encode(lookup(data.kubernetes_secret.azure_devops_sa_token.data, "token"))
ca_cert = base64encode(lookup(data.kubernetes_secret.azure_devops_sa_token.data, "ca.crt"))
}
}@guhcampos The permission based on the PAT not service account. Can you provide the logs?
guhcampos
commented
2 years ago @xuzhang3 I don't see why the PAT would be required, as the Service Token is capable of requesting most of the same permissions. Even so, I never found in the docs what permissions the PAT does require. The docs state that:
_
Custom Defined PAT Scopes Personal access tokens can be used to authorize full or scoped access associated with the token for deployment of different Azure DevOps resources. If you wish to limit the PAT scope select a custom defined authorization for the token and select the permission required for management of the respective resource.
_
But never mentions what are the required permissions required for each resource. I assumed I'd need the permissions stated in my previous post for managing service connections. In a real world scenario it's not desirable to have a fully scoped permission associated to either a PAT or a Service Token.
My stack trace follows:
2022-05-06T22:08:38.7651854Z │ Error: Request cancelled
2022-05-06T22:08:38.7652351Z │
2022-05-06T22:08:38.7653180Z │ with [REDACTED].kubernetes_service_connection["Devops Team"],
2022-05-06T22:08:38.7654423Z │ on .terraform/modules/[REDACTED] line 34, in resource "azuredevops_serviceendpoint_kubernetes" "kubernetes_service_connection":
2022-05-06T22:08:38.7655376Z │ 34: resource "azuredevops_serviceendpoint_kubernetes" "kubernetes_service_connection" {
2022-05-06T22:08:38.7655884Z │
2022-05-06T22:08:38.7656434Z │ The plugin.(*GRPCProvider).ReadResource request was cancelled.
2022-05-06T22:08:38.7656959Z ╵
2022-05-06T22:08:38.7658790Z ╷
2022-05-06T22:08:38.7659738Z │ Error: Plugin did not respond
2022-05-06T22:08:38.7660570Z │
2022-05-06T22:08:38.7662034Z │ with [REDACTED].kubernetes_service_connection["Arquitetura e Processos de TI"],
2022-05-06T22:08:38.7663879Z │ on .terraform/modules/[REDACTED] line 34, in resource "azuredevops_serviceendpoint_kubernetes" "kubernetes_service_connection":
2022-05-06T22:08:38.7666007Z │ 34: resource "azuredevops_serviceendpoint_kubernetes" "kubernetes_service_connection" {
2022-05-06T22:08:38.7666861Z │
2022-05-06T22:08:38.7667376Z │ The plugin encountered an error, and failed to respond to the
2022-05-06T22:08:38.7667999Z │ plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
2022-05-06T22:08:38.7668556Z │ details.
2022-05-06T22:08:38.7668916Z ╵
2022-05-06T22:08:38.7669326Z ╷
2022-05-06T22:08:38.7669774Z │ Error: Request cancelled
2022-05-06T22:08:38.7670184Z │
2022-05-06T22:08:38.7670947Z │ with [REDACTED].kubernetes_service_connection["Depositaria"],
2022-05-06T22:08:38.7672755Z │ on .terraform/modules/[REDACTED] line 34, in resource "azuredevops_serviceendpoint_kubernetes" "kubernetes_service_connection":
2022-05-06T22:08:38.7673685Z │ 34: resource "azuredevops_serviceendpoint_kubernetes" "kubernetes_service_connection" {
2022-05-06T22:08:38.7674187Z │
2022-05-06T22:08:38.7674786Z │ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
2022-05-06T22:08:38.7675290Z ╵
2022-05-06T22:08:39.1031559Z
2022-05-06T22:08:39.1033737Z Stack trace from the terraform-provider-azuredevops_v0.2.1 plugin:
2022-05-06T22:08:39.1034152Z
2022-05-06T22:08:39.1034939Z panic: runtime error: invalid memory address or nil pointer dereference
2022-05-06T22:08:39.1035755Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xcc9845]
2022-05-06T22:08:39.1036321Z
2022-05-06T22:08:39.1037020Z goroutine 46 [running]:
2022-05-06T22:08:39.1038442Z github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint.doBaseFlattening(0xc000120400, 0xc00012f050, 0xc00039a5a0)
2022-05-06T22:08:39.1039437Z github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint/commons.go:119 +0x1e5
2022-05-06T22:08:39.1040359Z github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint.flattenServiceEndpointKubernetes(0xc000120400, 0xc00012f050, 0xc00039a5a0)
2022-05-06T22:08:39.1041334Z github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_kubernetes.go:250 +0x5d
2022-05-06T22:08:39.1042478Z github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint.genServiceEndpointReadFunc.func1(0xc000120400, 0xd6ec80, 0xc000140900, 0x156e470, 0xc000062800)
2022-05-06T22:08:39.1043382Z github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint/commons.go:222 +0x374
2022-05-06T22:08:39.1044251Z github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc0002d1ea0, 0x10507f8, 0xc000189280, 0xc000120400, 0xd6ec80, 0xc000140900, 0x0, 0x0, 0x0)
2022-05-06T22:08:39.1045050Z github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:346 +0x1ee
2022-05-06T22:08:39.1045956Z github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0002d1ea0, 0x10507f8, 0xc000189280, 0xc00031a820, 0xd6ec80, 0xc000140900, 0xc00000ee68, 0x0, 0x0, 0x0)
2022-05-06T22:08:39.1047072Z github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:635 +0x1cb
2022-05-06T22:08:39.1047897Z github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00000d830, 0x10507f8, 0xc000189280, 0xc0001892c0, 0xf2d265, 0x12, 0x0)
2022-05-06T22:08:39.1048656Z github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/grpc_provider.go:576 +0x47d
2022-05-06T22:08:39.1049621Z github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0xc0000e6400, 0x10508a0, 0xc000189280, 0xc0001a0240, 0x0, 0x0, 0x0)
2022-05-06T22:08:39.1050339Z github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/tf5server/server.go:553 +0x322
2022-05-06T22:08:39.1051482Z github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0xedbf60, 0xc0000e6400, 0x10508a0, 0xc000362210, 0xc0001a01e0, 0x0, 0x10508a0, 0xc000362210, 0xc0006e5000, 0xf74)
2022-05-06T22:08:39.1052535Z github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
2022-05-06T22:08:39.1053155Z google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001fafc0, 0x105d2b8, 0xc000612480, 0xc0000fc000, 0xc0003468d0, 0x1530c50, 0x0, 0x0, 0x0)
2022-05-06T22:08:39.1053633Z google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
2022-05-06T22:08:39.1054069Z google.golang.org/grpc.(*Server).handleStream(0xc0001fafc0, 0x105d2b8, 0xc000612480, 0xc0000fc000, 0x0)
2022-05-06T22:08:39.1054484Z google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
2022-05-06T22:08:39.1054939Z google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000038340, 0xc0001fafc0, 0x105d2b8, 0xc000612480, 0xc0000fc000)
2022-05-06T22:08:39.1055377Z google.golang.org/grpc@v1.32.0/server.go:859 +0xab
2022-05-06T22:08:39.1055827Z created by google.golang.org/grpc.(*Server).serveStreams.func1
2022-05-06T22:08:39.1056334Z google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd
2022-05-06T22:08:39.1056808Z
2022-05-06T22:08:39.1057400Z Error: The terraform-provider-azuredevops_v0.2.1 plugin crashed!
2022-05-06T22:08:39.1057591Z
2022-05-06T22:08:39.1057894Z This is always indicative of a bug within the plugin. It would be immensely
2022-05-06T22:08:39.1058466Z helpful if you could report the crash with the plugin's maintainers so that it
2022-05-06T22:08:39.1059028Z can be fixed. The output above should help diagnose the issue.
2022-05-06T22:08:39.1059207Z
2022-05-06T22:08:39.1139531Z ##[error]Bash exited with code '1'.
2022-05-06T22:08:39.1156998Z ##[section]Finishing: >> terraform plan stg@guhcampos Can you help check if service returns the serviceEndpointProjectReferences in the response for the service endpoint?
API:
https://docs.microsoft.com/en-us/rest/api/azure/devops/serviceendpoint/endpoints/get?view=azure-devops-rest-6.0 or
curl --user basic:<PAT> https://dev.azure.com/{organization}/{project}/_apis/serviceendpoint/endpoints/{endpointId}?api-version=6.0
guhcampos
commented
2 years ago I don't understand: what should I use as {endpointId} if I'm creating a new Service Endpoint?
Moreover, this is quite obviously a bug, a user facing application cannot possibly crash with a SIGSEGV. If you are trying to access a struct property that may not be there, this access must be guarded with the proper error handling.
Still, as a workaround: what permissions are required for this to execute correctly?
xuzhang3
commented
2 years ago @guhcampos Not creating service endpoints. Get the created service endpoint and check the response body if serviceEndpointProjectReferences existed.
Yes, permission is required.
luismesa-io
commented
7 months ago I'm getting this same error with azuredevops 1.0.1
In my local machine:
I created the service endpoint a while back using CLI, and it's the same endpoint used for terraform azurerm in the pipeline:
az devops service-endpoint azurerm create --azure-rm-service-principal-id $SERVICE_PRINCIPAL_APP_ID --azure-rm-subscription-id $SUBSCRIPTION --azure-rm-subscription-name $SUBSCRIPTION_NAME --azure-rm-tenant-id $SERVICE_PRINCIPAL_TENANT_ID --name $SERVICE_PRINCIPAL_DISPLAY_NAME --org "https://dev.azure.com/$ORGANIZATION_NAME/" --project $PROJECT_NAME
I created a tf file for this endpoint.
I then imported it using:
terraform import -var-file="./config.azurerm.tfvars" azuredevops_serviceendpoint_azurerm.$ORGANIZATION_NAME-service-endpoint "$AZURE_DEVOPS_SERVICE_ENDPOINT_TERRAFORM_ID"
I did a terraform plan and no changes were detected, I did a terraform apply and got:
`No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.`
I pushed all this to the repo where the pipeline runs.
The pipeline fails on terraform plan with:
` │ │ with azuredevops_serviceendpoint_azurerm.luismesa-io-service-endpoint, │ on luismesa-io-service-endpoint.tf line 1, in resource "azuredevops_serviceendpoint_azurerm" "luismesa-io-service-endpoint": │ 1: resource "azuredevops_serviceendpoint_azurerm" "luismesa-io-service-endpoint" { │ │ The plugin.(*GRPCProvider).ReadResource request was cancelled. ╵
Stack trace from the terraform-provider-azuredevops_v1.0.1 plugin:
panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xca202e]
goroutine 43 [running]: github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint.resourceServiceEndpointAzureRMRead(0x0?, {0xd61b20?, 0xc0006fe580}) github.com/microsoft/terraform-provider-azuredevops/azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_azurerm.go:230 +0x10e github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(Resource).read(0x10966a0?, {0x10966a0?, 0xc00007f6b0?}, 0xd?, {0xd61b20?, 0xc0006fe580?}) github.com/hashicorp/terraform-plugin-sdk/v2@v2.23.0/helper/schema/resource.go:712 +0x178 github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(Resource).RefreshWithoutUpgrade(0xc0003bca80, {0x10966a0, 0xc00007f6b0}, 0xc000399860, {0xd61b20, 0xc0006fe580}) github.com/hashicorp/terraform-plugin-sdk/v2@v2.23.0/helper/schema/resource.go:1015 +0x585 github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(GRPCProviderServer).ReadResource(0xc00037a768, {0x10966a0?, 0xc00007f590?}, 0xc0004dcf80) github.com/hashicorp/terraform-plugin-sdk/v2@v2.23.0/helper/schema/grpc_provider.go:613 +0x497 github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(server).ReadResource(0xc00033f180, {0x10966a0?, 0xc00007edb0?}, 0xc000668600) github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/tf5server/server.go:748 +0x49e github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0xedf280?, 0xc00033f180}, {0x10966a0, 0xc00007edb0}, 0xc00014a3f0, 0x0) github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:349 +0x170 google.golang.org/grpc.(Server).processUnaryRPC(0xc0001e01e0, {0x1099400, 0xc000413380}, 0xc0002fa6c0, 0xc000404810, 0x16242d0, 0x0) google.golang.org/grpc@v1.56.3/server.go:1335 +0xde3 google.golang.org/grpc.(Server).handleStream(0xc0001e01e0, {0x1099400, 0xc000413380}, 0xc0002fa6c0, 0x0) google.golang.org/grpc@v1.56.3/server.go:1712 +0xa1b google.golang.org/grpc.(Server).serveStreams.func1.1() google.golang.org/grpc@v1.56.3/server.go:947 +0xca created by google.golang.org/grpc.(Server).serveStreams.func1 google.golang.org/grpc@v1.56.3/server.go:958 +0x15c
Error: The terraform-provider-azuredevops_v1.0.1 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely helpful if you could report the crash with the plugin's maintainers so that it can be fixed. The output above should help diagnose the issue. `
Notes: The user with which I created this service endpoint and the azurerm service principal are different. The pipeline user is this exact terraform resource being queried when the panic happens.
The environment variables I'm passing to terraform plan are: env: ARM_CLIENT_ID: $(tf-arm-client-id) ARM_TENANT_ID: $(tf-arm-tenant-id) ARM_CLIENT_SECRET: $(tf-arm-client-secret) ARM_SUBSCRIPTION_ID: $(tf-arm-subscription-id) DOMAIN: $(tf-domain) AZDO_ORG_SERVICE_URL: $(System.TeamFoundationCollectionUri) AZDO_PERSONAL_ACCESS_TOKEN: $(System.AccessToken) TF_STATE_ACCOUNT_NAME: $(tf-state-account-name) TF_STATE_CONTAINER_NAME: $(tf-state-container-name) TF_STATE_FILE_KEY: $(tf-state-file-key) TF_STATE_ACCESS_KEY: $(tf-state-access-key)
xuzhang3
commented
7 months ago @luismesa-io System.AccessToken is different from the PAT used in your local dev environment. More details: https://registry.terraform.io/providers/microsoft/azuredevops/latest/docs/guides/authenticating_using_the_personal_access_token
luismesa-io
commented
7 months ago @xuzhang3 "is different", ok how?
The link you gave me seems to suggest it can be used: _
A job access token (System.AccessToken) is a security token that is dynamically generated by Azure Pipelines for each job at run time. The agent on which the job is running uses the job access token in order to access these resources in Azure DevOps.
_
If it can't be used, then what's the correct value to put there? Or how is it configured? This pipeline has access to everything.
xuzhang3
commented
7 months ago @luismesa-io can you create new PAT and set it to the environment where the pipeline runs?
luismesa-io
commented
6 months ago @xuzhang3 Hi, sorry for the late reply. I'm not looking to use PAT in the pipeline. I see you commented that we can now use Service Principals in the provider. I'll try that instead. Thanks again.
Community Note
Terraform (and Azure DevOps Provider) Version
Terraform v1.1.0 on darwin_amd64
Affected Resource(s)
azuredevops_serviceendpoint_github_enterpriseTerraform Configuration Files
Expected Behavior
terraform planshould succeed without any changes to apply as the state is checked into the same repository and was updated by a coworker using the same Terraform script earlier the same day.Actual Behavior
The azuredevops plugin crashes when running
terraform plan:Originally posted by @andreassiegel in https://github.com/microsoft/terraform-provider-azuredevops/issues/491#issuecomment-990109561