search

microsoft / azure-pipelines-terraform

Azure Pipelines tasks for installing Terraform and running Terraform commands in a build or release pipeline.
MIT License
95 stars 59 forks source link

two service connections but reads backend data incorrectly #189

Open mericstam opened 9 months ago

mericstam commented 9 months ago 
          > I'm having a problem as well. I have two service connections:

When I enable logging TF_LOG=TRACE in my pipeline, download the logs, I can see that the task resolves the wrong service connection for environmentServiceNameAzureRM. Here is a snippet of the plan prepare pipeline log:

2023-09-28T08:21:25.4591150Z ##[section]Starting: Prepare job Plan
2023-09-28T08:21:25.5216160Z Variables:
2023-09-28T08:21:25.5216160Z   ResourceServiceConnection: $[ variablegroups.test-env.ResourceServiceConnection ]
2023-09-28T08:21:25.5216160Z   TerraformBackendResourceGroupName: $[ variablegroups.test-env.TerraformBackendResourceGroupName ]
2023-09-28T08:21:25.5216160Z   TerraformBackendServiceConnection: $[ variablegroups.test-env.TerraformBackendServiceConnection ]
2023-09-28T08:21:25.5216160Z   TerraformBackendStorageAccountName: $[ variablegroups.test-env.TerraformBackendStorageAccountName ]

And the plan pipeline log:

2023-09-28T08:21:46.4775808Z ##[debug]/opt/hostedtoolcache/terraform/1.5.7/x64/terraform arg: init
2023-09-28T08:21:46.4778692Z ##[debug]backendServiceArm=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
2023-09-28T08:21:46.4782223Z ##[debug]backendServiceArm=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
2023-09-28T08:21:46.4783813Z ##[debug]aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa auth scheme = ServicePrincipal
2023-09-28T08:21:46.4784477Z ##[debug]Setting up backend for authorization scheme: serviceprincipal.
2023-09-28T08:21:46.4785355Z ##[debug]backendAzureRmStorageAccountName=icraterraforms

....
2023-09-28T08:21:55.8655848Z ##[debug]/opt/hostedtoolcache/terraform/1.5.7/x64/terraform arg: plan
2023-09-28T08:21:55.8656309Z ##[debug]/opt/hostedtoolcache/terraform/1.5.7/x64/terraform arg: -var-file=environment.tfvars -out=infrastructure.tfplan -detailed-exitcode
2023-09-28T08:21:55.8656788Z ##[debug]environmentServiceNameAzureRM=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
2023-09-28T08:21:55.8657144Z ##[debug]aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa auth scheme = ServicePrincipal
2023-09-28T08:21:55.8657489Z ##[debug]Setting up provider for authorization scheme: serviceprincipal.
2023-09-28T08:21:55.8657869Z ##[debug]aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa data subscriptionid = aaaaaaaa-aaaa-aaaa-aaaa-bbbbbbbbbbbb
2023-09-28T08:21:55.8658258Z ##[debug]aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa data subscriptionid = aaaaaaaa-aaaa-aaaa-aaaa-bbbbbbbbbbbb
2023-09-28T08:21:55.8658652Z ##[debug]aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa auth param tenantid = xxx
2023-09-28T08:21:55.8659127Z ##[debug]aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa auth param serviceprincipalid = ***
2023-09-28T08:21:55.8659612Z ##[debug]aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa auth param serviceprincipalkey = ***
2023-09-28T08:21:55.8659981Z ##[debug]Finished up provider for authorization scheme: serviceprincipal.

I expected to see: environmentServiceNameAzureRM=ffffffff-ffff-ffff-ffff-ffffffffffff

This is a snippet of the pipeline:

...
stages:
- stage: DeployTest
  variables:
  - group: test-env
...
  jobs:
  - job: Plan
    steps:
    - task: TerraformTaskV4@4
      displayName: Terraform init
      inputs:
        command: init
      workingDirectory: $(System.DefaultWorkingDirectory)/infrastructure/applications
      backendAzureRmResourceGroupName: $(TerraformBackendResourceGroupName)
      backendAzureRmStorageAccountName: $(TerraformBackendStorageAccountName)
      backendAzureRmContainerName: ${{ parameters.TerraformAppName }}-${{ parameters.TerraformEnvironment }}
      backendAzureRmKey: ${{ parameters.TerraformAppName }}-${{ parameters.TerraformEnvironment }}.tfstate
      backendServiceArm: $(TerraformBackendServiceConnection)
      condition: succeeded()

    - task: TerraformTaskV4@4
      displayName: Terraform plan
      name: PlanTask
      inputs:
        command: plan
        workingDirectory: $(System.DefaultWorkingDirectory)/infrastructure/applications
        commandOptions: -var-file=environment.tfvars -out=infrastructure.tfplan
        environmentServiceNameAzureRM: $(ResourceServiceConnection)
      condition: succeeded()

I verified with another pipeline that all variables have been set correctly. I can rule out pipeline config issues. The limitation seems to be that environmentServiceNameAzureRM isn't used to pick up the tenant/subscription/service principal during a custom/plan/apply command.

Originally posted by @theo-albers in https://github.com/microsoft/azure-pipelines-terraform/issues/138#issuecomment-1738938210

mericstam commented 9 months ago

taking a look at this.

mericstam commented 9 months ago

@theo-albers

mericstam commented 9 months ago

Hi @theo-albers, I did have an issue in my setup and thought it was related, but my issue was I had two types of service connections. when I switched them to same type (Service Principal) it worked.. this is my setup:

 - task: TerraformTaskV4@4
      displayName: 'terraform init'
      inputs:
        provider: 'azurerm'
        command: 'init'
        backendServiceArm: 'spMSDN2'
        backendAzureRmResourceGroupName: 'terraform'
        backendAzureRmStorageAccountName: 'solstatestore2'
        backendAzureRmContainerName: 'state'
        backendAzureRmKey: 'state.tfstate'

- task: TerraformTaskV4@4
      name: plan
      displayName: 'terraform plan'
      inputs:
        commandOptions: '-out=tfplan -detailed-exitcode'
        provider: 'azurerm'
        command: 'plan'
        environmentServiceNameAzureRM: 'TerraformSP'

parts of debug log

for the init
....
##[debug]found: '/opt/hostedtoolcache/terraform/1.5.7/x64/terraform'
##[debug]which '/opt/hostedtoolcache/terraform/1.5.7/x64/terraform'
##[debug]found: '/opt/hostedtoolcache/terraform/1.5.7/x64/terraform'
##[debug]/opt/hostedtoolcache/terraform/1.5.7/x64/terraform arg: init
##[debug]backendServiceArm=5cd4e6b5-xxxx-479e-beed-xxxxxxxxx
##[debug]backendServiceArm=5cd4e6b5-xxxx-479e-beed-xxxxxxxxx
##[debug]5cd4e6b5-xxxx-479e-beed-xxxxxxxxx auth scheme = ServicePrincipal
##[debug]Setting up backend for authorization scheme: serviceprincipal.
##[debug]backendAzureRmStorageAccountName=solstatestore2
##[debug]backendAzureRmContainerName=state
##[debug]backendAzureRmKey=state.tfstate
##[debug]backendAzureRmResourceGroupName=terraform
.....

for the plan
....
##[debug]environmentServiceNameAzureRM=1c2ad8ed-xxxx-4d31-91b1-xxxxxxxxx
##[debug]1c2ad8ed-xxxx-4d31-91b1-xxxxxxxxx auth scheme = ServicePrincipal
##[debug]Setting up provider for authorization scheme: serviceprincipal.
...
...

##[debug]1c2ad8ed-xxxx-4d31-91b1-xxxxxxxxx auth param serviceprincipalid = ***
##[debug]1c2ad8ed-xxxx-4d31-91b1-xxxxxxxxx auth param serviceprincipalkey = ***
##[debug]Finished up provider for authorization scheme: serviceprincipal.
##[debug]exec tool: /opt/hostedtoolcache/terraform/1.5.7/x64/terraform
##[debug]arguments:
##[debug]   plan
##[debug]   -out=tfplan
##[debug]   -var
##[debug]   test=test
##[debug]   -detailed-exitcode

As a test can you hardcode the names of your service connections so you can rule out any variable configuration?

theo-albers commented 9 months ago

Hi @theo-albers, I did have an issue in my setup and thought it was related, but my issue was I had two types of service connections. when I switched them to same type (Service Principal) it worked.. this is my setup:

 - task: TerraformTaskV4@4
      displayName: 'terraform init'
      inputs:
        provider: 'azurerm'
        command: 'init'
        backendServiceArm: 'spMSDN2'
        backendAzureRmResourceGroupName: 'terraform'
        backendAzureRmStorageAccountName: 'solstatestore2'
        backendAzureRmContainerName: 'state'
        backendAzureRmKey: 'state.tfstate'

- task: TerraformTaskV4@4
      name: plan
      displayName: 'terraform plan'
      inputs:
        commandOptions: '-out=tfplan -detailed-exitcode'
        provider: 'azurerm'
        command: 'plan'
        environmentServiceNameAzureRM: 'TerraformSP'

parts of debug log

for the init
....
##[debug]found: '/opt/hostedtoolcache/terraform/1.5.7/x64/terraform'
##[debug]which '/opt/hostedtoolcache/terraform/1.5.7/x64/terraform'
##[debug]found: '/opt/hostedtoolcache/terraform/1.5.7/x64/terraform'
##[debug]/opt/hostedtoolcache/terraform/1.5.7/x64/terraform arg: init
##[debug]backendServiceArm=5cd4e6b5-xxxx-479e-beed-xxxxxxxxx
##[debug]backendServiceArm=5cd4e6b5-xxxx-479e-beed-xxxxxxxxx
##[debug]5cd4e6b5-xxxx-479e-beed-xxxxxxxxx auth scheme = ServicePrincipal
##[debug]Setting up backend for authorization scheme: serviceprincipal.
##[debug]backendAzureRmStorageAccountName=solstatestore2
##[debug]backendAzureRmContainerName=state
##[debug]backendAzureRmKey=state.tfstate
##[debug]backendAzureRmResourceGroupName=terraform
.....

for the plan
....
##[debug]environmentServiceNameAzureRM=1c2ad8ed-xxxx-4d31-91b1-xxxxxxxxx
##[debug]1c2ad8ed-xxxx-4d31-91b1-xxxxxxxxx auth scheme = ServicePrincipal
##[debug]Setting up provider for authorization scheme: serviceprincipal.
...
...

##[debug]1c2ad8ed-xxxx-4d31-91b1-xxxxxxxxx auth param serviceprincipalid = ***
##[debug]1c2ad8ed-xxxx-4d31-91b1-xxxxxxxxx auth param serviceprincipalkey = ***
##[debug]Finished up provider for authorization scheme: serviceprincipal.
##[debug]exec tool: /opt/hostedtoolcache/terraform/1.5.7/x64/terraform
##[debug]arguments:
##[debug]   plan
##[debug]   -out=tfplan
##[debug]   -var
##[debug]   test=test
##[debug]   -detailed-exitcode

As a test can you hardcode the names of your service connections so you can rule out any variable configuration?

I already verified the variable replacement via a temp debug pipeline. All variables were replaced properly and the service connections worked fine as well using an Azure Powershell task as test. Both service connections are Azure Resource Manager automatic connections with only the subscription filled in.

image

theo-albers commented 9 months ago

I have a Powershell script where I call terraform directly, locally. I added a secret to both App Registrations the service connections point to. When I do this, my Powershell script calling terraform init and plan works fine. For init I set the background SP and for plan the deployment SP. This confirms to me the issue is related to the pipeline task. 

terraform init `
  -var-file="$($TerraformVariablesFile)" `
  -backend-config="tenant_id=$($WorkspaceCredentials.TenantId)" `
  -backend-config="subscription_id=$($WorkspaceCredentials.SubscriptionId)" `
  -backend-config="client_id=$($WorkspaceCredentials.ClientId)" `
  -backend-config="client_secret=$($WorkspaceCredentials.ClientSecret)" `
  -backend-config="resource_group_name=$($TerraformResourceGroupName)" `
  -backend-config="storage_account_name=$($TerraformStorageAccountName)" `
  -backend-config="container_name=$($TerraformContainerName)" `
  -backend-config="key=$($TerraformStateFile)" 

$env:ARM_TENANT_ID       = $DeploymentCredentials.TenantId
$env:ARM_SUBSCRIPTION_ID = $DeploymentCredentials.SubscriptionId
$env:ARM_CLIENT_ID       = $DeploymentCredentials.ClientId
$env:ARM_CLIENT_SECRET   = $DeploymentCredentials.ClientSecret                  
terraform plan -var-file="$($TerraformVariablesFile)" -out="$($TerraformPlanFile)" -detailed-exitcode *>&1 | Out-Host
mericstam commented 9 months ago

I cannot replicate the issue you are seeing with the same id for both backend and environment service connections.

I will see if I can replicate your setup better. you have two subscriptions with the same tenant, correct?

br Manuel

theo-albers commented 9 months ago

I cannot replicate the issue you are seeing with the same id for both backend and environment service connections.

I will see if I can replicate your setup better. you have two subscriptions with the same tenant, correct?

br Manuel

Correct. The setup:

I don't see in this repo where the environment principal is resolved. I only see that it uses the property to detect the authentication schema. I don't see it actually uses the environment service connection to resolve the client/secret, but maybe I'm wrong. I also see no sign in log for this service connection in Azure DevOps under the service connection.

We are using a yaml pipeline, not a classic pipeline. Maybe your tests assume variables to be set by the pipeline infrastructure and maybe that's not the case with the yaml pipeline. Could there be a difference between classic and yaml pipeline?

mericstam commented 8 months ago

Hi, Unfortunately I can not replicate your setup as I do not have a way to create two subs in my tenants. (one is Visual studio subscription and the other I do not have high enough rights to create a new subscription)

I also use YAML pipelines. the two differences I see in my setup: 1. I have two tenants with one subscription in each. 2. I don't use variables for service connection names.

backendServiceArm: 'spMSDN2' and environmentServiceNameAzureRM: 'TerraformSP'

theo-albers commented 8 months ago

For me it's no longer an issue. I took the easy way out and I am targeting a single subscription.

rruenroeng commented 8 months ago

Hi there!

I was seeing a similar problem on a separate thread. I would really like to target a separate subscription. It is important for my organization to have our sensitive information (the tf state) on a different subscription that we can really limit access to.
I've taken the easy way out as a stopgap, but this is still an issue I'd love to see resolved.

theo-albers commented 8 months ago

Hi, Unfortunately I can not replicate your setup as I do not have a way to create two subs in my tenants. (one is Visual studio subscription and the other I do not have high enough rights to create a new subscription)

I also use YAML pipelines. the two differences I see in my setup: 1. I have two tenants with one subscription in each. 2. I don't use variables for service connection names.

backendServiceArm: 'spMSDN2' and environmentServiceNameAzureRM: 'TerraformSP'

When I look at the source code in this repo I don't see that the subscription and tenant id is being set based on environmentServiceNameAzureRM. Can you at least verify this, just by debugging the task? Where does it take the connection information from environmentServiceNameAzureRM?

theo-albers commented 8 months ago

We can confirm that the issue occurs when you have two subscriptions in a single tenant. Tenant A with subscription A for state and subscription B for deployment fails. It works when you have tenant A with subscription A for state and tenant B with subscription B for deployment.

We have a pipeline with the scenario that fails and one with the scenario that succeeds. Both pipelines use the same tasks. The failing pipeline succeeds for the stage where state and deployment use the same subscription.

mericstam commented 8 months ago

Hi, Unfortunately I can not replicate your setup as I do not have a way to create two subs in my tenants. (one is Visual studio subscription and the other I do not have high enough rights to create a new subscription) I also use YAML pipelines. the two differences I see in my setup: 1. I have two tenants with one subscription in each. 2. I don't use variables for service connection names. backendServiceArm: 'spMSDN2' and environmentServiceNameAzureRM: 'TerraformSP'

When I look at the source code in this repo I don't see that the subscription and tenant id is being set based on environmentServiceNameAzureRM. Can you at least verify this, just by debugging the task? Where does it take the connection information from environmentServiceNameAzureRM?

it is done here: https://github.com/microsoft/azure-pipelines-terraform/blob/ff6cc825dbaf72c902d7e198ef8ee7a9b604e267/Tasks/TerraformTask/TerraformTaskV4/src/azure-terraform-command-handler.ts#L62C42-L62C82

theo-albers commented 8 months ago

Yes, when I follow the flow, it should indeed work as intended. Strange....

image

How do you debug pipeline tasks? When I read this https://learn.microsoft.com/en-us/azure/devops/extend/develop/add-build-task?view=azure-devops I guess you are basically left with the mocha tests.

mericstam commented 8 months ago

Hi, it is pretty complex to debug but possible, however I have not been successful. I use console.log() a lot in the not public dev version of the task.

from the MS Learn docs https://learn.microsoft.com/en-us/azure/devops/extend/develop/add-build-task?view=azure-devops#run-the-task

docs from azure-pipelines-tasks https://github.com/microsoft/azure-pipelines-tasks/blob/master/docs/debugging.md#debugging-typescript-tasks-in-vs-code

not sure below works for bulid extensions, but should work for UI stuff. using hot reload https://github.com/microsoft/azure-devops-extension-hot-reload-and-debug