Open Soruk opened 3 years ago
And I see the same behavior with the AzureResourceManagerTemplateDeployment task.
Maybe there is a problem with deployment jobs.
I am experiencing exactly the same problem. Adding a
- script: echo ${{parameters.SubscriptionName}}
step, correctly prints the name of the subscription in the logs.
I am not able to repro this issue. Let me know If I am following correct steps.
SubscriptionName
jobs:
- template: samplecheck.yml
parameters:
SubscriptionName: $(SubscriptionName)
parameters:
- name: SubscriptionName
default: ''
jobs:
- job: DeploymentJob2
pool:
name: Azure Pipelines
vmImage: 'windows-latest'
steps:
- checkout: none
- task: AzurePowerShell@5
inputs:
azureSubscription: ${{parameters.SubscriptionName}}
ScriptType: 'InlineScript'
Inline: 'echo ${{parameters.SubscriptionName}}'
azurePowerShellVersion: 'LatestVersion'
It ran successfully
Hi @AmrutaKawade
As I said that previously, it works in normal job but no in a deployment job.
parameters:
- name: SubscriptionName
default: ''
jobs:
- deployment: DeploymentJob2
pool:
name: Azure Pipelines
vmImage: 'windows-latest'
environment: MyEnv
strategy:
runOnce:
deploy:
steps:
- checkout: none
- task: AzurePowerShell@5
inputs:
azureSubscription: ${{parameters.SubscriptionName}}
ScriptType: 'InlineScript'
Inline: 'echo ${{parameters.SubscriptionName}}'
azurePowerShellVersion: 'LatestVersion'
I was able to repro this when I added this variable here
My main yaml looks like this
variables:
- name: SubscriptionName
value: $(EnvironmentSubscriptionName)
jobs:
- template: samplecheck.yml
parameters:
SubscriptionName: $(SubscriptionName)
and template yml like this
parameters:
- name: SubscriptionName
default: ''
type: string
jobs:
- deployment: Dev
pool:
name: Azure Pipelines
vmImage: 'windows-latest'
environment: MyEnv
strategy:
runOnce:
deploy:
steps:
- checkout: none
- task: AzurePowerShell@5
inputs:
azureSubscription: ${{parameters.SubscriptionName}}
ScriptType: 'InlineScript'
Inline: 'echo ${{parameters.SubscriptionName}}'
azurePowerShellVersion: 'LatestVersion'
but it worked when I passed value directly in main.yaml like this
variables:
- name: SubscriptionName
value: 'abc'
jobs:
- template: samplecheck.yml
parameters:
SubscriptionName: $(SubscriptionName)
Can you try this approach and let me know if it works.
Hi @AmrutaKawade
It still is not working for me.
There was a resource authorization issue: "The pipeline is not valid. Job Dev: Step input azureSubscription references service connection $(EnvironmentSubscriptionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
Here is my template main.yml:
parameters:
- name: EnvironmentType
displayName: 'Environment Type'
type: string
default: Dev
values:
- Dev
- Test
- Prod
trigger: none
#- main
name: $(TeamProject)_$(Build.DefinitionName)_$(SourceBranchName)_$(Date:yyyyMMdd)$(Rev:.r)
variables:
- group: AzTestEnvironment
- name: EnvironmentType
value: ${{ lower(parameters.EnvironmentType) }}
- name: SubscriptionName
${{ if ne(lower(variables['EnvironmentType']), 'prod') }}:
value: $(EnvironmentSubscriptionName)
${{ if eq(lower(variables['EnvironmentType']), 'prod') }}:
value: $(EnvironmentSubscriptionNameProd)
stages:
- stage: 'TestStage'
displayName: 'Stage for testing the Azure powerShell Deployment'
jobs:
- template: template.yml
parameters:
SubscriptionName: $(SubscriptionName)
And the sub template.yml:
parameters:
- name: SubscriptionName
default: ''
type: string
jobs:
- deployment: Dev
pool:
vmImage: 'ubuntu-latest'
environment: MyEnv
strategy:
runOnce:
deploy:
steps:
- checkout: none
- task: AzurePowerShell@5
inputs:
azureSubscription: ${{parameters.SubscriptionName}}
ScriptType: 'InlineScript'
Inline: 'echo ${{parameters.SubscriptionName}}'
azurePowerShellVersion: 'LatestVersion'
The variable group 'AzTestEnvironment' contains:
I want to choose the subscription by choosing the given runtime parameter.
What i am suggesting is change below part of your pipeline
- name: SubscriptionName
${{ if ne(lower(variables['EnvironmentType']), 'prod') }}:
value: $(EnvironmentSubscriptionName)
${{ if eq(lower(variables['EnvironmentType']), 'prod') }}:
value: $(EnvironmentSubscriptionNameProd)
like this
- name: SubscriptionName
${{ if ne(lower(variables['EnvironmentType']), 'prod') }}:
value: 'SubPAYGDevTest'
${{ if eq(lower(variables['EnvironmentType']), 'prod') }}:
value: 'SubPAYG'
Hi,
Yes, it will be workaround.
But it is still a bug: we cannot use the variables from the library to choose subscription.
And therefore we cannot use the same template across multiple pipelines that deploys to different subscriptions that are set on given Azure DevOps projects.
Sorry its not a bug you are using variables inside variables. Currently its not possible to do that.
If you want generic pipeline you can change your main yml like this
parameters:
- name: EnvironmentType
displayName: 'Environment Type'
type: string
default: Dev
values:
- Dev
- Test
- Prod
trigger: none
#- main
name: $(TeamProject)_$(Build.DefinitionName)_$(SourceBranchName)_$(Date:yyyyMMdd)$(Rev:.r)
stages:
- stage: 'TestStage'
displayName: 'Stage for testing the Azure powerShell Deployment'
jobs:
- ${{ if ne(lower(parameters.EnvironmentType), 'prod') }}:
- template: template.yml
parameters:
SubscriptionName: $(EnvironmentSubscriptionName)
- ${{ if eq(lower(parameters.EnvironmentType), 'prod') }}:
- template: template.yml
parameters:
SubscriptionName: $(EnvironmentSubscriptionNameProd)
As I said it works for normal jobs, but not for the deployment jobs. And therefore in my opinion it is a bug.
For the workaround, I use use similar approach with the conditional insertion to chose right subscription but in each sub template. I will try your global approach.
I believe it is a bug as well as variable scoping is not being respected. https://github.com/microsoft/azure-pipelines-tasks/issues/12412 Same issue with the task AzureResourceManagerTemplateDeployment@3
Hello is anyone going to look at this issue it is a real blocker. I'm not sure why it was removed from bugs, @chshrikh without any reasoning.
Judging by this ticket here it is an issue that many people are facing https://developercommunity.visualstudio.com/t/using-a-variable-for-the-service-connection-result/676259?from=email&viewtype=all#T-ND1412705 Since 2019
If it helps for anyone else running into this issue, I did a dirty hack to get this working by implementing this workaround in a generic way. I run this code here to replace the subscription passed in the with the correct subscription
- ${{ each step in parameters.StepList }}:
- ${{ each pair in step }}:
${{ if ne(pair.key, 'parameters') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.key, 'parameters') }}:
parameters:
${{ each input in pair.value }}:
${{ if eq(input.key, 'AzureSubscription') }}:
AzureSubscription: ${{ parameters.stage.azuresubscription }}
${{ if ne(input.key, 'AzureSubscription') }}:
${{ input.key }}: ${{ input.value }}
This is an issue as well for us. What is a simple way to utilize a variable in a -job does not work in a -deployment
Will have to use a work-around approach. This 100% is a bug.
The simple fact that these tasks like AzureFunctionApp cannot accept a variable for input on the param azureSubscription, but can for the appName... is simply a defect.
Running into this exact same issue. I'm trying to set the value for azureResourceManagerConnection
using the matrix strategy because I need to test the same steps for multiple subscriptions but my runs fail because we can't do that. Assigning a global variable is also not a decent hack because I would need to hardcode in multiple tasks in the same pipeline which is a big pain to manager.
@AmrutaKawade / @nadesu any estimated ETA for the fix for this?
Why can't I use variables to resolve service connection / environment names? Resources are authorized before a stage can start running, so stage- and job-level variables aren't available. Azure doc process-the-pipeline
seems to be the unterlying issue.
Why can't I use variables to resolve service connection / environment names? Resources are authorized before a stage can start running, so stage- and job-level variables aren't available. Azure doc process-the-pipeline
seems to be the unterlying issue.
Agreed, that may state that you can't utilize vars, then the question (which really is the root of what everyone is asking here) what is the correct approach for handling the scenarios many of us have, with this limitation.
would like this feature too
I have the same issue with the HelmDeploy@0 task. Please resolve this Microsoft.
ran into this myself, any update? i get that deploy jobs have to do the resource checks ahead of time, but this is really annoying that it isn't supported even for static variables
Ran into the same issue and I also understand that it is due to checks but I believe there are better ways to do this. The behaviour is just weird
- task: AzureRmWebAppDeployment@4
inputs:
ConnectionType: '$(connectionType)'
azureSubscription: '$(azureSubscription)'
appType: '$(appType)'
WebAppName: '$(appName)'
From the above snippet, my connectionType is expanded based on the variable group within the yaml template but same can't be said of the azureSubscription. To confirm that the issue was specific to azureSubscription inputs, I passed in the value directly and it worked . All other variables from the variable group was expanded except azureSubscription
This is a major bug. The problem with this task is that it is inconsistent with other tasks and I have just spent hours of my time, wasted, to find out it is by design! I can pass in the azure subscription name as a variable to almost any task except this one! This is entirely inconsistent with other tasks and other input behaviors. Please fix this!
anyone have a viable workaround, the proposal by @AmrutaKawade doesn't work since you can ONLY use static variables in a template expression...if i knew the value based on a static variable i could just set the connection name in the first place !
here's what i ended up doing, yuck
- name: serviceConnection
value: $[replace(replace(eq(variables['environment'],'prod'), 'True', 'prod-connection'), 'False', 'other-connection')]
this is outside the stage. you could use in() instead of eq if you wanted to match multiple. Also instead of other-connection you could do yet another chain like this
- name: serviceConnection
value: $[replace(replace(replace(eq(variables['environment'],'prod'), 'True', 'prod-connection'), 'False', replace(eq(variables['environment'],'stage'), 'True', 'stage-connection'), 'False', 'other-connection-not-prod-or-stage')]
i feel dirty now and need to take a shower
This is still an issue. But i have a workaround that is quite simple. use a parameter inside the template that does not have a default.
The problem If you provide the azureSubscription input in an AzureRmWebAppDeployment@4 that is used inside a deployment job inside a template, passing it as a parameter it fails. it does not expand the parameter.
However it is a bit more specific. Parameters that have no default are expanded correctly. so in a template.yml file:
parameters:
subscription:
does work when using this parameter in the AzureRmWebAppDeployment@4 azuresubscription input field. But what fails is if it is passed with a default value:
parameters:
- subscription: environment
type: string
default: 'default'
the error message when saving it in to git from the edit pipeline screen results in a message saying that the azureSubscription 'default' does not exist.
Can this please be fixed? All other parameters within the - deployment:
job are expanded correctly but for the azureSubscription input of the AzureRmWebAppDeployment@4 it fails.
I think there is a validation that checks if a valid azureSubscription is used. But when checking this it takes the default if available instead of the expanded parameter.
Here's my code extract, that works for me. It might not be the best, but perhaps will help someone out there...
Parameterizing the azureSubscription
value, and passing into a template...
# azure-pipeline.yml
stages:
- stage: DEV
jobs:
- job: build_image
timeoutInMinutes: 120
variables:
environment: dev
gallery_name: mygallery01${{ variables.environment }}
${{ if eq( variables.environment, 'dev') }}:
azureServiceConnection: MYDEVCONNECTIONNAME
${{ if eq( variables.environment, 'prd') }}:
azureServiceConnection: MYPRODCONNECTIONNAME
steps:
- template: template_build_image.yml
parameters:
environment: ${{ variables.environment }}
azureServiceConnection: ${{ variables.azureServiceConnection }}
galleryName: $(gallery_name)
And then in the template, I have this:
# template_build_image.yml
parameters:
- name: environment
type: string
- name: azureServiceConnection
type: string
- name: galleryName
type: string
steps:
- checkout: self
- task: AzurePowerShell@5
inputs:
azureSubscription: '${{ parameters.azureServiceConnection }}'
ScriptPath: '$(System.DefaultWorkingDirectory)/scripts/BuildImage.ps1'
ScriptArguments: '-environment ${{ parameters.environment }} `
-galleryName ${{ parameters.galleryName }} `
-version $(Build.BuildNumber)'
azurePowerShellVersion: LatestVersion
pwsh: true
workingDirectory: '$(System.DefaultWorkingDirectory)'
Regards Darren
Wrong manipulation....
Same problem +1.
Is this going to be solved/implemented? I am running a matrix pipeline and the "azureResourceManagerConnection" doesn't support variables in the AzureResourceManagerTemplateDeployment@3 task.
This is something not working as expected for now, refer variables from Library in azureSubscription for deployment is not resolved properly.
It looks to me once the pipeline run kicks in, it's not passing the checking/validation on the ServiceConnection, and referring "$(ServiceConnection) which could not be found. The service connection does not exist"
By looking at it, the "checking/validation" is not resolving the variables fully, but I could be wrong.
Sorry Azure DevOps team but how can you say this is not a bug. It's been in backlog for 3 years and no sign of it being fixed. This renders proper pipeline templating USELESS.
~ THREE YEARS OLD AND STILL NOT FIXED ~
Seriously guys. This is crazy. Please fix this!
Sorry Azure DevOps team but how can you say this is not a bug. It's been in backlog for 3 years and no sign of it being fixed. This renders proper pipeline templating USELESS.
If they don't think it is a bug then at least give some guidance on HOW to parameterize the service connection name. How do you use reusable workflows/templates. How? Please explain guys.
@ForteUnited we now paramerize the service connection names - one is unable to pass it as a variable.. but can make it a parameter... we choose to do this within the environment parameter - giving us the ability to easily switch service connections for different environments - example.. within my root pipeline i'll have something like
extends: template: /SomePath/My_template.yml parameters: environments:
environment: PRODUCTION variableGroup: Variables-PROD AzureServiceConnection: 'Production_Tenant'
And in the templates I refer to is as so: azureSubscription: ${{ environment.AzureServiceConnection }}
Pasting my yaml here ruined all the indenting but you can get the general idea.
How is this still not fixed, and we still have to use workarounds?
@AmrutaKawade @PhilipsonJoseph
Please reclassify this as a BUG.
This still does not work inside a deployment job inside a template, passing it as a parameter... it fails.
Passing as a parameter works for me.
parameters:
- name: ServiceConnection
- deployment: "DeployApi"
environment: env-${{parameters.Site}}-${{parameters.Env}}
variables:
appName: "${{ parameters.Site }}-${{ parameters.Env }}-api"
strategy:
runOnce:
deploy:
steps:
- task: AzureRmWebAppDeployment@4
inputs:
ConnectionType: "AzureRM"
azureSubscription: ${{parameters.ServiceConnection}}
appType: "webAppLinux"
WebAppName: $(appName)
packageForLinux: "$(Agent.BuildDirectory)/WebApplication1"
Hi @khatran79
You are using the other task type "AzureRmWebAppDeploymenbt@4" and the problem is in the "AzurePowerShell@5" task type.
Try to publish your web app by Azure PowerShell Module and you will get this error.
AzurePowerShell@5 is working
parameters:
- name: ServiceConnection
stages:
- stage: IOC${{parameters.Site}}${{parameters.Env}}
dependsOn: [Build]
jobs:
- deployment: "DeployResources"
environment: env-${{parameters.Site}}-${{parameters.Env}}
strategy:
runOnce:
deploy:
steps:
- task: AzurePowerShell@5
inputs:
azureSubscription: ${{parameters.ServiceConnection}}
ScriptType: "InlineScript"
azurePowerShellVersion: "LatestVersion"
Inline: |
# You can write your azure powershell scripts inline here.
# You can also pass predefined and custom variables to this script using arguments
Write-Output ${{parameters.ServiceConnection}}
Hi @khatran79
As this issue was opened almost 2 years ago, I had to rebuild my test environment and I still have this issue. In your test you do not use Group Variables where subscriptions names are declared and later used in the YAML Pipeline.
Here is my main template azure-pipelines-main.yml
:
parameters:
- name: EnvironmentType
displayName: 'Environment Type'
type: string
default: Dev
values:
- Dev
- Gold
- Silver
trigger: none
#- main
name: $(TeamProject)_$(Build.DefinitionName)_$(SourceBranchName)_$(Date:yyyyMMdd)$(Rev:.r)
variables:
- group: AzTestEnvironment
- name: EnvironmentType
value: ${{ lower(parameters.EnvironmentType) }}
- name: SubscriptionName
${{ if eq(lower(variables['EnvironmentType']), 'dev') }}:
value: $(EnvironmentSubscriptionName)
${{ elseif eq(lower(variables['EnvironmentType']), 'gold') }}:
value: $(EnvironmentSubscriptionNameGold)
${{ elseif eq(lower(variables['EnvironmentType']), 'silver') }}:
value: $(EnvironmentSubscriptionNameSilver)
stages:
- stage: 'TestStage'
displayName: 'Stage for testing the Azure powerShell Deployment'
jobs:
- template: templates/azure-pipelines-ps.yml
parameters:
SubscriptionName: $(SubscriptionName)
And here is the referenced template azure-pipelines-ps.yml
:
parameters:
- name: SubscriptionName
default: ''
jobs:
- deployment: DeploymentPS
pool:
name: Azure Pipelines
vmImage: 'windows-latest'
environment: MyEnv
strategy:
runOnce:
deploy:
steps:
- checkout: none
- task: AzurePowerShell@5
inputs:
azureSubscription: ${{parameters.SubscriptionName}}
ScriptType: 'InlineScript'
Inline: 'echo ${{parameters.SubscriptionName}}'
azurePowerShellVersion: 'LatestVersion'
The variable group AzTestEnvironment
contains the following values:
And these values correspond to the name of service connections defined at the project level.
And I declared an empty environment MyEnv
.
And the execution of the pipeline fails whatever parameter of the EnvironmentType I choose:
There was a resource authorization issue: "The pipeline is not valid. Job DeploymentPS: Step input azureSubscription references service connection $(EnvironmentSubscriptionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
There was a resource authorization issue: "The pipeline is not valid. Job DeploymentPS: Step input azureSubscription references service connection $(EnvironmentSubscriptionNameGold) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
There was a resource authorization issue: "The pipeline is not valid. Job DeploymentPS: Step input azureSubscription references service connection $(EnvironmentSubscriptionNameSilver) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
Any updates on this yet, it's quite a blocker.
I found this bug too, waiting for an update
This is tediously ignored...
To me it seems like the variable is not resolved at an early enough stage so i end up with this error telling me service connection $(azureSubscription) could not be found.
@AmrutaKawade @nadesu @PhilipsonJoseph any update ?
Hello, is there any update on this issue?
Pretty sure they aren't going to fix this. It's been an issue for a long time, and it's just not a priority.
Same issue, lost quite some time until I found that it is a known problem for 3 years, incredible 🤯
Last week I had still this issue (variables from variables group were not working with subscriptions names). To bypass it, I declared hardcoded variables in each specific step.
Use a parameter. It’s the simplest way of doing this. You don’t need to hardcode variables in each step. Pass in the subscription name as a parameter from a root yaml file and you don’t need to hardcode anything.
Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature? Type: Bug
Enter Task Name: AzurePowerShell@5
Environment
Azure Pipelines
Hosted
Issue Description
When I use the AzurePowerShell in deployment job in a Yaml Template, the 'azureSubscription' input is not expanded to the final value of parameter / variables.
I have the main template the following variable:
And I passing it to my child template as parameter:
The sub template contains parameter
And the deployment job has the task
Why the value of ${{ parameters.SubscriptionName }} is not expanded to final value. It works welle with scripts task or in steps nit in deployment.
Task logs
There are no logs because the pipeline didn't run.
Error logs
There was a resource authorization issue: "The pipeline is not valid. Job PrimaryIaaSDeployment: Step input azureSubscription references service connection $(EnvironmentSubscriptionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."