search

Azure / azure-dev

A developer CLI that reduces the time it takes for you to get started on Azure. The Azure Developer CLI (azd) provides a set of developer-friendly commands that map to key stages in your workflow - code, build, deploy, monitor, repeat.
https://aka.ms/azd
MIT License
396 stars 190 forks source link

[Issue] DevOps pipeline clears remote environment's config.json on azd up operation #4256

Open MateuszPeczek opened 2 weeks ago

MateuszPeczek commented 2 weeks ago

Output from azd version azd version 1.9.6 (commit 0b1a8ad350177078312141f4465ad72205d7c077)

Describe the bug After setting remote storage for environment variables and using them from DevOps pipeline, on azd up step content of config.json file is overriden to empty content and pipeline fails with message:

ERROR: prompting for value: no default response for prompt 'Enter a value for the '[paramName]' infrastructure parameter:'

To Reproduce

  1. Enable remote environments in azure.yaml file 
    name: IntegrationEngine.AppHost
state:
remote:
backend: AzureBlobStorage
config:
  accountName: [SAName]
  containerName: environments
services:  
app:
language: dotnet
project: .\IntegrationEngine.AppHost.csproj
host: containerapp
  2. Create or select existing environment using azd env new or azd env select [envName]
  3. Configure pipeline using command: azd pipeline config --provider azdo --principal-name [principalName] --remote-name [remoteName] --environment [envName]
  4. Ensure that proper files were uploaded to selected storage account. Check content of config.json file to ensure it was properly populated
  5. Run pipeline containing deploy section like here: 
    - task: AzureCLI@2
displayName: Provision Infrastructure
inputs:
  azureSubscription: sp-ng-ie-ado
  scriptType: bash
  scriptLocation: inlineScript
  inlineScript: |
    cd src/IntegrationEngine.AppHost
    azd env select $(AZURE_ENV_NAME) # ensure selection of proper env
    azd env list # debug list of available environments to validate connection to storage account
    azd up --no-prompt --environment $(AZURE_ENV_NAME) # provision and deploy
env:
  AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
  AZURE_ENV_NAME: $(AZURE_ENV_NAME)
  AZURE_LOCATION: $(AZURE_LOCATION)
  AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)
  AZURE_RESOURCE_GROUP: $(AZURE_RESOURCE_GROUP)

After pipeline output: 

Downloading Bicep
  (✓) Done: Downloading Bicep
Initialize bicep provider
Retrieving subscriptions...

content of config.json is overriden to {} value and pipeline fails with error given in bug description. When debugging pipeline, I can see that all my environments are in remote state and accessible from pipeline host:

NAME              DEFAULT   LOCAL     REMOTE
dev               true      true      true
dev-mpec          false     false     true

Expected behavior Pipeline retrieves the content of downloaded config.json file for selected environment and uses it to provision and deploy resources.

Environment Information on your environment:

rajeshkamal5050 commented 2 weeks ago

Thanks for reporting @MateuszPeczek was this working earlier?

@vhvb1989 can you triage it? cc @wbreza

vhvb1989 commented 2 weeks ago

@MateuszPeczek , if you have enabled remote env, can you try removing this env var: AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)

AZD_INITIAL_ENVIRONMENT_CONFIG secret is meant as a bridge from your local config.json to your CI/CD config.json. There might be an issue for it trying to work aside remove env.

While we look into what is the issue, can you try w/o this env var?

MateuszPeczek commented 2 weeks ago

Hi @vhvb1989, thanks for suggestion.

I did as you suggested but it hasn't change anything. I did it in two steps, first with that variable commented out from pipeline task

- task: AzureCLI@2
    displayName: Provision Infrastructure
    inputs:
      azureSubscription: sp-ng-ie-ado
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        cd src/IntegrationEngine.AppHost
        azd env select $(AZURE_ENV_NAME)
        azd env list
        azd up --no-prompt --environment $(AZURE_ENV_NAME)
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)
      AZURE_LOCATION: $(AZURE_LOCATION)
      #AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)
      AZURE_RESOURCE_GROUP: $(AZURE_RESOURCE_GROUP)

and later with removing this variable completely from pipeline's variables keeping it's usage commented out. Both solutions hasn't change anything.

Thanks for looking into it though.

vhvb1989 commented 2 weeks ago

thank you @MateuszPeczek , that helps, as it could indicate an issue with syncing config.json with remote env. If you have some more time to play with it, could you try disabling remote env and only using AZD_INITIAL_ENVIRONMENT_CONFIG ?

MateuszPeczek commented 2 weeks ago

Actually I was using local environment from few weeks now and it worked without any problems. Today I've found some spare time to investigate remote setup and I've found this issue I described trying to set it up.

I wanted to try it since it will be easier to share deployed envs with people from my team, instead of forcing them to provide exact same values for cloud dev, prod etc. environments.

MateuszPeczek commented 2 weeks ago

Thanks for reporting @MateuszPeczek was this working earlier?

@vhvb1989 can you triage it? cc @wbreza

Sorry @rajeshkamal5050, I've missed that first part somehow. Yes, it was working when I was using local environments config files to set up the pipeline. Issue started when I've enabled remote environments.

rajeshkamal5050 commented 1 week ago

@v-xuto can you try reproducing this?

NanaXiong00 commented 1 week ago

@rajeshkamal5050 Based on the above issue description, we cannot reproduce the current issue.

rajeshkamal5050 commented 1 week ago

@NanaXiong00 can you confirm if the repro was using Aspire flow/project? cc @vhvb1989

Bpflugrad commented 1 week ago

This is also affecting me. I had run into this problem back in May while trying to get CI/CD working and logged #3850.

Sorry about all the redactions, just have been asked to not reference certain things in posts like this. Here's the state of my azure.yaml:

# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-dev/main/schemas/v1.0/azure.yaml.json

name: [REDACTED].Aspire
services:  
  app:
    language: dotnet
    project: .\Aspire\[REDACTED].Aspire.AppHost\[REDACTED].Aspire.AppHost.csproj
    host: containerapp
state:
  remote:
    backend: AzureBlobStorage
    config:
      accountName: [REDACTED]
      containerName: [REDACTED]

And my azure-dev.yml. We use GitFlow and that determines which environment, so I end up appending a suffix to the environment name $(AZURE_ENV_NAME)$(envSuffix) to get the full name "Project-stg", etc.

trigger: none

pool:
  vmImage: ubuntu-latest

variables:
  majorVersion: '1'
  minorVersion: '0'
  patchVersion: '0'
  ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/master') }}:
    envSuffix: ""
    revCounter: ""
    optionalVersion: ""
    buildConfiguration: 'Release'
  ${{ elseif eq(variables['Build.SourceBranch'], 'refs/heads/staging') }}:
    envSuffix: "-stg"
    revCounter: $[counter(format('{0}.{1}.{2}-beta', variables.majorVersion, variables.minorVersion, variables.patchVersion ), 1)]
    optionalVersion: $[ format('-beta.{0}', variables.revCounter) ]
    buildConfiguration: 'Debug'
  ${{ else }}:
    envSuffix: "-dev"
    revCounter: $[counter(format('{0}.{1}.{2}-alpha', variables.majorVersion, variables.minorVersion, variables.patchVersion ), 1)]
    optionalVersion: $[ format('-alpha.{0}', variables.revCounter) ]
    buildConfiguration: 'Debug'

steps:
  - task: DotNetCoreCLI@2
    displayName: Install Aspire Workload with Preview
    inputs:
      command: custom
      custom: 'workload'
      arguments: 'install aspire --include-previews --source https://api.nuget.org/v3/index.json'

  - task: DotNetCoreCLI@2
    displayName: Restore private packages from [REDACTED]
    inputs:
      command: restore
      projects: '**/*.csproj'
      feedsToUse: select
      feedRestore: '[REDACTED]'
      includeNuGetOrg: true

  - task: DotNetCoreCLI@2
    displayName: 'DotNet Test'
    inputs:
      command: test
      projects: '**/*Tests/*.csproj'

  - task: setup-azd@0 
    displayName: Install azd

  - pwsh: |
      azd config set auth.useAzCliAuth "true"
    displayName: Configure AZD to Use AZ CLI Authentication.

  - script: echo $(AZURE_ENV_NAME)$(envSuffix)
    displayName: Display Environment

  - task: AzureCLI@2
    displayName: Azd Version
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd version

  - task: AzureCLI@2
    displayName: Provision Infrastructure
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd env select $(AZURE_ENV_NAME)$(envSuffix)
        echo "=================================="
        azd env list
        echo "=================================="
        azd provision --no-prompt --environment $(AZURE_ENV_NAME)$(envSuffix)
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)$(envSuffix)
      AZURE_LOCATION: $(AZURE_LOCATION)
      AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)

  - task: AzureCLI@2
    displayName: Deploy Application
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd env select $(AZURE_ENV_NAME)$(envSuffix)
        echo "=================================="
        azd env list
        echo "=================================="
        azd deploy --no-prompt --environment $(AZURE_ENV_NAME)$(envSuffix) --debug
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)$(envSuffix)
      AZURE_LOCATION: $(AZURE_LOCATION)
      AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)

I added the azd env select after reading @NanaXiong00's comment today.

Locally I am using azd version 1.10.0 (commit 1558a34a99490a2be20b1a5c963ff54bf60d918d).

While this may not be ideal I've used azd env select to pick the production environment. Then run azd pipeline config --principal-role Contributor (I am not granted permissions to add the User Access Manager role so I have to specify only Contributor). I set up the pipeline config because we need to have the production env to append to.

I then do azd env select Project-dev to pick the development environment.

I then run azd env set blah blah to push my config to blob storage.

I then trigger the build in Azure DevOps.

Here is the output from step Provision Infrastructure above:

2024-09-06T15:53:04.8222701Z ##[section]Starting: Provision Infrastructure
2024-09-06T15:53:04.8229687Z ==============================================================================
2024-09-06T15:53:04.8229863Z Task         : Azure CLI
2024-09-06T15:53:04.8229946Z Description  : Run Azure CLI commands against an Azure subscription in a PowerShell Core/Shell script when running on Linux agent or PowerShell/PowerShell Core/Batch script when running on Windows agent.
2024-09-06T15:53:04.8230224Z Version      : 2.244.1
2024-09-06T15:53:04.8230298Z Author       : Microsoft Corporation
2024-09-06T15:53:04.8230390Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-cli
2024-09-06T15:53:04.8230542Z ==============================================================================
2024-09-06T15:53:05.1758724Z [command]/usr/bin/az --version
2024-09-06T15:53:05.8577197Z WARNING: You have 2 update(s) available. Consider updating your CLI installation with 'az upgrade'
2024-09-06T15:53:05.8579473Z azure-cli                         2.63.0 *
2024-09-06T15:53:05.8620306Z 
2024-09-06T15:53:05.8621066Z core                              2.63.0 *
2024-09-06T15:53:05.8621374Z telemetry                          1.1.0
2024-09-06T15:53:05.8621479Z 
2024-09-06T15:53:05.8621633Z Extensions:
2024-09-06T15:53:05.8622171Z azure-devops                       1.0.1
2024-09-06T15:53:05.8622364Z 
2024-09-06T15:53:05.8622531Z Dependencies:
2024-09-06T15:53:05.8623092Z msal                              1.30.0
2024-09-06T15:53:05.8623459Z azure-mgmt-resource               23.1.1
2024-09-06T15:53:05.8623636Z 
2024-09-06T15:53:05.8623912Z Python location '/opt/az/bin/python3'
2024-09-06T15:53:05.8624301Z Extensions directory '/opt/az/azcliextensions'
2024-09-06T15:53:05.8624499Z 
2024-09-06T15:53:05.8624767Z Python (Linux) 3.11.8 (main, Jul 31 2024, 03:39:39) [GCC 11.4.0]
2024-09-06T15:53:05.8624902Z 
2024-09-06T15:53:05.8625177Z Legal docs and information: aka.ms/AzureCliLegal
2024-09-06T15:53:05.8625368Z 
2024-09-06T15:53:05.8625440Z 
2024-09-06T15:53:05.8636620Z Setting AZURE_CONFIG_DIR env variable to: /home/vsts/work/_temp/.azclitask
2024-09-06T15:53:05.8645326Z Setting active cloud to: AzureCloud
2024-09-06T15:53:05.8650923Z [command]/usr/bin/az cloud set -n AzureCloud
2024-09-06T15:53:06.5816787Z [command]/usr/bin/az login --service-principal -u *** --tenant [REDACTED] --allow-no-subscriptions --federated-token ***
2024-09-06T15:53:07.6333239Z [
2024-09-06T15:53:07.6333703Z   {
2024-09-06T15:53:07.6333913Z     "cloudName": "AzureCloud",
2024-09-06T15:53:07.6334604Z     "homeTenantId": "[REDACTED]",
2024-09-06T15:53:07.6335058Z     "id": "[REDACTED]",
2024-09-06T15:53:07.6335351Z     "isDefault": true,
2024-09-06T15:53:07.6335944Z     "managedByTenants": [
2024-09-06T15:53:07.6336579Z       {
2024-09-06T15:53:07.6337081Z         "tenantId": "[REDACTED]"
2024-09-06T15:53:07.6337371Z       }
2024-09-06T15:53:07.6337546Z     ],
2024-09-06T15:53:07.6337837Z     "name": "[REDACTED]",
2024-09-06T15:53:07.6338029Z     "state": "Enabled",
2024-09-06T15:53:07.6343864Z     "tenantId": "[REDACTED]",
2024-09-06T15:53:07.6344122Z     "user": {
2024-09-06T15:53:07.6344531Z       "name": "***",
2024-09-06T15:53:07.6344720Z       "type": "servicePrincipal"
2024-09-06T15:53:07.6344873Z     }
2024-09-06T15:53:07.6345019Z   }
2024-09-06T15:53:07.6345178Z ]
2024-09-06T15:53:07.6370136Z [command]/usr/bin/az account set --subscription [REDACTED]
2024-09-06T15:53:08.3009020Z [command]/usr/bin/bash /home/vsts/work/_temp/azureclitaskscript1725637985170.sh
2024-09-06T15:53:10.5380034Z ==================================
2024-09-06T15:53:12.1769961Z NAME                    DEFAULT   LOCAL     REMOTE
2024-09-06T15:53:12.1771154Z [REDACTED]              false     false     true
2024-09-06T15:53:12.1771558Z [REDACTED]-dev          true      true      true
2024-09-06T15:53:12.1817548Z ==================================
2024-09-06T15:53:13.6470157Z Analyzing Aspire Application (this might take a moment...)
2024-09-06T15:53:22.9480228Z 
2024-09-06T15:53:22.9480912Z Provisioning Azure resources (azd provision)
2024-09-06T15:53:22.9481160Z Provisioning Azure resources can take some time.
2024-09-06T15:53:22.9481263Z 
2024-09-06T15:53:23.1123370Z Downloading Bicep
2024-09-06T15:53:24.2464597Z   (✓) Done: Downloading Bicep
2024-09-06T15:53:24.7531600Z Initialize bicep provider
2024-09-06T15:53:30.7813166Z Retrieving subscriptions...
2024-09-06T15:53:34.1037013Z 
2024-09-06T15:53:34.1072619Z ERROR: initializing provisioning manager: prompting for value: no default response for prompt 'Enter a value for the '[REDACTED]_worker_BlobContainerName' infrastructure parameter:'
2024-09-06T15:53:34.1099581Z 
2024-09-06T15:53:34.1157142Z ##[error]Script failed with exit code: 1
2024-09-06T15:53:34.1200904Z [command]/usr/bin/az account clear
2024-09-06T15:53:34.7976835Z ##[section]Finishing: Provision Infrastructure

Here's an image of the output as well showing the error:

image

Hopefully there's just something I'm missing, but I've been unable to get CI/CD working with remote state in blob storage in any case.

vhvb1989 commented 1 week ago

Thank you @Bpflugrad

Can you manually look at your remote-state, inside the Storage container, that the config.json is not empty?

I am suspecting that the way this could be reproduced is by running azd pipeline config using one environemnt called XX and then making CI/CD to switch to another environment called YY.

AZD ensured the prompts for the parameters during azd pipeline config and make them part of the environment that is in use in that moment. That environment is expected to be the one used in CI/CD. If you are planning to use a different azd-environment in CI/CD, which may be, you are manually creating and don't want it to ever be locally in any device, then you need to manually write the config.json for that environment that you will select on CI/CD and provide the parameters.

Bpflugrad commented 1 week ago

Hi @vhvb1989,

Thanks for the quick response.

Before running the pipeline the config.json is populated both for the production and -dev environment in the storage account and container. After the azd provision fails, the config.json is empty for -dev.

Do you suggest that we have unique pipelines for each environment? I can set that up and see if it helps.

It just seems that, I am populating config.json for all environments, I am selecting the proper environment and I am also using --environment and AZURE_ENV_NAME, so azd provision should be using the -dev environment. The echo shows that -dev is selected, remote, and local, so how is this execution different from my local machine?

Thanks again!

Bpflugrad commented 1 week ago

Hi @vhvb1989,

I edited the azure-dev.yml to not include the suffixing, reran azd pipeline config for the -dev environment, confirmed that config.json was populated, triggered the pipeline.

I still get the missing parameter error, and my config.json still becomes {} after the pipeline completes.

It doesn't appear that azd pipeline config supports having more than one pipeline per DevOps project. I can't find any flags that will change the pipeline name. So it would be difficult run this for each environment and have unique pipes per environment. I could update AZD_INITIAL_ENVIRONMENT_CONFIG manually I suppose for each when necessary.

vhvb1989 commented 5 days ago

Reading a the steps, can you confirm: 1) Set up remote store locally 2) Use current azd-env (let's call it A) to run pipeline config; config.json is populated and pipeline created. 3) Change local azd-env to another env (let's call it B) and use azd env set X Y to build-up this env remotely. 4) Manually update config.json for env B in the storage account. 5) Run pipeline in azdo and use a prefix to select env B during the run. Here you see the issue and azd turns config.json for env B in the storage account blank.

I haven't been able to reproduce on my side, so I want to make sure if there's something I'm missing.

NanaXiong00 commented 5 days ago

@NanaXiong00 can you confirm if the repro was using Aspire flow/project? cc @vhvb1989

@rajeshkamal5050 Re-testing using Aspire project, still unable to reproduce the current issue. image

Bpflugrad commented 3 days ago

Hi @vhvb1989 ,

I followed these instructions exactly, verifying the content of blob storage between each step. At step 5 my azd provision step fails, says the variables are not set, and the value in blob storage becomes {}.

I did have to manually change the environment name to B, since I created the pipe with A, since I had removed the suffixing. I'm doing azd env select and everything in the pipeline too, so the values should be brought locally.

Could this have anything to do with the presence of vault in my config.json?

Here's the complete config.json from blob storage:

{
  "infra": {
    "parameters": {
      "BlobContainerName": "blobs",
      "context_password": "vault://600f5f5c-bf55-4fab-8508-37b84dbbfc54/940386a5-a91f-44b8-994b-7f7582bd4a47",
      "QueueName": "queue",
      "storageAccountParam": "dev"
    }
  },
  "vault": "600f5f5c-bf55-4fab-8508-37b84dbbfc54"
}

I think you mentioned this was a defect in a previous version of azd back in May.

Thanks!

vhvb1989 commented 2 days ago

Thank you for your detailed exploration @Bpflugrad ; I understand the issue now (and know why its happening).

As a workaround, what you can do, if you want to use remote-env:

Context: The combination of "valut" field and parameters using values with references to a "vault://...." is the implementation of user-secrets feature. This feature prevents users for writing secrets within the repository files. The values are written in user's home directory and just referenced by ids. When you run azd pipeline config, azd updates the config.json to make it work in CI/CD w/o user-secrets (no vault references) and sets the AZD_INITIAL_ENV as a secret with that value. But, when using remote-env, azd is pulling the config.json with the vault references and then it fails to find the vault from user's home directory. In that case, the current fallback action is to consider the config.json as corrupted and creates a new one in blank, then azd saves the env, updating remote-env with an empty object.

I'll check with the team about how we want to handle this scenario and come back

Bpflugrad commented 1 day ago

Hi @vhvb1989,

Thanks for that workaround. Removing all vault references from the config in blob storage does allow me to proceed. However, after azd provision completes, vault references are added for the missing configuration. In this case context_password is automatically generated by AddSqlServer, and added to secrets.

The second time the pipeline runs, it fails and the config is returned to {}. This is because the build host from the pool no longer has the vault added by the previous run.

It seems like the only way to do this then, is to manually edit the config.json every time a deployment is going to happen. Or, I guess, maintain a vaultless version of it somewhere else and overwrite it with additional pipeline steps as part of each deployment.

This seems like it would be a major blocker to adoption as Aspire projects can't reasonably be deployed with CI/CD... I'm hoping there are plans to address this?

Again thanks for your engagement on here. :)

vhvb1989 commented 21 hours ago

I see. Thank you for trying and reporting about the workaround, @Bpflugrad.

I've started the conversation for how to solve this. Ideally, I am trying to make azd to detect when running on CI/CD and turn remote-env into read-only; I don't think jobs from CI/CD should be updating the remote-env.

@ellismg @wbreza @weikanglim FYI