search

Azure / azure-dev

A developer CLI that reduces the time it takes for you to get started on Azure. The Azure Developer CLI (azd) provides a set of developer-friendly commands that map to key stages in your workflow - code, build, deploy, monitor, repeat.
https://aka.ms/azd
MIT License
369 stars 166 forks source link

[Issue] azd deploy fails in azdo build pipeline when connection string value is expected from remote environment. #3850

Open Bpflugrad opened 2 weeks ago

Bpflugrad commented 2 weeks ago

Output from azd version Run azd version and copy and paste the output here: azd version 1.8.2 (commit 14600c7a54edac4f54397413f8638431f5c16327)

Describe the bug Deploying in an azdo pipeline:

Pool: Azure Pipelines
Image: ubuntu-latest
Agent: Hosted Agent
Started: Today at 1:39 PM

Using Aspire preview 6 8.0.0-preview.6.24214.1. Using AddConnectionString(string) to add a reference to an existing Azure resource by connection string. When deploying in an azdo pipeline, if a deployment has never been completed before (such as with azd up on a development machine), azd deploy fails with error message:

panic: send on closed channel

goroutine 55 [running]:
github.com/azure/azure-dev/cli/azd/pkg/async.(*TaskContextWithProgress[...]).SetProgress(...)
    /mnt/vss/_work/1/s/cli/azd/pkg/async/task_context.go:74
github.com/azure/azure-dev/cli/azd/pkg/project.syncProgress[...](0xc0002105a0, 0x0)
    /mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:678 +0xbd
created by github.com/azure/azure-dev/cli/azd/pkg/project.runCommand[...].func1 in goroutine 36
    /mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:658 +0xb9

This causes the pipeline to fail and deployment cannot be completed.

To Reproduce

pool: vmImage: ubuntu-latest

steps:

The build will be triggered, and at the Deploy Application step will fail with error:

Analyzing Aspire Application (this might take a moment...)

Deploying services (azd deploy)

Deploying service apiservice
Deploying service apiservice (Logging in to registry)
Deploying service apiservice (Pushing container image)
panic: send on closed channel

goroutine 55 [running]:
github.com/azure/azure-dev/cli/azd/pkg/async.(*TaskContextWithProgress[...]).SetProgress(...)
    /mnt/vss/_work/1/s/cli/azd/pkg/async/task_context.go:74
github.com/azure/azure-dev/cli/azd/pkg/project.syncProgress[...](0xc0002105a0, 0x0)
    /mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:678 +0xbd
created by github.com/azure/azure-dev/cli/azd/pkg/project.runCommand[...].func1 in goroutine 36
    /mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:658 +0xb9

##[error]Script failed with exit code: 2

Expected behavior Deployment should succeed since the required parameter is present in the environment.

Environment Information on your environment:

Additional context When not using azd infra synth the error is slightly different:

ERROR: failed deploying service 'webfrontend': failing invoking action 'deploy', failed executing template file: template: containerApp.tmpl.yaml:21:19: executing "containerApp.tmpl.yaml" at <securedParameter "BlobStorageConnection">: error calling securedParameter: parameter BlobStorageConnection not found

Also tried to updating the builder.AddConnectionString("BlobStorageConnection", "AZURE_BLOB_STORAGE_CONNECTION"); and still get the panic.

mip1983 commented 1 week ago

I've struggled with providing connection strings myself, providing these 'secure parameters' in a pipeline from an environment variable seems to be challenging. There may be something in my thread that helps you: https://github.com/Azure/azure-dev/issues/3597

Bpflugrad commented 1 week ago

@mip1983 thanks it looks like I might be able to work around my problem with the Variable Groups mentioned in your thread. The really unfortunate thing is that AddParameter works fine and azd infra synth shows that the same behavior of turning the AddConnectionString argument into AZURE_PASCAL_CASE is followed but unlike Parameters the value is not brought in from remote configuration in the same way. Really seems like a bug to me.

vhvb1989 commented 1 week ago

Hi @Bpflugrad , I can explain what is happening.

When you run azd infra synth, as you can see, azd is generating a bicep secured parameter for the connection string:

@secure()
param BlobStorageConnection string

If you azd pipeline config before adding the connection string to the AppHost program, azd will not ask/prompt for a value for the parameter (b/c it is not yet known). You would be using azd pipeline config just to set up the azdo repo and service connector. Then, when you add the connection string, run infra synth and commit the changes, your pipeline will run, but azd did not set any value for the parameter (as secret or variable) for the pipeline.

What you are doing is interesting, because you are hoping to set the env var in remote-state and have the pipeline to pick the env var from the remote state. Ideally, you should not add plain text to remote-state, as that's not a secured store. But, making the secure thing apart, azd is been able to pull the value from the remote-state to run provision, but azd is not expecting to find the securedParameter within the environment (in .env).

Azd uses the .azure/envName/config.json file to save the values of secured-parameters. That config.json is set in azdo as a secret, and that's where azd will try to find the value during deploying the app (for setting the value in the container app environment).

So, here's what you should be able to do, to make your sample work:

Now azd will be able to fetch the value from the remote-state (from the config.json) for provison and for deploy.

Now, let me just say it again. remote-state is not meant for secured values. We are still working/designing a secured store (like KeyVault) but, remote-state should not be considered secured. For azd version 1.9.0, we update how secrets are saved in the config.json, by writing the actual secret outside of the project (inspired by .Net user secrets). So, instead of all the manual steps you are doing to set up the remote-state, you can remove the remote-state config and just do 2 changes to your project:

  1. Update your pipeline config to reference AZD_INITIAL_ENVIRONMENT_CONFIG env var during provision, like:
 - task: AzureCLI@2
    displayName: Provision Infrastructure
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd provision --no-prompt --environment $(AZURE_ENV_NAME)
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)
      AZURE_LOCATION: $(AZURE_LOCATION)
     AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)
  1. After you add the connection string to the appHost, run azd pipeline config and let azd to prompt for the connection string. Enter the connection string. azd will set the value in a secret AZD_INITIAL_ENVIRONMENT_CONFIG automatically, and your pipeline will work. Then, you can open the .azure/envName/config.json and see that there is a ref to a local azd-vault, and not the raw plain text connection string, like
{
  "infra": {
    "parameters": {
      "BlobStorageConnection": "vault://1b14641c-bb16-4ee9-87f9-d5e19ef09b17/6675dd26-3372-47a7-a35e-b2026e84c2e9"
    }
  },
  "vault": "1b14641c-bb16-4ee9-87f9-d5e19ef09b17"
}

When azd sets AZD_INITIAL_ENVIRONMENT_CONFIG, it will resolve the vault reference and write the value as a secret in azdo.

Bpflugrad commented 1 week ago

Thanks for the help @vhvb1989 this has been a lot of trial and error.

I decided to go with your second suggestion, and updated my build pipe and re-ran azd pipeline config. I was prompted for the value, and the vault reference similar to what you pasted is in my local config.json.

When I run the pipe now I get a lot of security errors either saying the wrong client secret was provided: ERROR: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '***'. Or a panic from Go after I saved the service principal thinking it might need a new token: panic: don't know how to prompt for type *survey.Password My recent runs have all resulted in the panic error, tried adding azd infra synth back and a few other things.

This wasn't happening prior to my changes so my assumption is that attempting to resolve the vault reference is the issue. I'm not familiar with vault://, where are those and what do they refer to?

Bpflugrad commented 4 days ago

Hi @vhvb1989,

Sorry for the confusion Friday. I figured out why I was getting the panic message. Basically, when you have used AddConnectionString("Value") a line @secure() param Value string is added to the main.bicep. Even though this is only used by azd deploy it seems that some value must exist in the .env, even if it is just a fake value.

However, after successfully doing this, and verifying that the following appears in my remote config.json:

{
  "infra": {
    "parameters": {
      "BlobStorageConnection": "vault://9f52a64c-260e-4515-9d32-4b5463939db7/27440911-0f41-45f5-83f8-682a15aacdd4"
    }
  },
  "vault": "9f52a64c-260e-4515-9d32-4b5463939db7"
}

However, azd deploy is failing with:

2024/05/13 15:42:35 service_target_dotnet_containerapp.go:195: generating container app manifest from /home/vsts/work/1/s/AspireInfraExample.AppHost/AspireInfraExample.AppHost.csproj for project apiservice
Deploying service apiservice (Updating container app)
  (x) Failed: Deploying service apiservice

ERROR: failed deploying service 'apiservice': failing invoking action 'deploy', failed executing template file: template: containerApp.tmpl.yaml:21:19: executing "containerApp.tmpl.yaml" at <securedParameter "BlobStorageConnection">: error calling securedParameter: parameter BlobStorageConnection not found

I am not using azd infra synth in this case. I also changed the slashes from backslash \ to forward slash / in azure.yaml since you mentioned the regression in #3891 but that didn't make a difference.

If I do use azd infra synth I end up with:

2024/05/13 15:53:22 service_target_dotnet_containerapp.go:186: using container app manifest from /home/vsts/work/1/s/AspireInfraExample.AppHost/infra/apiservice.tmpl.yaml
panic: send on closed channel

goroutine 66 [running]:
github.com/azure/azure-dev/cli/azd/pkg/async.(*TaskContextWithProgress[...]).SetProgress(...)
    /mnt/vss/_work/1/s/cli/azd/pkg/async/task_context.go:74
github.com/azure/azure-dev/cli/azd/pkg/project.syncProgress[...](0xc0004ad380, 0x0)
    /mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:678 +0xbd
created by github.com/azure/azure-dev/cli/azd/pkg/project.runCommand[...].func1 in goroutine 28
    /mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:658 +0xb9

Also after each failed deployment the remote config.json is reset to {} so I have to do azd env set with a fake value each time. I made sure to run azd pipeline config so that AZD_INITIAL_ENVIRONMENT_CONFIG would be updated (I can't validate its content from Azure DevOps). I am passing AZD_INITIAL_ENVIRONMENT_CONFIG as env to both azd provision and azd deploy in my pipeline:

  - task: AzureCLI@2
    displayName: Provision Infrastructure
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd provision --no-prompt
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)
      AZURE_LOCATION: $(AZURE_LOCATION)
      AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)

  - task: AzureCLI@2
    displayName: Deploy Application
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd deploy --no-prompt --debug
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)
      AZURE_LOCATION: $(AZURE_LOCATION)
      AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)

You only mentioned adding AZD_INITIAL_ENVIRONMENT_CONFIG to azd provision but I figured since the value is needed in azd deploy that it wouldn't hurt to have it on both.

Anyway, perhaps I missed a step but I can't get your second suggestion to work.

Any help is appreciated!

vhvb1989 commented 4 days ago

@Bpflugrad , #3891 was not the only issue affecting azd on CI/CD. There was also https://github.com/Azure/azure-dev/pull/3897. The fix was merged already, but you need daily-build to get it. There's 2 things you can do:

  1. Update your pipeline definition and your local azd to use daily-azd build,
  2. or, before running azd pipeline config, update the config.json from what you have: 
    {
"infra": {
"parameters": {
  "BlobStorageConnection": "vault://9f52a64c-260e-4515-9d32-4b5463939db7/27440911-0f41-45f5-83f8-682a15aacdd4"
}
},
"vault": "9f52a64c-260e-4515-9d32-4b5463939db7"
}

    to what it should be in CI:

    {
"infra": {
"parameters": {
  "BlobStorageConnection": "The actual connection string HERE"
}
},
}

The issue on 1.9.0 is that azd is setting the config.json like:

{
  "infra": {
    "parameters": {
      "BlobStorageConnection": "The actual connection string HERE"
    }
  },
  "vault": "9f52a64c-260e-4515-9d32-4b5463939db7"
}

You can see that the secrets are resolved, but the vault field is still there, and when runining in CI, azd will not find that vault and will just create an empty config. In the fix in daily, azd is removing the vault field for the resolved-config

Bpflugrad commented 4 days ago

Thanks @vhvb1989. I updated to the daily build. The workaround (your second suggestion) where the raw value is included in config.json works.

However, using azd naturally (your first suggestion), including the secured values with the vault references still doesn't work. I updated both the build pipe and local azd to 1.10.0-beta.1-daily.3780018 (commit bb526cbd20cfbd9d2ebadc2e34f1058e8595ff4f). I'm still getting panic: send on closed channel

2024/05/13 21:22:15 service_target_dotnet_containerapp.go:195: generating container app manifest from /home/vsts/work/1/s/AspireInfraExample.AppHost/AspireInfraExample.AppHost.csproj for project apiservice
panic: send on closed channel

goroutine 56 [running]:
github.com/azure/azure-dev/cli/azd/pkg/async.(*TaskContextWithProgress[...]).SetProgress(...)
    /mnt/vss/_work/1/s/cli/azd/pkg/async/task_context.go:74
github.com/azure/azure-dev/cli/azd/pkg/project.syncProgress[...](0xc0004c0690, 0x0)
    /mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:689 +0xbd
created by github.com/azure/azure-dev/cli/azd/pkg/project.runCommand[...].func1 in goroutine 35
    /mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:669 +0xb9

After updating the azd version to the daily build, I reran azd pipeline config just in case vault creation and resolution had changed.

Thanks again for your help, I can at least proceed with the insecure workaround until the vaults can be figured out.

Bpflugrad commented 2 days ago

Hi @vhvb1989 ,

I tried a few permutations of my build script. If I have AZD_INITIAL_ENVIRONMENT_CONFIG only on azd provision or only on azd deploy, I get the missing parameter error message from azd deploy like: ERROR: failed deploying service 'apiservice': failed executing template file: template: containerApp.tmpl.yaml:21:19: executing "containerApp.tmpl.yaml" at <securedParameter "BlobStorageConnection">: error calling securedParameter: parameter BlobStorageConnection not found If I have AZD_INITIAL_ENVIRONMENT_CONFIG on BOTH azd provision and azd deploy I get the panic message:

panic: send on closed channel

goroutine 56 [running]:
github.com/azure/azure-dev/cli/azd/pkg/async.(*TaskContextWithProgress[...]).SetProgress(...)
    /mnt/vss/_work/1/s/cli/azd/pkg/async/task_context.go:74
github.com/azure/azure-dev/cli/azd/pkg/project.syncProgress[...](0xc0004c0690, 0x0)
    /mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:689 +0xbd
created by github.com/azure/azure-dev/cli/azd/pkg/project.runCommand[...].func1 in goroutine 35
    /mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:669 +0xb9

I've never worked with Go before but looking at this stack trace, service_manager.go and service_target_dotnet_containerapp.go it appears that there's a silent failure between here and here. Somehow ServiceProgress is getting closed and causing the send on closed channel. At least that is my guess.

vhvb1989 commented 2 days ago

Hi @Bpflugrad We just released 1.9.1 which should have a fix for how AZD_INITIAL_ENVIRONMENT_CONFIG is set when you run azd pipeline config Can you try using 1.9.1 to run azd pipeline config please. Let me know what you get

Bpflugrad commented 2 days ago

Hi @vhvb1989,

Thanks for the reply. I updated my local machine to 1.9.1 (commit aadbc26943c2e3e5437a6ffa528fe5264887a10c), and I updated the pipeline to not use the daily build anymore, confirmed it is using the same version.

I took the following steps:

Unfortunately, azd deploy still fails with panic: send on closed channel.