azd provision: Aspire Redis container fails to deploy (hangs) when container apps environment is inside vnet

[mip1983]

Output from azd version Run azd version and copy and paste the output here:

azd version 1.6.0 (commit f90c77f78899db817f56cde9eab4b9421ec15535)

Describe the bug Description of issue you're seeing...

I've used azd infra synth to generate bicep files for my aspire app, which is based on the aspire starter app template in vs2022.

I've referenced my existing vnet in the resources.bicep file and successfully connected the container apps environment to the vnet, the web and API containers are working with this setup (there is a network security group on the subnet which has been configured for inbound http/https).

I'm now wanting to use a redis container which I've added the same as the starter template does. However, during azd provision, the process just seems to get stuck and the cache container is created but never starts, the process and azd seems to hang until time out.

The revision appears stuck in a 'processing' state:

There's nothing in the console logs, the system logs mostly show this repeating:

{"TimeStamp":"2024-02-14 15:33:33 \u002B0000 UTC","Type":"Normal","ContainerAppName":"cache","RevisionName":"","ReplicaName":"","Msg":"Updating containerApp: cache","Reason":"ContainerAppUpdate","EventSource":"ContainerAppController","Count":8}

But occasionally there is a warning like this:

{"TimeStamp":"2024-02-14 15:33:33 \u002B0000 UTC","Type":"Warning","ContainerAppName":"cache","RevisionName":"","ReplicaName":"","Msg":"ContainerApp : cache failed to set traffic. Error : ingress routes not ready","Reason":"ContainerAppFailure","EventSource":"ContainerAppController","Count":6}

{"TimeStamp":"2024-02-15 15:02:39 \u002B0000 UTC","Type":"Normal","ContainerAppName":"cache","RevisionName":"","ReplicaName":"","Msg":"Setting traffic weight of \u0027100%\u0027 for revision \u0027cache--rpdik5p\u0027","Reason":"RevisionUpdate","EventSource":"ContainerAppController","Count":5}

{"TimeStamp":"2024-02-15 15:02:48 \u002B0000 UTC","Type":"Warning","ContainerAppName":"cache","RevisionName":"","ReplicaName":"","Msg":"ContainerApp : cache failed to set traffic. Error : ingress routes not ready","Reason":"ContainerAppFailure","EventSource":"ContainerAppController","Count":5}

edit: And this now:

{"TimeStamp":"2024-02-15 14:35:52 \u002B0000 UTC","Type":"Normal","ContainerAppName":"cache","RevisionName":"","ReplicaName":"","Msg":"EnvironmentManagedStorageJob: VNet is not set for cluster kindstone-9ddf6b2a.","Reason":"","EventSource":"ContainerAppController","Count":0}

This is the resource that bicep generated:

resource cache 'Microsoft.App/containerApps@2023-05-02-preview' = {
  name: 'cache'
  location: location
  properties: {
    environmentId: containerAppEnvironment.id
    configuration: {
      service: {
        type: 'redis'
      }
    }
    template: {
      containers: [
        {
          image: 'redis'
          name: 'redis'
        }
      ]
      scale: {
        minReplicas: 1
      }
    }
  }
  tags: union(tags, {'aspire-resource-name': 'cache'})
}

To Reproduce Steps to reproduce the behavior...

Use the out of the box Aspire started template and create a vnet and subnet manually in azure to use for the container apps environment.

Edit the 'resources.bicep' file to add the vnet to your container apps environment:

resource vnet 'Microsoft.Network/virtualNetworks@2022-07-01' existing = {
  name: '<yourVNETName>'
  scope: resourceGroup('<yourResourceGroup>')
}

resource subnet 'Microsoft.Network/virtualNetworks/subnets@2023-04-01' existing = {
    name: 'ContainerTestSubnet'
    parent: vnet
}

resource containerAppEnvironment 'Microsoft.App/managedEnvironments@2023-05-01' = {
  name: 'cae-${resourceToken}'
  location: location
  properties: {
    appLogsConfiguration: {
      destination: 'log-analytics'
      logAnalyticsConfiguration: {
        customerId: logAnalyticsWorkspace.properties.customerId
        sharedKey: logAnalyticsWorkspace.listKeys().primarySharedKey
      }
    }
    vnetConfiguration: {
      infrastructureSubnetId: subnet.id
    }
  }
  tags: tags
}

run azd provision

Expected behavior Successful deployment of my redis container with 1 running revision.

Environment Information on your environment:

.NET Aspire, .NET 8, aspire preview 3, Visual Studio 2022

[rajeshkamal5050]

@jongio @mcollier please chime-in, if you have any inputs on the vnet resources.

[mip1983]

Small update, no real progress but I tried putting allow anything in/out rules on the network security group for the subnet and it still didn’t want to come up, so not sure what it is about the vnet it doesn’t like.

[rajeshkamal5050]

Currently aspire has no support for adding a vnet to the azure container app.

Until the core modeling support is added to Aspire. You can try changing the bicep files for vnet configurations. Try taking a look at these template's bicep files for reference - https://azure.github.io/awesome-azd/?name=vnet

[mip1983]

Yea, been using azd infra synth and bicep files to connect it to the vnet, that all works but redis does not, and fails in this silent difficult to understand way.

Found out it's a documented as a limitation here:

https://learn.microsoft.com/en-us/azure/container-apps/services#limitations

The support persons I've spoke to have said they need to improve the error and feedback mechanisms so hopefully that'll stop people going down this rabbit hole.

So far I'm working around the issue with the container apps environment outside the vnet, and using nsg rules to allow access to what it needs, will either stick with this approach or use azure managed redis.