microsoft / azure-container-apps

Roadmap and issues for Azure Container Apps
MIT License
360 stars 29 forks source link

Failed to create Container Apps Environment with custom virtual network #451

Open pbakun opened 1 year ago

pbakun commented 1 year ago

Please provide us with the following information:

This issue is a: (mark with an x)

Issue description

In West Europe region I have failed to create Container Apps Environment with virtual network. I followed documentation provided here. Tried both with Azure CLI and Azure Powershell.

Steps to reproduce

  1. Copy paste all commands from documentation until New-AzContainerAppManagedEnv (including it) into powershell script.
  2. Run script.

Expected behavior Successfully create Container Apps Environment with connection to subnet

Actual behavior Error message is display: Message: Fail to create managed environment because provided subnet Id is invalid

Screenshots
image

Additional context

The issue occurred when using Azure Portal, Azure CLI and Azure Powershell.

Additionally I paste my powershell script:

$LOCATION='west europe'
$SubnetArgs = @{
    Name = $SUBNET_NAME
    AddressPrefix = '10.0.0.0/23'
}
$subnet = New-AzVirtualNetworkSubnetConfig @SubnetArgs

$VnetArgs = @{
    Name = $VNET_NAME
    Location = $LOCATION
    ResourceGroupName = $RESOURCE_GROUP
    AddressPrefix = '10.0.0.0/16'
    Subnet = $subnet 
}
$vnet = New-AzVirtualNetwork @VnetArgs
$InfrastructureSubnet=(Get-AzVirtualNetworkSubnetConfig -Name $SubnetArgs.Name -VirtualNetwork $vnet).id

$WorkspaceArgs = @{
    Name = 'myworkspace'
    ResourceGroupName = $RESOURCE_GROUP
    Location = $LOCATION
    PublicNetworkAccessForIngestion = 'Enabled'
    PublicNetworkAccessForQuery = 'Enabled'
}
New-AzOperationalInsightsWorkspace @WorkspaceArgs
$WorkspaceId = (Get-AzOperationalInsightsWorkspace -ResourceGroupName $RESOURCE_GROUP -Name $WorkspaceArgs.Name).CustomerId
$WorkspaceSharedKey = (Get-AzOperationalInsightsWorkspaceSharedKey -ResourceGroupName $RESOURCE_GROUP -Name $WorkspaceArgs.Name).PrimarySharedKey

$EnvArgs = @{
    EnvName = $CONTAINERAPPS_ENVIRONMENT
    ResourceGroupName = $RESOURCE_GROUP
    Location = $LOCATION
    AppLogConfigurationDestination = 'log-analytics'
    LogAnalyticConfigurationCustomerId = $WorkspaceId
    LogAnalyticConfigurationSharedKey = $WorkspaceSharedKey
    VnetConfigurationInfrastructureSubnetId = $InfrastructureSubnet
    VnetConfigurationInternal = $true
}
New-AzContainerAppManagedEnv @EnvArgs
macpak commented 1 year ago

I had the same issue using Pulumi. I kinda solved it in a way that I created the Container App Environment in Azure Portal, then deleted it and re-created using Pulumi. I believe creating the environment for the first time via Azure Portal enables something, that's why subsequent deployments work correctly from Pulumi. Hope that helps :)

kendallroden commented 1 year ago

Hi there @pbakun - have you registered your subscription with the appropriate providers? You need one for Microsoft.ContainerServices and one for Microsoft.OperationalInsights. Bicep/ARM does not auto-register these for you when you are using them to deploy.

az provider register -n Microsoft.ContainerService az provider register -n Microsoft.OperationalInsights

If this doesn't unblock you, let me know. I ran into this issue myself recently.

cc @tomer-ha

markdebruijne commented 1 year ago

I run into the same the same issue (after registering both providers) @kendallroden , via Azure CLI

"azure-cli": "2.39.0",
  "extensions": {
    "containerapp": "0.3.13",

provide argument pointing to VNET subnet: --infrastructure-subnet-resource-id /subscriptions/36470000-0000-0000-0000-000000000000/resourceGroups/*****-poc/providers/Microsoft.Network/virtualNetworks/*****poc-vnet/subnets/container-environment. Thinking the default NSG (rules without explicit NSG) might be interfering, created a NSG following these instructions: https://learn.microsoft.com/en-us/azure/container-apps/firewall-integration. Errors remains.

markdebruijne commented 1 year ago

Retried after configuring subnet delegation with Microsoft.Apps/environments, but wasn't a solution :)

pbakun commented 1 year ago

I tried to create it through portal like @macpak suggested and environment created successfully, but it is without having --internal-only flag set. Will try to recreate it through CLI when I find a minute. I'm not sure about Microsoft.OperationalInsights, but I had Microsoft.ContainerService registered for sure. Right now both are registered.

kendallroden commented 1 year ago

Are you still seeing this issue @pbakun

ghost commented 1 year ago

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

onionhammer commented 1 year ago

I am seeing this issue when I try to create an app env with the CLI


az containerapp env create `
    -n $envName `
    -l centralus `
    -g $rgName `
    --logs-workspace-id $WORKSPACE_ID `
    --logs-workspace-key $workspaceKey `
    -s $INFRASTRUCTURE_SUBNET
"deploymentErrors": "ErrorCode: ManagedEnvironmentSubnetIdInvalid, Message: Fail to create managed environment because provided subnet Id is invalid, refer to https://go.microsoft.com/fwlink/?linkid=2198255 for more detail.",

{ "azure-cli": "2.42.0", "azure-cli-core": "2.42.0", "azure-cli-telemetry": "1.0.8", "extensions": { "application-insights": "0.1.16", "azure-devops": "0.25.0", "containerapp": "0.3.13", "cosmosdb-preview": "0.21.0", "front-door": "1.0.17", "interactive": "0.4.5", "mesh": "0.10.7", "storage-preview": "0.8.3" } }

markdebruijne commented 1 year ago

Closing an issue only because time passed-by? Would be nice if all issues disappear by just sitting out :)

@kendallroden why the question, was there a CLI update with potential fix, or backend update on the resourceprovider perhaps? https://github.com/microsoft/azure-container-apps/issues/451#issuecomment-1299326294

hoffmanntravis commented 1 year ago

I am encountering this as well with bicep. We were still specifying both subnets (infrastructureSubnetId and runtimeSubnetId). runtimeSubnetId property is deprecated according to this:

https://github.com/microsoft/azure-container-apps/issues/374

Hoping this resolves our issue. I will report back if it does and I remember to do so.

hoffmanntravis commented 1 year ago

Removing runtimeSubnetId from the bicep definition did appear to fix our issue, but we've seen quite a bit of intermittency with Azure Container App environment creation so it's difficult to say with certainty that was the fix. I believe it was.

kendallroden commented 1 year ago

The issue is closed automatically when the poster is requested to provide info and doesn't respond within a given period of time! Will keep it open if this is still an issue @markdebruijne

kendallroden commented 1 year ago

Hi @hoffmanntravis I am glad you were able to remove the runtimeSubnetId and resolve your issue. Wondering if you are still running into this or if it is potentially just intermittent? Trying to get reproducible failure. @markdebruijne

onionhammer commented 1 year ago

@kendallroden I think the issue is that invalid configurations around vnets dont "fail fast" and provide decent error reporting, so a user who is misconfiguring a container app wont find out about it for.. 45 minutes? It just times out and fails to deploy without telling you why.

pbakun commented 1 year ago

Hi, sorry for beeing so quiet, but I recently had a lot on my head.

I just tried to create containerapp environment with vnet again on new subscription after using az containerapp env show I got:

"deploymentErrors": "ErrorCode: ManagedEnvironmentSubnetIdInvalid, Message: Fail to create managed environment because provided subnet Id is invalid, refer to 
https://go.microsoft.com/fwlink/?linkid=2198255 for more detail.",
"provisioningState": "Failed",

So it seems that the issue is still there. I guess the workaround with creating it couple times through Azure Portal will finally make it work like last time, but I don't think this should be a way to go with it. I will not try the workaround yet, so I will have a way to try it again if there will be any fixes released.

hoffmanntravis commented 1 year ago

@kendallroden Hi Kendall. Unfortunately it looks like this popped up in a fresh deploy for us via bicep tonight for a new client (so all new resources, new resource group, etc) without the runtimeSubnetId property set from the start. I am able to see the deployment status in the json view of the container environment so we notice before an hour and cancel the deploy, but it usually takes me 15 minutes to realize something went south.

The more difficult part is so far the fix seems to be to keep deleting the resource and re-deploying until it works. It's possible there is some sort of race condition in our bicep but I'm not sure what it would be or how. We explored that pretty carefully. Honestly it feels more like a backend timing issue but that's a bit of a guess.

hoffmanntravis commented 1 year ago

One thing I have noticed is it seems like container environment creation is only available in the Azure Portal UI if I first try to create a container app and then create the container environment as a new resource inside a sub menu (It may be available somewhere but I don't see it when I search for it, only container app in Marketplace. Is it valid to create a container environment without any containers referenced either in bicep or through other tooling? We are dynamically creating containers based on some information in bicep via a module with a loop and a batch size of 1. I'm wondering if that might be related here somehow to this? Perhaps it has nothing to do with anything.

@batchSize(1)
module containers 'resources-container.bicep' = [for item in buildInfo.Containers: {
  name: item.name

relevant lines of code above. buildInfo.Containers is a deserialized JSON list of objects.

I would think the bicep would get expanded out and ultimately run as ARM before any of it get's to Azure, so I don't think it's a relevant point, but I'm not certain honestly. By the time it does make it to Azure I would expect the list of container apps to indeed exist on our side. image

JP-Ellis-KPMG commented 1 year ago

I have encountered this issue myself. While trying to identify the root issue, I have created the minimal example below to reproduce the issue. It results in the following non-descript error:

{
    "status": "Failed",
    "error": {
        "code": "ResourceDeploymentFailure",
        "message": "The deployment operation status is unknown."
    }
}

but annoyingly, the status on Azure shows the deployment as still "In progress" (or in the CLI, it shows as "Running ...")

Minimal Example

mwe.bicep ```bicep param location string = resourceGroup().location resource vnet 'Microsoft.Network/virtualNetworks@2022-01-01' = { name: 'vnet-${uniqueString(resourceGroup().id)}' location: location properties: { addressSpace: { addressPrefixes: [ '10.1.0.0/16' ] } subnets: [ { name: 'default' properties: { addressPrefix: '10.1.0.0/24' } } { name: 'cae' properties: { addressPrefix: '10.1.100.0/23' delegations: [ { name: 'ContainerAppEnvironmentDelegation' properties: { serviceName: 'Microsoft.App/environments' } } ] } } ] } resource defaultSubnet 'subnets' existing = { name: 'default' } resource caeSubnet 'subnets' existing = { name: 'cae' } } resource managementEnvironment 'Microsoft.App/managedEnvironments@2022-06-01-preview' = { name: 'cae-${uniqueString(resourceGroup().id)}' location: location sku: { name: 'Consumption' } properties: { vnetConfiguration: { internal: true infrastructureSubnetId: vnet::caeSubnet.id } } } ```

In the example above, it does not seem to matter whether internal: true or internal: false matters. If I comment out vnetConfiguration is commented out, the deployment succeeds.

hoffmanntravis commented 1 year ago

@JP-Ellis-KPMG That's really helpful to know, at least we can comment out the vnet block and prescribe that portion manually if necessary. Another thing, you can see the error in the 'json' view of the container environment resource which might be helpful for people troubleshooting this issue. This is useful to know since the error is not self evident in the process executing the bicep as far as I know.

You can look at properties.provisioningstate and properties.deploymentErrors of the json (screenshot where to find that below) for more information. It took me a long time to realize those 'deployment in progress errors' that say failed with an unclear message (I don't have one on hand since things are working) actually have a useful error embedded in the json of the resource. The resource group deployment log does not have that detailed information as far as I could tell. And the correlation ID I ran against get-azlog came back null. Fundamentally this information doesn't fix anything but might help others struggling with this issue.

Also worth mentioning, manual creation worked on the first try for me last night with the custom vnet assigned, internal network set to true.

Finally, this is somewhat unrelated, but we have occasionally seen issues with container revisions failing behind the scenes with those silent deployment errors (very rare). In those cases we had to delete the containers (not the container environment) and recreate. I point this out in case there is something going on with container apps more broadly. I haven't raised a ticket on that since we weren't aware of the json field that produces a useful error at the time and I haven't seen that for a few weeks now. The other engineer that saw that said he hadn't seen the error for about six months.

Json field with detailed error info: image

hoffmanntravis commented 1 year ago

This is the error that appears when executing infinitely by the way when you look in resource group deployments for the container environment. I didn't have a sample the other day, but I do today.

{
    "status": "Failed",
    "error": {
        "code": "ResourceDeploymentFailure",
        "message": "The deployment operation status is unknown."
    }
}
hoffmanntravis commented 1 year ago

I did see this error in this latest deploy. Not sure if this is related root cause or not, but we are referencing the insights like this in bicep:

resource containerEnv 'Microsoft.App/managedEnvironments@2022-01-01-preview' = {
  name: buildInfo.Resources.ContainerAppEnvironment.name
  location: buildInfo.Resources.ContainerAppEnvironment.location
  tags: buildInfo.Tags
  properties: {
    appLogsConfiguration: {
      destination: 'log-analytics'
      logAnalyticsConfiguration: {
        customerId: logAnalytics.properties.customerId
        sharedKey: logAnalytics.listKeys().primarySharedKey
      }
    }
    vnetConfiguration: {
      internal: true
      infrastructureSubnetId: ctrlSubnetId
    }
  }
}

Log Analytics is created like this:

resource logAnalytics 'Microsoft.OperationalInsights/workspaces@2021-12-01-preview' = {
  name: buildInfo.Resources.LogAnalytics
  location: location
  tags: buildInfo.Tags
  properties: any({
    retentionInDays: 30
    features: {
      searchVersion: 1
    }
    sku: {
      name: 'pergb2018'
    }
  })
}

buildInfo is a json object that is deserialized as I stated earlier in this thread (not obvious at a glance, also probably not really germaine to the issue but perhaps helpful context)

The Resource 'Microsoft.Insights/components/{azure-insights}' under resource group '{resource-group}' was not found

hoffmanntravis commented 1 year ago

I attempted to create manually in the portal after having a failed deployment via create the resource in a failed state and got the error below (I did that by mistake, but it might be a useful data point): "deploymentErrors": "ErrorCode: ManagedEnvironmentUpgradeError, Message: Update the managed environment failed."

Also, relative to my previous point logAnalytics dependsOn appears superfluous if I add it manually to bicep, but Microsoft.OperationalInsights/workspaces still might be some sort of dependency or race condition etc. image

hoffmanntravis commented 1 year ago

I just noticed @JP-Ellis-KPMG and myself are on preview versions of bicep... :disappointed: will be testing this against general availability Microsoft.App/managedEnvironments@2022-03-01 to see if that improves things

hoffmanntravis commented 1 year ago

We have had two good fresh deploys without any issues using the GA sku over the preview sku in bicep. Not sure that is enough data to say that was the cause for certain, but wanted to share that detail. Previously we were seeing many failures and had not seen clean executions across at least 10 new environment buildouts until we switched skus.

chinadragon0515 commented 1 year ago

For creation failed with invalid subnet Id, several comments,

  1. Make sure you have run "az provider register -n Microsoft.ContainerService" if you are using cli/arm template/bicep
  2. Make sure the subnet is not delegated any service. Azure container apps subnet does require delegation yet.

If you still see issues, let us know.

andyquinterom commented 1 year ago

This is an issue for us, we are unable to deploy container apps from the Azure Portal.

andyquinterom commented 1 year ago

We are trying to deploy from the Azure portal and selecting our own VNet. The deployment status stalls with

{
    "status": "Failed",
    "error": {
        "code": "ResourceDeploymentFailure",
        "message": "The deployment operation status is unknown."
    }
}

We are practically unable to deploy our service.

tedvanderveen commented 1 year ago

For creation failed with invalid subnet Id, several comments,

  1. Make sure you have run "az provider register -n Microsoft.ContainerService" if you are using cli/arm template/bicep
  2. Make sure the subnet is not delegated any service. Azure container apps subnet does require delegation yet.

If you still see issues, let us know.

@chinadragon0515 not sure what you're saying regarding 2): Azure container apps subnet does require delegation yet? Or did you mean to say: Azure container apps subnet does not require delegation yet?

kasattejaswi commented 1 year ago

I believe this issue is still ongoing. I was trying to deploy via the portal and it was timing out with no proper error message. I tried multiple things and finally got to make it work. The primary issue that I found is undocumented.

In the azure document, it is written that any subnet equal to or larger than /23 will work, however, this was not the case. I tried a deployment with /21 subnet in VNET and it didn't fail. I believe this is the solution to the entire issue.

Make sure you have the required providers registered. And after deployment, you will also be needing a private DNS zone to make private endpoints resolvable from other resources.

kasattejaswi commented 1 year ago

We are trying to deploy from the Azure portal and selecting our own VNet. The deployment status stalls with

{
    "status": "Failed",
    "error": {
        "code": "ResourceDeploymentFailure",
        "message": "The deployment operation status is unknown."
    }
}

We are practically unable to deploy our service.

Checkout my above comment. This may fix your issue.

kristerksat commented 1 year ago

Can anyone confirm if you need a /21 subnet for zone redundancy? For us a /23 subnet ends with deployment timeout when adding zone redundancy. Without zone redundancy the same deployment succeeds

pbakun commented 1 year ago

I'm still facing this issue either using Azure Portal or Azure Powershell. The environment creation takes 30 mins and eventually fails. The error message I got in Portal was: {"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"OperationFailed","message":"Managed environment failed to initialize due to managed cluster provision failed"}]}

richardjharding commented 1 year ago

I'm having the same issue - I've tried increasing the subnet to /21 and that made no difference I've also tried deploying via bicep and in the portal

In the bicep example I created the vnet and subnet in advance and in the portal I used the wizard to create a new vnet and subnet (but used an existing log workspace) and it failed after 6-7 minutes with the same error

"Managed environment failed to initialize due to cluster provision failed"

I was able to create the container app environment without the vnet integration in both portal and bicep with no issues

example bicep I was using


resource env 'Microsoft.App/managedEnvironments@2022-10-01' = {
  name: name
  location: location
  properties:{    
    appLogsConfiguration: {
      destination: 'log-analytics'
      logAnalyticsConfiguration: {
        customerId: lawClientId
        sharedKey: lawSecret
      }
    }
    vnetConfiguration: {
      infrastructureSubnetId: infraId
      internal: false
    }
    zoneRedundant: false        
  }  
}```
richardjharding commented 1 year ago

An update - I was able to resolve the deployment failure so posting here in case it helps After reading this comment I checked to see whether this could be the cause of my issue - it seems that for vnet integration the service creates an additional resource group and resources with the resource group name starting with MC_ Our subscription had some policy constraints that enforced certain tags to be present when creating a new resource group and sure enough these had triggered events during my container app deployments When the policy was disabled I was able to create the vnet integrated container app without issue

rpetersson commented 1 year ago

An update - I was able to resolve the deployment failure so posting here in case it helps After reading this comment I checked to see whether this could be the cause of my issue - it seems that for vnet integration the service creates an additional resource group and resources with the resource group name starting with MC_ Our subscription had some policy constraints that enforced certain tags to be present when creating a new resource group and sure enough these had triggered events during my container app deployments When the policy was disabled I was able to create the vnet integrated container app without issue

Can anyone confirm that this is the case? Its very common to have these restrictions on an enterprise environment and if we need to make policy exemptions this should be outlined in the documentation.

gcrockenberg commented 1 year ago

It would be great if the Container Apps Team would post a bicep example demonstrating the successful provisioning of an internal Container Environment with subnet.

Never mind ... bicep reverse engineer on ARM Template from Azure works great! I was able to take it from there.

az bicep decompile --file template.json

muhammad-shameem commented 1 year ago

An update - I was able to resolve the deployment failure so posting here in case it helps After reading this comment I checked to see whether this could be the cause of my issue - it seems that for vnet integration the service creates an additional resource group and resources with the resource group name starting with MC_ Our subscription had some policy constraints that enforced certain tags to be present when creating a new resource group and sure enough these had triggered events during my container app deployments When the policy was disabled I was able to create the vnet integrated container app without issue

-- Yes, even it worked for me, we have a policy in place which denies public ip creation

gpwink commented 3 months ago

Same issue on our subscription. I can create a Container Apps environment on an existing vnet with 2 subnets on the Portal, one default, which is not offered by the UI and a second which is made /23, but when I try to create the same environment from the Azure CLI I get:

(ManagedEnvironmentInvalidNetworkConfiguration) The environment network configuration is invalid: Invalid vnet resource ID provided, or the virtual network could not be found.

No details given on the event log.

I can't really recommend using ContainerApps at the moment for any solution without proper provisioning commands we can use.

If anything, the fix to this problem should be adding details to the errors raised, which are probably a set of multiple issues.

vienleidl commented 3 months ago

So do I have got the same issue. I had to create a brand-new environment and a Container App on Azure portal firstly, then it was succeeded to create with the other command I would like to use like: az containerapp compose create (by default this command doesn't have an argument to create new environment and apps with a VNET).

WEMcJJJ commented 3 months ago

Same issue on our subscription. I can create a Container Apps environment on an existing vnet with 2 subnets on the Portal, one default, which is not offered by the UI and a second which is made /23, but when I try to create the same environment from the Azure CLI I get:

(ManagedEnvironmentInvalidNetworkConfiguration) The environment network configuration is invalid: Invalid vnet resource ID provided, or the virtual network could not be found.

No details given on the event log.

I can't really recommend using ContainerApps at the moment for any solution without proper provisioning commands we can use.

If anything, the fix to this problem should be adding details to the errors raised, which are probably a set of multiple issues.

Same issue here, unable to create a containerapp environment using Azure CLI (latest version 2.60.0) using --infrastructure-subnet-resource-id, I get the following error:

(ManagedEnvironmentInvalidNetworkConfiguration) The environment network configuration is invalid: Invalid vnet resource ID provided, or the virtual network could not be found.

The resource exists and the resource ID that is being used is valid (verified in Azure Resource Explorer).

Greedygre commented 3 months ago

Same issue on our subscription. I can create a Container Apps environment on an existing vnet with 2 subnets on the Portal, one default, which is not offered by the UI and a second which is made /23, but when I try to create the same environment from the Azure CLI I get: (ManagedEnvironmentInvalidNetworkConfiguration) The environment network configuration is invalid: Invalid vnet resource ID provided, or the virtual network could not be found. No details given on the event log. I can't really recommend using ContainerApps at the moment for any solution without proper provisioning commands we can use. If anything, the fix to this problem should be adding details to the errors raised, which are probably a set of multiple issues.

Same issue here, unable to create a containerapp environment using Azure CLI (latest version 2.60.0) using --infrastructure-subnet-resource-id, I get the following error:

(ManagedEnvironmentInvalidNetworkConfiguration) The environment network configuration is invalid: Invalid vnet resource ID provided, or the virtual network could not be found.

The resource exists and the resource ID that is being used is valid (verified in Azure Resource Explorer).

Hi @WEMcJJJ

Can you show me the commands you create the sub-net? Thanks. Or can you show me the delegation of the subnet with this command?: az network vnet subnet show -g {resource-group} -n {subnet-name} --vnet-name {vnet-name}

For --infrastructure-subnet-resource-id, the subnet used should be delegated to Microsoft.App/environmentsaz network vnet subnet create --address-prefixes '14.0.0.0/23' --delegations Microsoft.App/environments -n sub -g {resource_group} --vnet-name {vnet}

Greedygre commented 3 months ago

Same issue on our subscription. I can create a Container Apps environment on an existing vnet with 2 subnets on the Portal, one default, which is not offered by the UI and a second which is made /23, but when I try to create the same environment from the Azure CLI I get:

(ManagedEnvironmentInvalidNetworkConfiguration) The environment network configuration is invalid: Invalid vnet resource ID provided, or the virtual network could not be found.

No details given on the event log.

I can't really recommend using ContainerApps at the moment for any solution without proper provisioning commands we can use.

If anything, the fix to this problem should be adding details to the errors raised, which are probably a set of multiple issues.

Hi @WEMcJJJ @vienleidl

May I ask what is the location the vnet in and what is the command you used to create the containerapp environment? The location should be consistent, need to specify the location with --location: az containerapp env create -g {resource-group} -n {env-name} --location {vnet-location} -s {subnet} --debug

gpwink commented 3 months ago

Same issue on our subscription. I can create a Container Apps environment on an existing vnet with 2 subnets on the Portal, one default, which is not offered by the UI and a second which is made /23, but when I try to create the same environment from the Azure CLI I get: (ManagedEnvironmentInvalidNetworkConfiguration) The environment network configuration is invalid: Invalid vnet resource ID provided, or the virtual network could not be found. No details given on the event log. I can't really recommend using ContainerApps at the moment for any solution without proper provisioning commands we can use. If anything, the fix to this problem should be adding details to the errors raised, which are probably a set of multiple issues.

Hi @WEMcJJJ @vienleidl

May I ask what is the location the vnet in and what is the command you used to create the containerapp environment? The location should be consistent, need to specify the location with --location: az containerapp env create -g {resource-group} -n {env-name} --location {vnet-location} -s {subnet} --debug

The issue is with --infrastructure-subnet-resource-id taking a subnet id looking like a filesystem path. That I fixed with:

MSYS2_ARG_CONV_EXCL="*" az containerapp env create --name "$CONTAINER_ENV" --resource-group "$RESOURCE_GROUP_NAME"...

WEMcJJJ commented 3 months ago

@Greedygre - I followed the Microsoft tutorial here. I modified the variables a little to fit in with work I was already doing. Location was the same for all resources created (westus2).

az network vnet create \
  --resource-group $RESOURCE_GROUP \
  --name $VNET_NAME \
  --location $LOCATION \
  --address-prefix 10.0.0.0/16
  --tags $TAGS

az network vnet subnet create \
  --resource-group $RESOURCE_GROUP \
  --vnet-name $VNET_NAME \
  --name $SUBNET_NAME \
  --address-prefixes 10.0.0.0/23

Here's what I used to try to create the environment:

INFRASTRUCTURE_SUBNET=$(az network vnet subnet show --resource-group ${RESOURCE_GROUP} --vnet-name $VNET_NAME --name $SUBNET_NAME --query "id" -o tsv)

az containerapp env create \
  --name $CONTAINER_ENVIRONMENT \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --infrastructure-subnet-resource-id $INFRASTRUCTURE_SUBNET \

As far as the delegation is concerned, I ran the following command:

az network vnet subnet show -g $RESOURCE_GROUP -n $SUBNET_NAME --vnet-name $VNET_NAME

And there were no delegations listed:

{
  "addressPrefix": "10.0.0.0/23",
  "delegations": [],

I got the same result if I ran the following:

 az network vnet subnet show \
    --resource-group $RESOURCE_GROUP \
    --name $SUBNET_NAME \
    --vnet-name $VNET_NAME\
    --query delegations
[]

I tried your suggestion to add the --delegations option (with and without quotes), when creating the subnet, but either way it said "--delegations: command not found".

az network vnet subnet create \
  --resource-group $RESOURCE_GROUP \
  --vnet-name $VNET_NAME \
  --name $SUBNET_NAME \
  --address-prefixes 10.0.0.0/23
  --delegations "Microsoft.App/environments"

I did some more digging and tried some suggestions here, and the following worked (not sure why it wouldn't work with the create command):

az network vnet subnet update \
    --resource-group $RESOURCE_GROUP \
    --name $SUBNET_NAME \
    --vnet-name $VNET_NAME\
    --delegations Microsoft.App/environments

After running the above, it now shows the delegation if I query it:

az network vnet subnet update \
    --resource-group $RESOURCE_GROUP \
    --name $SUBNET_NAME \
    --vnet-name $VNET_NAME\
    --query delegations
[
  {
    "actions": [
      "Microsoft.Network/virtualNetworks/subnets/join/action"
    ],
    "etag": "W/\"xxxxx-xxxxx-xxxxx-xxxxx\"",
    "id": "/subscriptions/xxxxx-xxxxx-xxxxx-xxxxx/resourceGroups/<rgName>/providers/Microsoft.Network/virtualNetworks/<vnetName>/subnets/infrastructure-subnet/delegations/0",
    "name": "0",
    "provisioningState": "Succeeded",
    "resourceGroup": "<rgName>",
    "serviceName": "Microsoft.App/environments",
    "type": "Microsoft.Network/virtualNetworks/subnets/delegations"
  }
]

However, I am still not able to create the environment:

az containerapp env create \
  --name $CONTAINER_ENVIRONMENT \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --infrastructure-subnet-resource-id $INFRASTRUCTURE_SUBNET

No Log Analytics workspace provided.
Generating a Log Analytics workspace with name "workspace-<rgName>"
(ManagedEnvironmentInvalidNetworkConfiguration) The environment network configuration is invalid: Invalid vnet resource ID provided, or the virtual network could not be found.

I verified that the resource ID appears to be correct:

echo $INFRASTRUCTURE_SUBNET
/subscriptions/xxxxx-xxxxx-xxxxx-xxxxx/resourceGroups/<rgName>/providers/Microsoft.Network/virtualNetworks/<vNetName>/subnets/infrastructure-subnet

As previously stated, I'm running the latest version of Azure CLI:

az version
{
  "azure-cli": "2.60.0",
  "azure-cli-core": "2.60.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {}
}

az upgrade
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
You already have the latest azure-cli version: 2.60.0
Upgrade finished.You can enable auto-upgrade with 'az config set auto-upgrade.enable=yes'. More details in https://docs.microsoft.com/cli/azure/update-azure-cli#automatic-update

Just to double-check, I ran the exact same command to create the environment in Azure Cloud Shell, and it worked...go figure. Not sure why it works in Azure Cloud Shell and not locally.

@gpwink - not sure I follow, I tried running the following and still get the same error (even tried putting each variable in double quotes (i.e. "$INFRASTRUCTURE_SUBNET"):

MSYS2_ARG_CONV_EXCL="*" az containerapp env create \
  --name $CONTAINER_ENVIRONMENT \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --infrastructure-subnet-resource-id $INFRASTRUCTURE_SUBNET \

I even tried manually putting in the subnet resource ID without a leading "/" and with a a double "//" at the beginning and that didn't make any difference, still got an error.

Greedygre commented 3 months ago

INFRASTRUCTURE_SUBNET=$(az network vnet subnet show --resource-group ${RESOURCE_GROUP} --vnet-name $VNET_NAME --name $SUBNET_NAME --query "id" -o tsv)

Hi @WEMcJJJ

Can you execute the az containerapp env create command with --debug? I want to check the value of --infrastructure-subnet-resource-id in the command arguments. I checked some logs and found there is some invalid string included in it when throw this error, please help to confirm, thanks! Just like this:

"infrastructureSubnetId": "C:/Program Files/Git/subscriptions/xxxxx-xxxxx-xxxxx-xxxxx/resourceGroups/<rgName>/providers/Microsoft.Network/virtualNetworks/<vNetName>/subnets/infrastructure-subnet
",

I can repro it when I using Gitbash in windows. The command arguments like this:

$ az containerapp env create -g <rg> -n <env-name> --location northcentralus  --logs-destination none --infrastructure-subnet-resource-id $INFRASTRUCTURE_SUBNET  --debug
DEBUG: cli.knack.cli: Command arguments: ['containerapp', 'env', 'create', '-g', '<rg>', '-n', '<env-name>', '--location', 'northcentralus', '--logs-destination', 'none', '--infrastructure-subnet-resource-id', 'C:/Program Files/Git/subscriptions/xxxxx-xxxxx-xxxxx-xxxxx/resourceGroups/<rgName>/providers/Microsoft.Network/virtualNetworks/<vNetName>/subnets/infrastructure-subnet', '--debug']
WEMcJJJ commented 3 months ago

INFRASTRUCTURE_SUBNET=$(az network vnet subnet show --resource-group ${RESOURCE_GROUP} --vnet-name $VNET_NAME --name $SUBNET_NAME --query "id" -o tsv)

Hi @WEMcJJJ

Can you execute the az containerapp env create command with --debug? I want to check the value of --infrastructure-subnet-resource-id in the command arguments. I checked some logs and found there is some invalid string included in it when throw this error, please help to confirm, thanks! Just like this:

"infrastructureSubnetId": "C:/Program Files/Git/subscriptions/xxxxx-xxxxx-xxxxx-xxxxx/resourceGroups/<rgName>/providers/Microsoft.Network/virtualNetworks/<vNetName>/subnets/infrastructure-subnet
",

I can repro it when I using Gitbash in windows. The command arguments like this:

$ az containerapp env create -g <rg> -n <env-name> --location northcentralus  --logs-destination none --infrastructure-subnet-resource-id $INFRASTRUCTURE_SUBNET  --debug
DEBUG: cli.knack.cli: Command arguments: ['containerapp', 'env', 'create', '-g', '<rg>', '-n', '<env-name>', '--location', 'northcentralus', '--logs-destination', 'none', '--infrastructure-subnet-resource-id', 'C:/Program Files/Git/subscriptions/xxxxx-xxxxx-xxxxx-xxxxx/resourceGroups/<rgName>/providers/Microsoft.Network/virtualNetworks/<vNetName>/subnets/infrastructure-subnet', '--debug']

@Greedygre - looks like that's the issue. I'm using Gitbash in Windows and the argument gets modified:

az containerapp env create \
  --name $CONTAINER_ENVIRONMENT \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --infrastructure-subnet-resource-id $INFRASTRUCTURE_SUBNET \
  --debug
cli.knack.cli: Command arguments: ['containerapp', 'env', 'create', '--name', '<envName>', '--resource-group', '<rgName>', '--location', 'westus2', '--infrastructure-subnet-resource-id', 'C:/Program Files/Git/subscriptions/<SubID>/resourceGroups/<rgName>/providers/Microsoft.Network/virtualNetworks/<VNetName>/subnets/infrastructure-subnet', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
Greedygre commented 3 months ago

@Greedygre - looks like that's the issue. I'm using Gitbash in Windows and the argument gets modified: @WEMcJJJ

The argument got modified by Git Bash, Git Bash will convert some paths to POSIX paths. You can use powershell or CMD.

WEMcJJJ commented 3 months ago

@Greedygre - looks like that's the issue. I'm using Gitbash in Windows and the argument gets modified: @WEMcJJJ

The argument got modified by Git Bash, Git Bash will convert some paths to POSIX paths. You can use powershell or CMD.

What a pain. :( @Greedygre, thanks for helping me figure it out though!

LeighS commented 1 month ago

As a heads up. The issue for me is related to how the Virtual network is created. You can define the virtual network with either an addressPrefix or addressPrefixes block.

If you use an address prefixes block on your VNET like this:

It appears that the container apps environment will fail validation with the following message.
 

 performing CreateOrUpdate: unexpected status 400 with error: ManagedEnvironmentInvalidNetworkConfiguration: The environment network configuration is invalid: The subnet and its addressPrefix could not be found.

If you recreate your VNET specifying only an addressPrefix it will pass the validation and create the container app enviroment.

lukemurraynz commented 1 month ago

As a heads up. The issue for me is related to how the Virtual network is created. You can define the virtual network with either an addressPrefix or addressPrefixes block.

If you use an address prefixes block on your VNET like this:

It appears that the container apps environment will fail validation with the following message.  

 performing CreateOrUpdate: unexpected status 400 with error: ManagedEnvironmentInvalidNetworkConfiguration: The environment network configuration is invalid: The subnet and its addressPrefix could not be found.

If you recreate your VNET specifying only an addressPrefix it will pass the validation and create the container app enviroment.

100% - this question was asked in Microsoft Q&A and I dod some troubleshooting and came to the same conclusion; I raised a blog post: https://luke.geek.nz/azure/containerapps-env-missing-subnet/