Failed to provision container app

SoucianceEqdamRashti commented 2 years ago

Hello,

I have tried to deploy my container to an environment based on an image from my acr. That fails and the main message I get is this: Deployment failed. Correlation ID: 28704c6f-44c8-447f-aa73-787016e2f2f8. Failed to provision revision for container app 'containerappdemo3'. Error details: Code: 90714, ExtendedCode: 90714, Message: Error provisioning the container app. Container 'containerappdemo3' failed to start-up. Please check Log Analytics workspace for container logs..

When I look in log analytics this is what I see. I cannot really tell from the logs what has gone wrong. And no where in the portal is there is any good indication of why the container instance didn't provision. Note that the container itself is just a web api template for the weather forecast. Nothing more than that. I have tried to create the app from the az cli as well as the portal and the same issue occurs.

If I provision nginix from dockerhub that works fine. Would appreciately some hint on what has gone wrong. Thanks.

`

Go to Log Analytics and run query

`

inuvano-labs commented 2 years ago

Can you double check your deployment? In the above log the only registries I see in "ContainerImage_s" are:

docker.io
mcr.microsoft.com

I would expect to see your ACR there?

kendallroden commented 2 years ago

@SoucianceEqdamRashti hi there! What is the application doing? Are you making use of TCP? We are still working on the TCP support in Container Apps, based on the logs it looks like that could be the case. Let me know!

SoucianceEqdamRashti commented 2 years ago

@kendallroden , the application isn't doing anything special. I just cloned the sotnet 6 webapi template with docker support and then tried to deploy it to azure container apps.

I think I know what the issue is but strange that is does not appear in the logs. I checked the box for https support. Running it in vs studio worked since it generates and accepte the self signed cert. But I didnt so it when running the container manually in the dockerfile. So I suspect that is the issue. But I would expect thatto be seen in the server logs..instead just error that the container does not start.

kevinruder commented 2 years ago

I had the same error @SoucianceEqdamRashti when trying to deploy my python FastAPI application.

Found this in log analytics:

msg_s :

| error connecting to tcp address

error_s:

| dial tcp [::1]:80: connect: connection refused

@kendallroden could this be related to the TCP support which you mentioned?

SoucianceEqdamRashti commented 2 years ago

I have tried to now deploy a container app (dotnet 6) with and without https and same issue.

If I run it locally with docker compose then it works fine.

It would be nice if there was a tutorial that went through the steps of pushing a asp.net core web api with and without https to container apps just to see if there is something missing in my steps nor not.

SoucianceEqdamRashti commented 2 years ago

Here is another example: https://my-container-app8.wonderfulsand-400d92f5.northeurope.azurecontainerapps.io/swagger/index.html I deployed it via the cli and get back a fqdn. But the response back is just 404 on any path.

I ran: az containerapp create --name my-container-app8 --resource-group $RESOURCE_GROUP --image $CONTAINER_IMAGE_NAME --environment $CONTAINERAPPS_ENVIRONMENT --registry-login-server $REGISTRY_LOGIN_SERVER --registry-username $REGISTRY_USERNAME --registry-password $REGISTRY_PASSWORD --target-port 80 --ingress 'external' --query configuration.ingress.fqdn

All it does is login to my ACR and pulls the image..

inuvano-labs commented 2 years ago

@SoucianceEqdamRashti I'm running a web app in the same region (albeit on different port to you) and works fine for me.

What does the logs for the container show under the LA workspace? Perhaps something is failing on startup?

inuvano-labs commented 2 years ago

I had the same error @SoucianceEqdamRashti when trying to deploy my python FastAPI application.

Found this in log analytics:

msg_s :

| error connecting to tcp address

error_s:

| dial tcp [::1]:80: connect: connection refused

@kendallroden could this be related to the TCP support which you mentioned?

are you seeing this from the probe-shim container or the application container in the LA workspace? I seem numerous connection refused errors, but I think this is just down to liveness/readiness probing.

JeroenvdBurg commented 2 years ago

be sure to set the port you expose in the docker container to the same port on your ingress settings. In my case port 3000

ffrankozz commented 2 years ago

I have the same issue I think. I am deploying through the portal. I have provisioned a resource group in N. Europe. I can deploy a web api whose image is in docker hub However, I cannot deploy a simple blazor app whose image is also in docker hub. I have created the image via Visual Studio code. Container app is created but during deployment phase I get conflict error. When looked into the logs: I read: "error connecting to tcp address" I then spotted from querying the logs that it is complaining about being "unable to start Kestrel server". Unable to start Kestrel. |
System.Net.Sockets.SocketException (13): Permission denied

Any idea why this my be happening for Blazor app?

timoknapp commented 2 years ago

I had the same error @SoucianceEqdamRashti when trying to deploy my python FastAPI application. Found this in log analytics: msg_s : | error connecting to tcp address error_s: | dial tcp [::1]:80: connect: connection refused @kendallroden could this be related to the TCP support which you mentioned?

are you seeing this from the probe-shim container or the application container in the LA workspace? I seem numerous connection refused errors, but I think this is just down to liveness/readiness probing.

I see the exact same issue in my logs when starting one of my containers (https://github.com/microsoft/azure-container-apps/issues/118). The errors are always thrown by the "probe-shim" complaining about not being able to create a TCP connection (dial tcp [::1]:8080: connect: connection refused). While looking at my logs I can clearly see that my application (Spring Boot) is still in the startup phase while "probe-shim" is already complaining about not being able to establish a connection.

So my best guess is actually that the initialDelaySeconds (see K8s docs) config for the readiness/liveness probes is not configured ideally within the underlying K8s environment of Container Apps.

I tested it also with a Go-based container and in here it works out just fine. So I assume that this issue only occurs with 'non lightweight' applications which need more time while starting.

I guess this could be fixed by allowing custom probe configuraton, as described in (#118)

@kendallroden Are you referring to TCP support for readiness/liveness probes or do you mean allowing TCP connections within Container Apps in general?

mrentier commented 1 year ago

I am seeing a very similar issue. Took the standard weather API and deployed it with following bicep:

resource logAnalyticsWorkspace 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
  name: logAnalyticsWorkspaceName
  location: location
  properties: any({
    retentionInDays: 30   
    features: {
      searchVersion: 1
    }
    sku: {
      name: 'PerGB2018'
    }
  })
}

resource applicationInsights 'Microsoft.Insights/components@2020-02-02' = {
  name: appInsightsName
  location: location
  kind: 'web'
  properties: {
    Application_Type: 'web'
    WorkspaceResourceId: logAnalyticsWorkspace.id
  }
}

resource containerAppsEnvironment 'Microsoft.App/managedEnvironments@2022-03-01' = {
  name: containerAppsEnvironmentName
  location: location
  properties: {    
    appLogsConfiguration: {
      destination: 'log-analytics'
      logAnalyticsConfiguration: {
        customerId: logAnalyticsWorkspace.properties.customerId
        sharedKey: logAnalyticsWorkspace.listKeys().primarySharedKey
      }
    }
  }
}

and module:

param location string
param uniqueSeed string
@description('The name of the containerApp to be created.')
param appName string  = 'weather-api-${uniqueString(uniqueSeed)}'
param containerAppsEnvironmentId string
param registryServer string
param registry string
param containerRegistryUsername string
@secure()
param containerRegistryPassword string

var containerRegistryPasswordRef = 'container-registry-password'

resource containerApp 'Microsoft.App/containerApps@2022-03-01' = {
  name: appName
  location: location
  properties: {
    managedEnvironmentId: containerAppsEnvironmentId
    template: {     
      containers: [
        {
          name: 'weatherforecast-api'
          image: '${registryServer}/${registry}/weatherforecast.api:dev'
          resources: {
            cpu: '0.5'
            memory: '1Gi'
          }       
          probes: [
          {
            type: 'liveness'
            initialDelaySeconds: 15
            periodSeconds: 15
            failureThreshold: 3
            successThreshold: 1
            httpGet: {
              port: 80
            }
          }              
          {
            type: 'readiness'
            initialDelaySeconds: 15
            failureThreshold: 25
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 5
            httpGet: {
              port: 80
            }
          }
          {
            type: 'startup'
            initialDelaySeconds: 15
            failureThreshold: 25
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 5
            httpGet: {
            port: 80
          }
        }]   
      }]
      scale: {
        minReplicas: 1
        maxReplicas: 5
        rules: [
            {
            name: 'http-rule'
            http: {
              metadata: {
                concurrentRequests: '50'
              }
            }
            }
          ]
      }
    }
    configuration: { 
     secrets: [
        {
          name: containerRegistryPasswordRef
          value: containerRegistryPassword
        }     
      ]        
      registries: [
        {
          server: registryServer
          username: containerRegistryUsername
          passwordSecretRef: containerRegistryPasswordRef
        }
      ]     
      activeRevisionsMode: 'single'
      dapr: {
        enabled: false
        //appId: 'weatherforecast-api'
        //appPort: 80
      }
      ingress: {
        external: true
        targetPort: 80
      }
    }
  }
}

The container is never provisioned. Only error I am seeing in the logs is the probes timing out.

edgarhuichen commented 7 months ago

I have got a similar issue. I have a FastAPI-based container app that accepts HTTP requests, processes some data, and returns a response. My app is running a consumption plan with HTTP scaling rules. When a new replica is created, it can't start the app instance with this error: azure container apps readiness probe failed: connection refused. As a result, the request times out.

microsoft / azure-container-apps

Failed to provision container app #72