Azure / azure-dev

A developer CLI that reduces the time it takes for you to get started on Azure. The Azure Developer CLI (azd) provides a set of developer-friendly commands that map to key stages in your workflow - code, build, deploy, monitor, repeat.
https://aka.ms/azd
MIT License
406 stars 195 forks source link

[Issue] Allow skipping deployment validation while deploying Kubernetes resources #1909

Open sabbour opened 1 year ago

sabbour commented 1 year ago

Some Kubernetes deployments, for example using Azure Service Operator to create an Azure Cache for Redis take a considerable amount of time, upwards of 10 minutes. If a deployment is using a secret from Azure Service Operator, it will remain in a not ready state until such provisioning is complete.

Meanwhile, azd deploy will eventually timeout.

We should allow the option to skip deployment validation while deploying Kubernetes resources. Kubernetes reconciliation will mean that eventually, the deployment will be running properly.

Deploying services (azd deploy)

  (x) Failed: Deploying service backend

ERROR: deploying service: failed deploying service 'backend': failing invoking action 'deploy', failed publishing service 'backend': failing invoking action 'publish', failed waiting for resource, resource 'deployment' is not ready, resource is not ready
wbreza commented 1 year ago

@sabbour I could see us addressing this in a couple different ways.

  1. Increase the timeout or make it configurable
  2. Provide configuration to opt-out of resource validation

I would prefer we enable both but stay with the default of attempting to wait/validate resources when deployment has completed.

sabbour commented 1 year ago

I actually think there's more to this issue, specifically around dependencies.

Let's say azure.yml defines two services:

Frontend has a liveness probe that will only be Ready when it can connect to the Backend.

Today, it seems there is no guarantee on deployment ordering, so it could very well happen that azd deploy would wait (until it times out) on the Frontend to be ready before it goes to deploy the Backend and the entire deployment fails.

I would suggest having a hybrid approach like how Bicep can execute deployments in parallel as long as there is no explicit dependencies between them, and like how Kubernetes operates with eventual consistency/reconciliation loop in place.

wbreza commented 1 year ago

@sabbour Thanks for the great feedback!

I actually think there's more to this issue, specifically around dependencies.

Let's say azure.yml defines two services:

  • Frontend
  • Backend

Frontend has a liveness probe that will only be Ready when it can connect to the Backend.

Yes, this is another great use case that we will need to consider.

Today, it seems there is no guarantee on deployment ordering, so it could very well happen that azd deploy would wait (until it times out) on the Frontend to be ready before it goes to deploy the Backend and the entire deployment fails.

100% agree - dependency ordering is in our backlog to allow better granular support for these types of scenarios.

I would suggest having a hybrid approach like how Bicep can execute deployments in parallel as long as there is no explicit dependencies between them, and like how Kubernetes operates with eventual consistency/reconciliation loop in place.

Parallel execution is also in our backlog :)