[Proposal] Extend Multi app run for k8s in dev/test environment

[mukundansundar]

Describe the proposal

This is a proposal to extend multi app run dapr run -f to include k8s deployments in the dev/test environments.

Goal: Run the given prebuilt user container with the right annotations and configurations where the initial values are picked from the multi-app run template which can also be used for self-hosted mode.

Release Note

RELEASE NOTE:

[mukundansundar]

Multi-app run for DAPR Kubernetes mode

Introduction

Currently for the self-hosted mode in Dapr, we have a command dapr init to initialize Dapr with Redis and Zipkin, we also have a way to run multiple apps with a single command dapr run -f.

With these two options in place, it is easy for the developers to quickly iterate on their apps in the self-hosted mode. When they want to transition to using Kubernetes, they have to do a lot of manual work to get all prerequisites setup.

For example, they would have to create Redis and Zipkin deployments, create a Dapr component for Redis and Zipkin, and create a deployment for each of their apps with right annotations. This is a lot of work and it is error prone. This design is to enhance the dapr run command to support running multiple apps in Kubernetes for a dev/test scenario alone.

Goals

• Enhance the dapr init -k command to support creating Redis and Zipkin deployments in Kubernetes.
• Enhance the dapr run -f command to support running multiple apps in Kubernetes for a dev/test scenario alone.
• Enhance the dapr stop -f command to delete the applications run using dapr run -f command.
• Enhance the dapr uninstall -k command to delete the Redis and Zipkin deployments created using dapr init -k enhanced command.

Non-Goals

This design is only for dev/test environment and not for production.

Design

Enhance dapr init -k command

For installing the Redis and Zipkin containers in self-hosted mode, dapr init automatically downloads the relevant Redis and Zipkin containers and if necessary, initializes them in a docker network if one is provided.

For Kubernetes mode though, when dapr init -k is run, it will install Dapr in the cluster in the default namespace of dapr-system or the user provided namespace with the -n flag.

With an additional --devel, dapr init -k --devel, install Redis and Zipkin. Since this command is recommended to run only in development/test setups, we will install Redis and Zipkin to kubernetes default namespace. If -n flag is also added in the command, then the Dapr control plane will be installed to user specified namespace, the Redis and Zipkin deployments will still be in the default namespace.

Optionally an additional switch --data-plane-namespace flag can be provided to let the user define the namespace to install Redis and Zipkin in.

On dapr init -k --devel, we need to create Dapr components where the pubsub and statestore components refer to the Redis and Zipkin deployments in the correct namespace. The configuration CRD should also be created in the same namespace as well.

The steps taken on dapr init -k --devel

• Create Redis and Zipkin when run with --devel flag in the default namespace.
• (Optional) Provide a --data-plane-namespace flag to allow users to customize the namespace for Redis and Zipkin deployments and their component creations also.
• Create the Redis and Zipkin components and the configuration CRD to allow tracing to work with the installed Zipkin.
• (Optional) Create other Dapr components such that all of the dapr/quickstarts work out of the box in Kubernetes mode.

Enhance dapr run -f command

Add a -k flag to enhance the dapr run -f command to translate the multi-app run template that is taken as input into a Kubernetes deployment YAML with proper annotations to enable Dapr.

Introduce a containerImage field for each app such that the container image to be used can be specified in the multi-app run template. If this field is not specified, dapr run -f -k will error out.

Some fields from the multi-app run template will be ignored for the Kubernetes run as the paths defined will be for self-hosted mode and cannot be directly used in Kubernetes.

• resourcesPaths
• configFilePath
• runtimePath
• command

Introduce a createService field, which will be used for defining a basic Service(either ClusterIP or LoadBalancer) in Kubernetes with which targets the --app-port defined in the multi-app run template. This will allow the app to be accessed from outside the cluster. If this field is not specified, the app will not be accessible from outside the cluster.

For dapr run -f -k command, the following steps will be performed:

• Create a Kubernetes Deployment for each app in the multi-app run template using the containerImage field.
• If app-port is specified, create a Kubernetes containerPort field for the container.
• If createService is specified, create a Kubernetes Service (LoadBalancer/ClusterIP) for the app.
• Default annotations as needed for the app, dapr.io/enabled and dapr.io/app-id will be added to the Deployment. All other fields in multi-app run template will be converted to Kubernetes annotations as available.
• The deployment will be a single replica deployment.
• The environment variables resolved for an app will be set as environment variables in the container.
• Generate a file in .dapr/deploy folder within each application folder with the Kubernetes Deployment YAML for the app named as dapr_<app-id>.yaml. Each time dapr run -f -k is run, this file will be overwritten.
• Apply the YAMLs present in the <appPath>/.dapr/deploy folder to the Kubernetes cluster.

Note: The assumption here is that the apps will use the default components created for pubsub and statestore and the tracing configuration created during the previous dapr init -k –devel run. Currently there is no way to override the components and configuration for the apps.

(Optional Let users define those components in <appPath>/.dapr/deploy folder, and when dapr run -f -k is run, it will also deploy those components and configurations.

Sample template

version: 1
common: # optional section for variables shared across apps
  env:  # any environment variable shared across apps
    DEBUG: true
apps:
  - appID: webapp # optional
    appDirPath: .dapr/webapp/ # REQUIRED
    appProtocol: http
    appPort: 8080
    appHealthCheckPath: "/healthz"
    createService: true
    containerImage: "docker.io/username/webapp:latest"

Initially the logs will be part of the Kubernetes containers alone. We can explore the option of streaming the logs to a file or console similar to the way we stream the container logs in E2E tests in Dapr.

(Optional) Add a namespace field to the multi-app run command

Add a -n, --namespace flag to the dapr run -f -k . command. This will allow deploying the apps in a namespace other than the default namespace.

Enhance dapr stop -f command

Create a dapr stop -f -k command to delete the Kubernetes Deployment created by dapr run -f -k command.

Enhance dapr uninstall -k command

For the dapr uninstall -k –all command, both the Redis and Zipkin deployments will be deleted. If the --data-plane-namespace flag was used during dapr init -k –devel run, then the Redis and Zipkin deployments will be deleted from that namespace if the flag is provided.

Only for this command run, will the default component and configurations files be deleted.

Scope

MVP Scope:

• Changes to the dapr init -k with the --devel flag
• Changes to dapr run -f
• Changes to dapr stop -f to delete deployed apps
• Changes to dapr uninstall -k to delete resources created on --devel install

Not in Scope

• Containerization of apps
• Having multiple replicas for the apps
• Modifying the deployment template by use of metadata fields or such in the multi-app run template

Open Questions

• Should dapr status -k show the Redis and Zipkin installations?
• Should daprd also be exposed as a service? If so, how do we handle the port mapping?

[pravinpushkar]

createService: true containerImage: "docker.io/username/webapp:latest"

@mukundansundar Should we also add imagePullPolicy to the template ?

[mukundansundar]

redis statestore component name: daprstatestore redis pubsub component name: daprpubsub app configuration name: daprzipkinconfig

[mukundansundar]

@paulyuk The issue here is, in self-hosted mode, the names of the components are plain statestore and pubsub .... But in k8s they will be daprstatestore and daprpubsub .... Here the quickstarts will be affected wherein quickstarts would have to accept an environment variable to resolve the component name ....

[mukundansundar]

redis statestore component name: daprstatestore redis pubsub component name: daprpubsub app configuration name: daprzipkinconfig

based on further discussions .... This is going to change to the following All components and configurations will be in the default namespace only for the --dev flag

redis statestore component name: statestore redis pubsub component name: pubsub app configuration name: appconfig

If any of the components or configurations are already present, print a warning statement during init that it is already present and that it will not be overwritten.

[paulyuk]

Thank you @mukundansundar . I believe for now it will help to keep the default redis statestore and pubsub name the same so the same app can run in either standalone mode or kube mode without being changed. Having errors for naming conflicts in the default namespace I hope will mitigate your concern. I'd love to get the communities feedback on this too.