Pull-based deploy to Kubernetes clusters

martinothamar commented 5 months ago

Status

Proposed

Context

While working on Maskinporten integration and automation in Altinn 3, we also discussed more general improvements around app clusters and integration with Altinn Studio. This is something that has been discussed before in the Platform team.

Azure DevOps (ADO) pipelines are currently used for deployment from Studio to service owner app clusters. When users click the deploy button in Altinn Studio, a pipeline is run that is parameterized by Altinn Studio backend code to target the correct environment, git-repo, cluster and Azure subscription among other things.

The pipeline

Pulls the app repo locally
Initializes a helm values file for environment-specific configuration
- Known based on input parameters from Studio and/or variables in ADO
Runs helm upgrade

In the future we will be dynamically provisioning per-app infrastructure using Kubernetes operators (some of which won't live in the app Git repo), which will lead to more configuration that needs to work well at deployment-time. For instance, the configuration for which Maskinporten scopes the app needs will be mutated at design-time, but not stored in Gitea. So pulling the repo in the pipeline is not sufficient to build the whole artifact.

There are some drawbacks to the ADO approach

Complexity of GUI-based release pipelines based on shell scripts
Need PAT token / credentials to deploy from Studio
Inflexible configuration options (ADO pipeline parameters, which are flat key-value pairs)
Unclear history, since there is a single pipeline running deployments for all apps

Decision

Rather than extending ADO pipelines to accommodate our new configuration needs, we suggest that Altinn Studio

Builds the complete artifact directly, containing
- Code from the Gitea repo
- Static configuration from the Gitea repo
- Dynamic configuration stored in Studio PostgreSQL DB
Uploads the artifact to a Azure Blob Storage container
- Using Managed Identity (MI)/Workload Identity (WI) to avoid credentials
TBD: when an app is created, the corresponding Flux config has to be created in k8s, which means 1 blob container per app probably

Then in the app clusters we can

Configure Flux to pull from Azure Blob Storage
- Using MI/WI

Building the artifact could consist of

Create values.yaml files from dynamic configuration that is not in the Gitea repo
Render the helm-chart (?), create a flux HelmRelease or similar
Zip the output

For more concrete technical information, read the platform team RFC: https://github.com/Altinn/altinn-platform/blob/rfc/pull-based-cd/rfcs/0000-pull-based-cd.md

Consequences

Less logic in ADO pipelines
Less credentials (due to Azure MI/WI)
More logic and complexity (including knowledge of environments) in Altinn Studio
Altinn Studio still has to tell Flux where to pull from
GitOps is a paradigm shift, there is no GUI pipeline anymore, and there are different tools to view the internal state of things (but Altinn Studio will still have the same list of release/deploy information in the UI)

martinothamar commented 5 months ago

cc @sduranc @bengtfredh - part 2 of k8s/platform related ADRs. Feel free to correct and rewrite as needed

bengtfredh commented 5 months ago

@martinothamar Have a look at this rfc. https://github.com/Altinn/altinn-platform/blob/rfc/pull-based-cd/rfcs/0000-pull-based-cd.md It is focused around deploy platform components, but we want to extend it to cover service owner apps the same way.

altinnadmin commented 5 months ago

@martinothamar Nice! I think we should change the title of this decision to reflect the main architectural change: Pull-based deploy to Kubernetes app clusters.

martinothamar commented 5 months ago

Have a look at this rfc.

Cool! Added the "For more concrete technical information, read the platform team RFC..." paragraph with a link

Nice! I think we should change the title of this decision to reflect the main architectural change

Done

sduranc commented 2 months ago

LGTM. Although, it sounds more like it's specific for Altinn Studio.

Uploads the artifact to a Azure Blob Storage container

Currently we're pushing the OCI artifacts to ACR, using OCI repositories. But that's just implementation details, so it shouldn't matter, ie. we could also end up pushing the OCI artifact to buckets in some point, if that makes sense.

altinnadmin commented 2 months ago

LGTM. Although, it sounds more like it's specific for Altinn Studio.

@sduranc Yup, we agreed to make this more general, like #6, and review it next week.

Altinn / altinn-decision-log