Overview

Currently provider configuration and build of resources is all within the KFP-Operator helm install, this makes changing/adding new providers cumbersome and it should be split into its own custom resource which then controller for that will manage the creation/update/deletion of associated resources.

Work has already started on this on this branch: https://github.com/sky-uk/kfp-operator/tree/provider-crd

Acceptance Criteria

[x] The existing provider ConfigMap should no longer exist
[x] A custom resource for the newly defined CRD should replace the ConfigMap
[x] When a Pipeline is created/updated, the workflow uses the custom resource and not the ConfigMap
[x] When a RunConfiguration is created/updated, the workflow uses the custom resource and not the ConfigMap
[x] When a RunSchedule is created/updated, the workflow uses the custom resource and not the ConfigMap
[x] When a Run is created/updated, the workflow uses the custom resource and not the ConfigMap

Tasks

[x] Define CRD (CustomResourceDefinition)
[x] Custom resources will be created as part of the Operator helm chart (replacing the creation of the config map). Leave values for the provider CRD in the helm values.
[x] Load configuration from the custom resource rather than mounting the config map
[x] Configuration is passed from operator into workflow as raw data mounted in volume
[x] Update documentation

Technical Details

See:

Example CR:

apiVersion: pipelines.kubeflow.org/v1alpha5
kind: Provider
metadata:
  name: provider-name (e.g. kfp/vai)
spec:
  image: provider-image-url:0.0.1
  executionMode: v1
  serviceAccount: kfp-operator
  defaultBeamArgs:
  - name: arg1
    value: value1
  pipelineRootStorage: file:///directory
  parameters:
    pipelineBucket: bucket-path
    etc: etc

sky-uk / kfp-operator

Create new custom resource for provider config and remove ConfigMap #329

Overview

Acceptance Criteria

Tasks

Technical Details