[bug] Recurring runs not working after re-installing kubeflow-pipelines modules

[saemaromoon]

Environment

• How do you deploy Kubeflow Pipelines (KFP)?

Using Kubeflow 1.5.1 on EKS and it is updated from version 1.3.1 I did kustomize build apps/pipeline | kubectl delete -f - and kustomize build apps/pipeline | kubectl apply -f -

registering a pipeline and calling a run works fine but registered recurring runs not working

• KFP version: gcr.io/ml-pipeline/api-server:1.8.2 gcr.io/ml-pipeline/metadata-writer:1.8.2 gcr.io/ml-pipeline/persistenceagent:1.8.2 gcr.io/ml-pipeline/scheduledworkflow:1.8.2 gcr.io/ml-pipeline/frontend:1.8.2 gcr.io/ml-pipeline/viewer-crd-controller:1.8.2 gcr.io/ml-pipeline/visualization-server:1.8.2 gcr.io/ml-pipeline/workflow-controller:v3.2.3-license-compliance gcr.io/tfx-oss-public/ml_metadata_store_server:1.5.0 gcr.io/ml-pipeline/cache-server:1.8.2 gcr.io/ml-pipeline/metadata-envoy:1.8.2

• KFP SDK version:

Steps to reproduce

Hi, is there anyone experienced registered recurring runs not working after updating and installing kubeflow-pipeline? I re-deployed kubeflow-pipeline to update the version from 1.3.1 to 1.5.1 Recurring runs registered previously are not being updated but I see new recurring runs are being updated normally

Tested query: SELECT uuid, displayname, name, namespace, serviceaccount, description, from_unixtime(CreatedAtInSec) CreatedAtInSec, from_unixtime(UpdatedAtInSec) UpdatedAtInSec, from_unixtime(CronScheduleStartTimeInSec) CronScheduleStartTimeInSec, from_unixtime(CronScheduleEndTimeInSec) CronScheduleEndTimeInSec FROM mlpipeline.jobs; It seems like a certain demon in pipeline module misses records to trace.

Expected result

registered recurring runs updated and triggered normally after upgrading kubeflow-pipeline module

Materials and reference

Labels

---

Impacted by this bug? Give it a 👍.

[saemaromoon]

I figured out that registering a recurring run creates a scheduled workflow crd instance. Removing the pipeline modules from the k8s cluster, also destroyed those crd instances, so the remaining metadata is meaningless. Is there a safer and more efficient way to upgrade the kubeflow-pipeline module without losing those crd objects?

[midhun1998]

@saemaromoon you are doing a clean install of the new version which should ideally behave as you told(CRD getting removed). But in order to upgrade the existing installation, you can just run kustomize build apps/pipeline | kubectl apply -f - with updated manifests n the directory and it should work without any problems from what I have seen. I hope this helps. 🙂

[jlyaoyuli]

Hi @saemaromoon, it looks like you can fresh install KFP to resolve this problem. Feel free to re-open this issue if you still have any question.