kubeflow / kfp-tekton

Kubeflow Pipelines on Tekton
https://developer.ibm.com/blogs/kubeflow-pipelines-with-tekton-and-watson/
Apache License 2.0
175 stars 122 forks source link

ConditionRef is generated, but the referenced condition is not #325

Closed Udiknedormin closed 4 years ago

Udiknedormin commented 4 years ago

/kind bug

What steps did you take and what happened:

from kfp import dsl
from kfp_tekton.compiler import TektonCompiler as Compiler

def some_pipeline():
    task_0 = dsl.ContainerOp(
      name='op',
      image='image',
      file_outputs={'result': '/tmp/result'}
    )

    with dsl.Condition(task_0.outputs['result'] == ''):
      task_1 = dsl.ContainerOp(name='op1', image='image')

if __name__ == '__main__':
    Compiler().compile(some_pipeline, __file__ + '.yaml')

Results in Tekton yaml:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  annotations:
    pipelines.kubeflow.org/pipeline_spec: '{"name": "Some pipeline"}'
    sidecar.istio.io/inject: 'false'
    tekton.dev/artifact_bucket: mlpipeline
    tekton.dev/artifact_endpoint: minio-service.kubeflow:9000
    tekton.dev/artifact_endpoint_scheme: http://
    tekton.dev/artifact_items: '{"op": [["result", "$(results.result.path)"]], "op1":
      []}'
    tekton.dev/input_artifacts: '{}'
    tekton.dev/output_artifacts: '{"op": [{"key": "artifacts/$PIPELINERUN/op/result.tgz",
      "name": "op-result", "path": "/tmp/result"}]}'
  name: some-pipeline
spec:
  pipelineSpec:
    tasks:
    - name: op
      taskSpec:
        results:
        - description: /tmp/result
          name: result
        stepTemplate:
          volumeMounts:
          - mountPath: /tmp
            name: result
        steps:
        - image: image
          name: main
        - image: busybox
          name: copy-results
          script: '#!/bin/sh

            set -exo pipefail

            cp /tmp/result $(results.result.path);

            '
        volumes:
        - emptyDir: {}
          name: result
    - conditions:
      - conditionRef: super-condition
        params:
        - name: operand1
          value: $(tasks.op.results.result)
        - name: operand2
          value: ''
        - name: operator
          value: ==
      name: op1
      taskSpec:
        steps:
        - image: image
          name: main

Tekton-pipeline-controller fails with:

Reconcile error: Couldn't retrieve Condition "super-condition": condition.tekton.dev "super-condition" not found

What did you expect to happen: Condition with name super-condition should be defined.

Environment:

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the labels:

Label Probability
area/sdk-dsl-compiler 0.81

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the labels:

Label Probability
area/sdk-dsl-compiler 0.81

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

Tomcli commented 4 years ago

The super-condition is no longer produced in our SDK, it's deployed as part of the single and multi-user of KFP-Tekton.

Single user deployment: https://github.com/IBM/manifests/blob/master/pipeline/installs/tekton/kfp-tekton/catalog-condition.yaml

Multi-user deployment: https://github.com/IBM/manifests/blob/master/pipeline/installs/tekton-multi-user/sync.py#L250-L281

Udiknedormin commented 4 years ago

@Tomcli Yes, I've just come to that in the meantime and was about to close the issue. Thank you.