Kubeflow Pipelines V2 does not work with artifacts

[Barteus]

When running simple code with KFP V2 in the notebook:

import kfp
from kfp.v2 import dsl
from kfp.v2.dsl import component, OutputPath, Artifact, Output

@component(
    packages_to_install=['transformers'],
    output_component_file='download-comp.yaml',
    base_image='python:3.9.13'
)
def download_artifacts(repo_id: str,
                       revision: str,
                       output_file: Output[Artifact]):
    pass

@dsl.pipeline(
  name='chatbot-pipeline',
  description='The pipeline to create and deploy the chatbot model',
)
def my_pipeline():
    download_artifacts_op = download_artifacts(repo_id='facebook/detr-resnet-50',revision='c783425425d573f30483efb0660bf6207deea991')

client = kfp.Client()

client.create_run_from_pipeline_func(
    my_pipeline,
    namespace="admin",
    mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE,
    arguments={})

The log in the Pod:

I1016 19:06:23.565782      43 env.go:30] cannot find launcher configmap: name="kfp-launcher" namespace="admin"
I1016 19:06:23.565945      43 launcher.go:144] PipelineRoot defaults to "minio://mlpipeline/v2/artifacts".
I1016 19:06:23.566163      43 cache.go:143] Cannot detect ml-pipeline in the same namespace, default to ml-pipeline.kubeflow:8887 as KFP endpoint.
I1016 19:06:23.566179      43 cache.go:120] Connecting to cache endpoint ml-pipeline.kubeflow:8887
F1016 19:06:25.594237      43 main.go:50] Failed to execute component: unable to get pipeline with PipelineName "pipeline/chatbot-pipeline" PipelineRunID "168f2ed7-39b6-40a8-9130-9b9b0a752db8": Failed GetContextByTypeAndName(type="system.Pipeline", name="pipeline/chatbot-pipeline")

Kubeflow bundle - 1.6 (full bundle) + mlflow, microk8s 1.22

Am I doing something wrong?

[ca-scribner]

I haven't tried using v2 yet so these are guesses.

I think the kfp.v2 package is meant to use some yet-to-be-released backend features (aimed for kf 1.7). So out of the box, it might be trying to access those. Looking at this guide, I see they import only component from kfp.v2.dsl (and use the kfp.dsl.pipeline, not the v2 version). So it might be that you're using the v2 compiler atm, and submitting that v2 pipeline to our v1 backend. They also mention using the mode=...V2_COMPATIBLE arg. I think that's telling the 1.7 compiler to compile in v2's syntax, but it might mean the other way around and might help you here.

As far as I know, kfp hasn't released for wide use the v2 backend (I think they plan that for the next release), although I believe the images we deploy atm do support it. I expect there's some feature flags that would enable it, but I haven't looked into them.