Cannot connect to/launch runs in the Pipelines client from GitLab CI/CD

[qiuyuanyi1867]

Bug Description

Description: The goal is to automate the execution of Kubeflow pipelines through GitLab. However, I am encountering issues when attempting to connect to and launch runs in the Pipelines client from GitLab CI/CD.

When running the pipeline/pipeline_run.py script in the GitLab CI/CD pipeline, I encounter the following error:

ERROR:root:Failed to read a token from file '/var/run/secrets/kubeflow/pipelines/token' ([Errno 2] No such file or directory: '/var/run/secrets/kubeflow/pipelines/token').

.gitlab-ci.yml Configuration

stages:
  - deploy

deploy_pipeline:
  stage: deploy
  tags:
    - kubernetes
  image: python:3.8-slim  
  script:
    - pip install kfp  
    - python pipeline/pipeline_run.py  
  only:
    - main  

pipeline/pipeline_run.py Content

from kfp import client
from main_pipeline import onvm_pipeline

def trigger_pipeline():
    kfp_client = client.Client()
    run = kfp_client.create_run_from_pipeline_func(
        onvm_pipeline
    )

if __name__ == "__main__":
    trigger_pipeline()

To Reproduce

1. Setup a single-node Kubernetes cluster using kubeadm.
2. Install Kubeflow using kustomize following the Kubeflow Manifests repository guide.
3. Configure GitLab Runner with Kubernetes executor and necessary permissions.
4. Create a .gitlab-ci.yml file in gitlab
5. Run the pipeline in GitLab CI/CD and observe the error

Environment

• Kubernetes Setup: Single-node Kubernetes cluster set up with kubeadm
• Kubernetes Version: v1.29.3
• Kubeflow Version: Installed using kustomize with the following components:
• Kubeflow Pipelines: 2.2.0
• GitLab Runner Version: 16.11.0
• GitLab Runner Helm Chart Version: 0.64.0
• KFP Python SDK Version: 2.4.0

Relevant Log Output

ERROR:root:Failed to read a token from file '/var/run/secrets/kubeflow/pipelines/token' ([Errno 2] No such file or directory: '/var/run/secrets/kubeflow/pipelines/token').

Additional Context

I referred to GitHub issue Notebook cannot implicitly connect to/launch runs in the Pipelines client #423 for methods to resolve a similar issue in a notebook. Based on this, I attempted to use PodDefault for configuration. However, while this method worked when using the Jupyter notebook client, it did not work with GitLab Runner.

Troubleshooting Steps Taken:

1. Created PodDefault in gitlab-runner namespace:

   apiVersion: kubeflow.org/v1alpha1
   kind: PodDefault
   metadata:
   name: access-kfp
   namespace: gitlab-runner
   spec:
   desc: "Automatically mount token for KFP access"
   selector:
   matchLabels:
     access-kfp: "true"
   env:
   - name: KF_PIPELINES_SA_TOKEN_PATH
     value: /var/run/secrets/kubeflow/pipelines/token
   volumeMounts:
   - mountPath: /var/run/secrets/kubeflow/pipelines
     name: volume-kf-pipeline-token
     readOnly: true
   volumes:
   - name: volume-kf-pipeline-token
     projected:
       sources:
         - serviceAccountToken:
             path: token
             expirationSeconds: 7200
             audience: pipelines.kubeflow.org

2. Granted necessary permissions:

   
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
   name: gitlab-runner-kf-edit-binding
   roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: kubeflow-pipelines-edit
   subjects:
   - kind: ServiceAccount
   name: gitlab-runner
   namespace: gitlab-runner

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: gitlab-runner-kf-view-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubeflow-pipelines-view subjects:

• kind: ServiceAccount name: gitlab-runner namespace: gitlab-runner

3. Updated GitLab Runner configuration:

   runners:
   config: |
   [[runners]]
    [runners.kubernetes]
      namespace = "gitlab-runner"
      image = "alpine"
      [runners.kubernetes.pod_labels]
        access-kfp = "true"
      [runners.kubernetes.volumes]
         [[runners.kubernetes.volumes.secret]]
           name = "kfp-sa-token"
           mount_path = "/var/run/secrets/kubeflow/pipelines"
           read_only = true
           items = [
             { key = "token", path = "token"}
            ]

Request for Assistance: I am seeking help on:

1. Correctly generating a token with the appropriate audience for the Kubeflow pipeline.
2. Ensuring that the GitLab Runner mounts this token correctly for CI/CD pipeline runs. Any guidance or suggestions on how to resolve this issue would be greatly appreciated. Thank you!

[syncronize-issues-to-jira[bot]]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5818.

This message was autogenerated