search

RedHatQuickCourses / rhods-pipelines

Automation using Data Science Pipelines
https://redhatquickcourses.github.io/rhods-pipelines/
2 stars 6 forks source link

Instructions to use an extra Environment Variable in the workbench cause issues with pipeline submission. #16

Closed erwangranger closed 5 months ago

erwangranger commented 5 months ago

A few people have reported that the pipeline submission step causes errors:

image

text (from another env):

Failed to initialize `kfp.Client()` against: 'https://ds-pipeline-pipelines-definition-pipelines-example.apps.cluster-lpsch.dynamic.redhatworkshops.io/' - Check Kubeflow Pipelines runtime configuration: 'odh_dsp' - 
[TIP: did you mean to set 'https://ds-pipeline-pipelines-definition-pipelines-example.apps.cluster-lpsch.dynamic.redhatworkshops.io/pipeline' as the endpoint, take care not to include 's' at end]

I reproduced this issue by using a RHOAI 2.8 environment, with a project that had a functional pipeline server. When I added the PIPELINES_SSL_SA_CERTS=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt variable to my workbench, my pipelines submissions failed with the above.

When I removed it, it started working again.

These instructions (https://redhatquickcourses.github.io/rhods-pipelines/rhods-pipelines/1.33/chapter1/elyra-pipelines.html#_set_up_the_workbench) might have worked fine in earlier instances of RHOAI, but I'm wondering if they should not be updated to be more version-specific.

This change was made based on this issue: https://github.com/RedHatQuickCourses/rhods-pipelines/issues/12

Until the instructions are updated, if anyone runs into this issue, I'd suggest: 1) stop the workbench 2) edit its definition to remove the environment variable 3) restart the workbench 4) retry the pipeline submission

rsriniva commented 5 months ago

Thank you for the tip @erwangranger - Do I rollback the fix for the TLS cert previously reported in v2.6? Was the old issue a bug in v2.6 then?

rsriniva commented 5 months ago

@noelo - FYI. Are you observing anything different in v2.8? any other gotchas for the latest release?

erwangranger commented 5 months ago

Based on the discussion in this jira I'd say it confirms what I've been saying. If we want these instructions to work in 2.8 or 2.8.1, we should remove the references to that variable. With that said, I don't know if people are supposed to use a 2.6 env or a 2.8 for the RH quick course, so I will let the authors decide on the best course of action.

rsriniva commented 5 months ago

Thank you for confirming @erwangranger. Students can use v2.8. I'll fix this in a couple of days