elyra-ai / elyra

Elyra extends JupyterLab with an AI centric approach.
https://elyra.readthedocs.io/en/stable/
Apache License 2.0
1.83k stars 343 forks source link

Can't export kubeflow pipelines if I use parameters #3172

Open MartinKlefas opened 1 year ago

MartinKlefas commented 1 year ago

Describe the issue I've downloaded your notebook server image and successfully submitted a number of pipelines based on your examples and the early stages of my work. I'd like to add parameters to my notebooks, but at present have made a very simple test notebook to try to get my first parametric pipeline created.

To Reproduce I've made a simple python notebook which just prints "hello {person}". I've then followed this guide https://medium.com/codait/build-versatile-pipelines-in-elyra-using-pipeline-parameters-5634f8d5ae8d#:~:text=Parameterizing%20Pipelines%20in%20Elyra,Node%20Properties%20%2C%20all%20shown%20below. to make a kubeflow pipeline of sorts with single node. I can save it with no parameters, and can export it to YAML. If I add a parameter, I can still export it to YAML. If I add that parameter to the single node, I'm no longer able to export the pipeline to YAML and get the below error from "argo".

Screenshots or log output Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/kfp/processor_kfp.py", line 676, in _compile_pipeline_dsl kfp_argo_compiler.Compiler().compile(pipeline_function, output_file, pipeline_conf=pipeline_conf) File "/opt/conda/lib/python3.8/site-packages/kfp/compiler/compiler.py", line 1175, in compile self._create_and_write_workflow( File "/opt/conda/lib/python3.8/site-packages/kfp/compiler/compiler.py", line 1227, in _create_and_write_workflow workflow = self._create_workflow(pipeline_func, pipeline_name, File "/opt/conda/lib/python3.8/site-packages/kfp/compiler/compiler.py", line 1005, in _create_workflow pipeline_func(*args_list, **kwargs_dict) File "/tmp/tmpdwpcjjxg/generated_dsl.py", line 44, in generated_pipeline TypeError: Run a file() got an unexpected keyword argument 'toWhom'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/tornado/web.py", line 1713, in _execute result = await result File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/handlers.py", line 108, in post pipeline_exported_path = await PipelineProcessorManager.instance().export( File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/processor.py", line 111, in export res = await asyncio.get_event_loop().run_in_executor( File "/opt/conda/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/kfp/processor_kfp.py", line 523, in export self._compile_pipeline_dsl(pipeline_dsl, workflow_engine, absolute_pipeline_export_path, pipeline_conf) File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/kfp/processor_kfp.py", line 678, in _compile_pipeline_dsl raise RuntimeError( RuntimeError: Failed to compile pipeline with workflow_engine 'argo' to '/home/jovyan/untitled1.yaml'

Expected behavior exported YAML file

Deployment information Describe what you've deployed and how:

Pipeline runtime environment

attachment I've attached the notebook and the pipeline file as those are probably what I've done wrong

MartinKlefas commented 1 year ago

It's also worth mentioning that saving as a dsl file produces the python file as expected, but when I run that from the command line I get: Traceback (most recent call last): File "untitled1.py", line 116, in kfp.compiler.Compiler().compile( File "/opt/conda/lib/python3.8/site-packages/kfp/compiler/compiler.py", line 1175, in compile self._create_and_write_workflow( File "/opt/conda/lib/python3.8/site-packages/kfp/compiler/compiler.py", line 1227, in _create_and_write_workflow workflow = self._create_workflow(pipeline_func, pipeline_name, File "/opt/conda/lib/python3.8/site-packages/kfp/compiler/compiler.py", line 1005, in _create_workflow pipeline_func(*args_list, **kwargs_dict) File "untitled1.py", line 44, in generated_pipeline factory_31cf8eea6110c297eb6e7ce7400159bd041c724a42334c025116d58a337198cf( TypeError: Run a file() got an unexpected keyword argument 'toWhom'

where toWhom is my parameter name.