elyra-ai / elyra

Elyra extends JupyterLab with an AI centric approach.
https://elyra.readthedocs.io/en/stable/
Apache License 2.0
1.86k stars 344 forks source link

elyra-pipeline validate CLI command fails for pipelines with parameters and default properties #3190

Closed mamurak closed 6 months ago

mamurak commented 1 year ago

Describe the issue Running elyra-pipeline validate and by extension elyra-pipeline export fails silently with pipelines that contain both pipeline parameters and default properties.

To Reproduce Steps to reproduce the behavior:

  1. Create a simple Elyra pipeline with a single node.
  2. Add a pipeline parameter in the Pipeline Parameters configuration tab.
  3. Open the Pipeline Properties configuration tab.
  4. Save the pipeline, e.g. as test.pipeline.
  5. Validate the pipeline through the CLI: elyra-pipeline validate test.pipeline.

Screenshots or log output The following error message without details is printed after running the above validation command:

Log Output
(app-root) (app-root) elyra-pipeline validate test.pipeline 

────────────────────────────────────────────────────────────────
 Elyra Pipeline Validation
────────────────────────────────────────────────────────────────

Validating pipeline...
Error: Pipeline validation FAILED.

Expected behavior Given that pipelines with parameters and default properties are valid in general, the validation command should resolve without any errors. In case of actual issues within the pipeline file, there should be detailed information about these issues.

Deployment information Describe what you've deployed and how:

mamurak commented 1 year ago

Example pipeline:

{
  "doc_type": "pipeline",
  "version": "3.0",
  "json_schema": "http://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json",
  "id": "elyra-auto-generated-pipeline",
  "primary_pipeline": "primary",
  "pipelines": [
    {
      "id": "primary",
      "nodes": [
        {
          "id": "54d57acc-4ad0-410e-8d79-268cd52a45e9",
          "type": "execution_node",
          "op": "execute-python-node",
          "app_data": {
            "component_parameters": {
              "pipeline_parameters": [
                "param"
              ],
              "dependencies": [],
              "include_subdirectories": false,
              "outputs": [],
              "env_vars": [],
              "kubernetes_pod_annotations": [],
              "kubernetes_pod_labels": [],
              "kubernetes_secrets": [],
              "kubernetes_shared_mem_size": {},
              "kubernetes_tolerations": [],
              "mounted_volumes": [],
              "filename": "untitled.py",
              "runtime_image": "quay.io/modh/runtime-images@sha256:7dd23e58291cad7a0ab4a8e04bda06492f2c027eb33b226358380db58dcdd60b"
            },
            "label": "",
            "ui_data": {
              "label": "untitled.py",
              "image": "/notebook/fraud-detection/tensorflow/static/elyra/python.svg",
              "x_pos": 505,
              "y_pos": 210,
              "description": "Run Python script"
            }
          },
          "inputs": [
            {
              "id": "inPort",
              "app_data": {
                "ui_data": {
                  "cardinality": {
                    "min": 0,
                    "max": -1
                  },
                  "label": "Input Port"
                }
              }
            }
          ],
          "outputs": [
            {
              "id": "outPort",
              "app_data": {
                "ui_data": {
                  "cardinality": {
                    "min": 0,
                    "max": -1
                  },
                  "label": "Output Port"
                }
              }
            }
          ]
        }
      ],
      "app_data": {
        "ui_data": {
          "comments": []
        },
        "version": 8,
        "runtime_type": "KUBEFLOW_PIPELINES",
        "properties": {
          "name": "untitled",
          "runtime": "Data Science Pipelines",
          "pipeline_parameters": [
            {
              "name": "param",
              "description": "",
              "default_value": {
                "type": "String",
                "value": "val"
              },
              "required": false
            }
          ],
          "pipeline_defaults": {
            "kubernetes_shared_mem_size": {},
            "kubernetes_pod_labels": [],
            "mounted_volumes": [],
            "kubernetes_pod_annotations": [],
            "kubernetes_tolerations": [],
            "kubernetes_secrets": [],
            "env_vars": []
          }
        }
      },
      "runtime_ref": ""
    }
  ],
  "schemas": []
}

Validation through the CLI fails. Remove either pipeline_parameters or pipeline_defaults, and validation succeeds.