getindata / kedro-vertexai

Kedro Plugin to support running workflows on GCP Vertex AI Pipelines
https://kedro-vertexai.readthedocs.io
Apache License 2.0
35 stars 11 forks source link

Passing in runtime parameters seems to fail. #174

Open marnux opened 3 months ago

marnux commented 3 months ago

When trying to do a, as indicated in the docstring with '='

kedro vertexai run-once --image test:latest --pipeline default --param date_param=20240800

i always end up with this value error

ValueError: The pipeline parameter date_param=20240800 is not found in the pipeline job input definitions.

When using the ':' like described in the kedro documentation

kedro vertexai run-once --image test:latest --pipeline default --param date_param:20240800

I get

ValueError: The pipeline parameter date_param is not found in the pipeline job input definitions.

The parameter is set within the base config -> model_name -> data-preparation -> parameters.yaml It's defined as:

param1: value1
param2: value2
data_preparation:
  training_month: "${runtime_params:date_param, 20230900}"
  param3: value3
  ...

Within the pipeline code it's called as:

...
# Create training and prediction DFs
def create_pipeline(**kwargs) -> Pipeline:
    """Function to create the data prep pipeline."""
  return pipeline(
      [
            ...
            node(
                func=nodes_data_prep.prepare_df,
                inputs={
                    "some_df": "some_df",
                    "training_month": "params:data_preparation.training_month",
                  },
                  outputs="model_input.master_df",
                  name="create_master_df",
                  tags=["data_preparation_prod", "group.data_preparation"],
            ),
            ...
      ] 
  )

I'm sure, i'm missing something, but i can't figure out how the --param argument passes in runtime parameters and how to define them correctly. Param definitions tried:


--param date_param=20240800
--param training_month=20240800
--param date_param:20240800
--param training_month:20240800
brukted commented 2 months ago

+1