aws / sagemaker-python-sdk

A library for training and deploying machine learning models on Amazon SageMaker
https://sagemaker.readthedocs.io/
Apache License 2.0
2.09k stars 1.14k forks source link

[Bug Report] Unable to use Join with pipeline parameters within ConditionGreaterThanOrEqualTo #4418

Open clausagerskov opened 7 months ago

clausagerskov commented 7 months ago

Describe the bug When defining a condition step like in the code snippet below, if one is using a parameter for the evaluation metric (commented out line) to enable switching model evaluation metric at pipeline execution creation, the pipeline upsert fails since it expects the json_path to be valid json. Should it not be supported to have this evaluation metric be a parameter?
I am already running everything using pipeline sessions, though there does not seem to be a way to specify that in the ConditionStep.

cond_gte = ConditionGreaterThanOrEqualTo(
    left=JsonGet(
        step_name=step_evaluate_model.name,
        property_file=evaluation_report,
        # json_path = Join(on=".", values=["binary_classification_metrics", param_eval_metric, "value"]), # this doesnt work! json_path must be json serializable
        json_path=f"binary_classification_metrics.{eval_metric}.value"
    ),
    right=param_eval_threshold,
)

step_condition_check = ConditionStep(
    name="evaluation-greater-than-threshold-condition",
    conditions=[cond_gte],
    if_steps=[step_register_model],
    else_steps=[step_fail],
)

I am using sagemaker 2.207.

nmadan commented 7 months ago

Hi, thanks for reaching out. How do you defined param_eval_metric and eval_metric? Can you also paste the full exception message?

clausagerskov commented 7 months ago

@nmadan param_eval_metric = ParameterString(name="EvalMetric", default_value="auc") eval_metric = "auc"

clausagerskov commented 7 months ago

seems to be this issue which is closed but not fixed https://github.com/aws/sagemaker-python-sdk/issues/3463

clausagerskov commented 7 months ago

@nmadan before this gets closed as well, where can the feature requests be tracked?