kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.53k stars 1.59k forks source link

[sdk] SlicedClassificationMetrics load_roc_readings error #10250

Closed petethegreat closed 4 months ago

petethegreat commented 8 months ago

Environment

kfp                      2.3.0
kfp-pipeline-spec        0.2.2
kfp-server-api           2.0.2

Steps to reproduce

SlicedClassificationMetrics.load_roc_readings() gives an error. When called, it tries to invoke ClassificationMetrics.load_roc_readings() - this method does not exist, giving an error. The corresponding ClassificationMetrics method is log_roc_curve The order of the parameters (fpr, tpr, thresholds) are different - documentation for load_roc_readings suggests thresholds, tpr, fpr, but log_roc_curve wants fpr, tpr, thresholds

Also, as mentioned in #8117 - SlicedClassificationMetrics does not initialise _sliced_metrics itself, resulting in an error if this is not done manually.

Expected result

When calling SlicedClassificationMetrics.load_roc_readings, the ROC metrics should be loaded correctly without error

Materials and Reference

example to reproduce:

from kfp import dsl
from kfp import compiler

@dsl.component(base_image='python:3.11')
def slicedmetrics_component(
    evaluation_metrics: dsl.Output[dsl.SlicedClassificationMetrics]):

    evaluation_metrics._sliced_metrics = {}

    # dummy roc data
    fpr = [ 0.0, 0.0, 0.0, 1.0]
    tpr = [0.0, 0.5, 1.0, 1.0]
    thresholds = [float('inf'), 0.99, 0.8, 0.01]

    evaluation_metrics.load_roc_readings(
        'test_slice', [thresholds, tpr, fpr]
    )

@dsl.pipeline(
    name='slicedclassification_sample_pipeline')
def example_pipeline():
    metrics_op = slicedmetrics_component()

compiler.Compiler().compile(example_pipeline, package_path='slicedclassification_pipeline.json')

Impacted by this bug? Give it a 👍.

zijianjoy commented 6 months ago

/assign @connor-mccarthy

Possible guess: float('inf') blocks json serialization.

petethegreat commented 6 months ago

pr that addresses this is here: #10350

github-actions[bot] commented 4 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 4 months ago

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.