deepchecks / deepchecks

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.
https://docs.deepchecks.com/stable
Other
3.6k stars 254 forks source link

[BUG] ModelErrorAnalysis display is empty when there is no meaningful splits #1810

Open TheSolY opened 2 years ago

TheSolY commented 2 years ago

The expected behavior is to display a message such as "No meaningful properties found" Current display: image

Code to reproduce:

import numpy as np
from deepchecks.vision.datasets.classification import mnist
from deepchecks.vision.checks import ModelErrorAnalysis

model = mnist.load_model()
train_ds = mnist.load_dataset(train=True, object_type='VisionData')
test_ds = mnist.load_dataset(train=False, object_type='VisionData')

def med_prop(batch):
    return [np.median(x) for x in batch]

def min_prop(batch):
    return [np.min(x) for x in batch]

new_properties = [{'name': 'median brightness', 'method': med_prop, 'output_type': 'numerical'},
                  {'name': 'min brightness', 'method': min_prop, 'output_type': 'numerical'}]

train_ds.image_properties = new_properties
test_ds.image_properties = new_properties

check = ModelErrorAnalysis(image_properties=new_properties, min_error_model_score=-0.1)
result = check.run(train_ds, test_ds, model)
result.save_as_html()
noamzbr commented 2 years ago

Should wait for implementation of #1780