evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
https://www.evidentlyai.com/evidently-oss
Apache License 2.0
5.46k stars 604 forks source link

If current df doesn't have an expected column, running tests raises assertion error #1055

Open aguschin opened 8 months ago

aguschin commented 8 months ago

Hey there! Discovered this stepping away from OSS Tutorial

from sklearn.datasets import fetch_california_housing

from evidently.test_suite import TestSuite
from evidently.test_preset import DataStabilityTestPreset
from evidently.tests import *

data = fetch_california_housing(as_frame=True)
housing_data = data.frame
housing_data.head(1)

reference = housing_data.sample(n=5000, replace=False)
current = housing_data.sample(n=5000, replace=False).drop(["AveOccup"], axis=1)

suite = TestSuite(tests=[
    DataStabilityTestPreset(),
])

suite.run(reference_data=reference, current_data=current)
suite

tb:

{
    "name": "AssertionError",
    "message": "",
    "stack": "---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
File ~/Git/kc-mlops/testing_monitoring/.venv/lib/python3.12/site-packages/IPython/core/formatters.py:347, in BaseFormatter.__call__(self, obj)
    345     method = get_real_method(obj, self.print_method)
    346     if method is not None:
--> 347         return method()
    348     return None
    349 else:

File ~/Git/kc-mlops/testing_monitoring/.venv/lib/python3.12/site-packages/evidently/suite/base_suite.py:167, in Display._repr_html_(self)
    166 def _repr_html_(self):
--> 167     dashboard_id, dashboard_info, graphs = self._build_dashboard_info()
    168     template_params = TemplateParams(
    169         dashboard_id=dashboard_id,
    170         dashboard_info=dashboard_info,
    171         additional_graphs=graphs,
    172     )
    173     return self._render(determine_template(\"auto\"), template_params)

File ~/Git/kc-mlops/testing_monitoring/.venv/lib/python3.12/site-packages/evidently/test_suite/test_suite.py:212, in TestSuite._build_dashboard_info(self)
    210 renderer.color_options = color_options
    211 by_status[test_result.status] = by_status.get(test_result.status, 0) + 1
--> 212 html = renderer.render_html(test)
    213 replace_test_widget_ids(html, generator)
    214 test_results.append(html)

File ~/Git/kc-mlops/testing_monitoring/.venv/lib/python3.12/site-packages/evidently/tests/data_integrity_tests.py:993, in TestColumnsTypeRenderer.render_html(self, obj)
    990 info = super().render_html(obj)
    992 parameters = obj.get_result().parameters
--> 993 assert isinstance(parameters, ColumnTypesParameter)
    994 info.details = [
    995     DetailsInfo(
    996         title=\"\",
   (...)
   1006     ),
   1007 ]
   1008 return info

AssertionError: "
}
aguschin commented 8 months ago

evidently==0.4.18