evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
https://www.evidentlyai.com/evidently-oss
Apache License 2.0
5.44k stars 602 forks source link

Loading a Report snapshot fails for RecSys metrics with k>=10 #1233

Open soogui opened 3 months ago

soogui commented 3 months ago

Hi!

I am trying to use the snapshot functionality to store my reports and later loading and visualizing them somewhere else. I am using metrics to evaluate a recommender and run into a problem when I change the k parameter.

To reproduce the issue, I modified the following notebook: examples/how_to_questions/how_to_run_recsys_metrics.ipynb

I modified the cell in the Metrics section that creates the report to keep only a single HitRate metric:

report = Report(metrics=[
    HitRateKMetric(k=5),
])

With k=5, running the report, loading it back and displaying works fine:

report.save('report.json')
loaded_report = Report.load('report.json')
loaded_report

However, when I change the k=10 parameter:

report = Report(metrics=[
    HitRateKMetric(k=10),
])

I get the following error when I attempt to display the loaded_report:


ValueError                                Traceback (most recent call last)
File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/indexes/range.py:345, in RangeIndex.get_loc(self, key)
    344 try:
--> 345     return self._range.index(new_key)
    346 except ValueError as err:

ValueError: 10 is not in range

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/IPython/core/formatters.py:344, in BaseFormatter.__call__(self, obj)
    342     method = get_real_method(obj, self.print_method)
    343     if method is not None:
--> 344         return method()
    345     return None
    346 else:

File ~/Documents/repos/evidently/src/evidently/suite/base_suite.py:216, in Display._repr_html_(self)
    215 def _repr_html_(self):
--> 216     dashboard_id, dashboard_info, graphs = self._build_dashboard_info()
    217     template_params = TemplateParams(
    218         dashboard_id=dashboard_id,
    219         dashboard_info=dashboard_info,
    220         additional_graphs=graphs,
    221     )
    222     return self._render(inline_iframe_html_template, template_params)

File ~/Documents/repos/evidently/src/evidently/report/report.py:230, in Report._build_dashboard_info(self)
    228 # set the color scheme from the report for each render
    229 renderer.color_options = color_options
--> 230 html_info = renderer.render_html(metric)
    231 set_source_fingerprint(html_info, metric)
    232 replace_widgets_ids(html_info, id_generator)

File ~/Documents/repos/evidently/src/evidently/metrics/recsys/base_top_k.py:79, in TopKMetricRenderer.render_html(self, obj)
     77 metric_result = obj.get_result()
     78 k = metric_result.k
---> 79 counters = [CounterData.float(label=\"current\", value=metric_result.current[k], precision=3)]
     80 if metric_result.reference is not None:
     81     counters.append(CounterData.float(label=\"reference\", value=metric_result.reference[k], precision=3))

File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/series.py:1007, in Series.__getitem__(self, key)
   1004     return self._values[key]
   1006 elif key_is_scalar:
-> 1007     return self._get_value(key)
   1009 if is_hashable(key):
   1010     # Otherwise index.get_value will raise InvalidIndexError
   1011     try:
   1012         # For labels that don't resolve as scalars like tuples and frozensets

File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/series.py:1116, in Series._get_value(self, label, takeable)
   1113     return self._values[label]
   1115 # Similar to Index.get_value, but we do not fall back to positional
-> 1116 loc = self.index.get_loc(label)
   1118 if is_integer(loc):
   1119     return self._values[loc]

File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/indexes/range.py:347, in RangeIndex.get_loc(self, key)
    345         return self._range.index(new_key)
    346     except ValueError as err:
--> 347         raise KeyError(key) from err
    348 if isinstance(key, Hashable):
    349     raise KeyError(key)

KeyError: 10"