Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
I am trying to use the snapshot functionality to store my reports and later loading and visualizing them somewhere else. I am using metrics to evaluate a recommender and run into a problem when I change the k parameter.
To reproduce the issue, I modified the following notebook:
examples/how_to_questions/how_to_run_recsys_metrics.ipynb
I modified the cell in the Metrics section that creates the report to keep only a single HitRate metric:
report = Report(metrics=[
HitRateKMetric(k=5),
])
With k=5, running the report, loading it back and displaying works fine:
I get the following error when I attempt to display the loaded_report:
ValueError Traceback (most recent call last)
File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/indexes/range.py:345, in RangeIndex.get_loc(self, key)
344 try:
--> 345 return self._range.index(new_key)
346 except ValueError as err:
ValueError: 10 is not in range
The above exception was the direct cause of the following exception:
KeyError Traceback (most recent call last)
File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/IPython/core/formatters.py:344, in BaseFormatter.__call__(self, obj)
342 method = get_real_method(obj, self.print_method)
343 if method is not None:
--> 344 return method()
345 return None
346 else:
File ~/Documents/repos/evidently/src/evidently/suite/base_suite.py:216, in Display._repr_html_(self)
215 def _repr_html_(self):
--> 216 dashboard_id, dashboard_info, graphs = self._build_dashboard_info()
217 template_params = TemplateParams(
218 dashboard_id=dashboard_id,
219 dashboard_info=dashboard_info,
220 additional_graphs=graphs,
221 )
222 return self._render(inline_iframe_html_template, template_params)
File ~/Documents/repos/evidently/src/evidently/report/report.py:230, in Report._build_dashboard_info(self)
228 # set the color scheme from the report for each render
229 renderer.color_options = color_options
--> 230 html_info = renderer.render_html(metric)
231 set_source_fingerprint(html_info, metric)
232 replace_widgets_ids(html_info, id_generator)
File ~/Documents/repos/evidently/src/evidently/metrics/recsys/base_top_k.py:79, in TopKMetricRenderer.render_html(self, obj)
77 metric_result = obj.get_result()
78 k = metric_result.k
---> 79 counters = [CounterData.float(label=\"current\", value=metric_result.current[k], precision=3)]
80 if metric_result.reference is not None:
81 counters.append(CounterData.float(label=\"reference\", value=metric_result.reference[k], precision=3))
File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/series.py:1007, in Series.__getitem__(self, key)
1004 return self._values[key]
1006 elif key_is_scalar:
-> 1007 return self._get_value(key)
1009 if is_hashable(key):
1010 # Otherwise index.get_value will raise InvalidIndexError
1011 try:
1012 # For labels that don't resolve as scalars like tuples and frozensets
File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/series.py:1116, in Series._get_value(self, label, takeable)
1113 return self._values[label]
1115 # Similar to Index.get_value, but we do not fall back to positional
-> 1116 loc = self.index.get_loc(label)
1118 if is_integer(loc):
1119 return self._values[loc]
File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/indexes/range.py:347, in RangeIndex.get_loc(self, key)
345 return self._range.index(new_key)
346 except ValueError as err:
--> 347 raise KeyError(key) from err
348 if isinstance(key, Hashable):
349 raise KeyError(key)
KeyError: 10"
Hi!
I am trying to use the snapshot functionality to store my reports and later loading and visualizing them somewhere else. I am using metrics to evaluate a recommender and run into a problem when I change the k parameter.
To reproduce the issue, I modified the following notebook:
examples/how_to_questions/how_to_run_recsys_metrics.ipynb
I modified the cell in the
Metrics
section that creates the report to keep only a single HitRate metric:With
k=5
, running the report, loading it back and displaying works fine:However, when I change the
k=10
parameter:I get the following error when I attempt to display the
loaded_report
: