search

evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
https://www.evidentlyai.com/evidently-oss
Apache License 2.0
5.37k stars 596 forks source link

Evaluating the classification quality of model #14

Closed xsansha closed 2 years ago

xsansha commented 3 years ago

Mathematically, the classification quality of the model is known as Matthews Correlation Coefficient.

The metric is a true judge for the classification problems having an imbalanced set of classes.

https://en.wikipedia.org/wiki/Matthews_correlation_coefficient

xsansha commented 3 years ago

Model Accuracy

model_accuracy

Classification quality of model

model_classification_quality

If we observe, the accuracy is not giving us a clear picture of the classification capability of the model.

xsansha commented 3 years ago

Can this feature be added?

emeli-dral commented 3 years ago

Hi @xsansha ,

It can indeed be useful for problems with imbalanced classes!

However, we try to minimise the number of plots for now. The goal is to provide maximum information in a single report without duplication. One of the concerns is not inflate the report in size too much (since all visualizations for now are stored in .html file). We had to vote out a few plots in a heated debate :)

The reason we excluded Matthews Correlation is that 1) it is mostly useful for imbalanced classification (especially binary), whereas we have a single report that works for all binary and multi-class alike 2) the metric itself is harder to interpret. We suggest to refer to Quality Metrics by Class to understand the variations in model performance per class.

That said, we understand some might prefer different plots for different use cases, and it is impossible to build universal "defaults". We are working on a service version of a tool where all the plots can be configured, and expect to include the full selection of options there!

xsansha commented 3 years ago

Thanks @emeli-dral

emeli-dral commented 2 years ago

Hi @xsansha ,

I just wanted to mention that now you can design custom Metrics, Widgets and Tabs. Maybe you will find convenient to customize or create your ClassificationPerformanceTab, where you can remove all irrelevant visuals and add your own ones :) Here is the blog where we introduced the customization functionality: https://evidentlyai.com/blog/evidently-custom-tabs-widgets-and-metrics

And here you can see how to create a custom widget with a different statistical test https://docs.evidentlyai.com/step-by-step-guides/report-customization/add-a-custom-widget-or-tab

If you need any help with specific code or implementation, please join our Discord https://discord.com/invite/xZjKRaNp8b support channel!