search

squaredev-io / whitebox

[Not Actively Maintained] Whitebox is an open source E2E ML monitoring platform with edge capabilities that plays nicely with kubernetes
https://squaredev.io/whitebox/
MIT License
184 stars 5 forks source link

Indicator of significance for the reports #132

Open NickNtamp opened 1 year ago

NickNtamp commented 1 year ago

Not all reports have the same significance!

Below you can find an example of drift metrics but this also applies on performance.

Lets say that:

  1. On Monday the client inserts an inference dataset with 1000 rows and a report of "Drift detected=FALSE" is produced.
  2. On Tuesday the client inserts an inference dataset with 1000 rows and a report of "Drift detected=FALSE" is produced.
  3. On Wednesday the client inserts an inference dataset with 35 rows and a report of "Drift detected=TRUE" is produced.

Even though we have a drift detection, seems that it is not important as there are much less datapoints for the specific day. We have to find a way to indicate if something is important or not (for instance for being important the drift have to be detected in a dataset with datapoints at least 50% of the average datapoints of the previous time periods - in this case for example 500 rows).

momegas commented 1 year ago

This should be covered my monitors at the moment, where the user can create a monitor and specify the severity of the alerts produced. I the monitors feature does not cover the full extend for this but should be ok for the current milestone. Keeping this as it a nice idea.