search

sktime / skpro

A unified framework for tabular probabilistic regression, time-to-event prediction, and probability distributions in python
https://skpro.readthedocs.io/en/latest
BSD 3-Clause "New" or "Revised" License
250 stars 46 forks source link

[ENH] merge `test_probabilistic_metrics` into `TestAllDistrMetrics` #398

Open fkiraly opened 5 months ago

fkiraly commented 5 months ago

The module test_probabilistic_metrics contains tests for interval and quantile distributions with manual collection. These metrics are also automatically collected in TestAllDistrMetrics, which should be the location where these tests are carried out systematically.

Hence, we should move the existing tests in test_probabilistic_metrics to there.

Whether a metric is an interval or quantile metric can be inspectedby reading the "scitype:y_pred" tag.

fkiraly commented 5 months ago

FYI @benHeid, turns out skpro is halfway to the point of a fully programmatic test suite for the proba metrics. The quantile and interval metrics will miss out on some interface tests when checked via check_estimator.