aeon-toolkit / aeon

A toolkit for machine learning from time series
https://aeon-toolkit.org/
BSD 3-Clause "New" or "Revised" License
1.02k stars 128 forks source link

[ENH] Implement range-based precision and recall within aeon (replacing prts package) #2066

Open SebastianSchmidl opened 2 months ago

SebastianSchmidl commented 2 months ago

Describe the feature or idea you want to propose

The package prts is used in the performance evaluation module for anomaly detection as a soft-dependency. However, there have not been any changes to it for 3 years, the maintainers have not reacted to my PR, and the package was not implemented by the original authors of the method/paper.

Describe your proposed solution

We should implement the metrics within aeon directly:

Target module: aeon.benchmarking.metrics.anomaly_detection Original publication:

Tatbul, Nesime, Tae Jun Lee, Stan Zdonik, Mejbah Alam, and Justin Gottschlich. "Precision and Recall for Time Series." In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 1920–30. 2018. http://papers.nips.cc/paper/7462-precision-and-recall-for-time-series.pdf.

Describe alternatives you've considered, if relevant

not relevant

Additional context

No response

aryanpola commented 2 months ago

@aeon-actions-bot assign @aryanpola

aryanpola commented 2 months ago

@CodeLionX So, the true anomaly ranges would use the timepoints to create a range to compare with the algorithm-detected anomaly range. Maybe later find an intersection between them.

SebastianSchmidl commented 2 months ago

Hi @aryanpola, yes and no. Please read the paper carefully, it is quite detailed in its descriptions on how to compute ts_precision and ts_recall respectively (math formulas and pseudocode).

In their formulation, they convert both the ground truth (point) values and the (point) predictions into ranges, and then iterate over the real/true anomaly ranges for recall, but over the predicted anomaly ranges for precision.

You can use the existing implementation in prts to generate test cases to validate that your implementation is doing the same.