eth-easl / modyn

Modyn is a research-platform for training ML models on dynamic datasets.
MIT License
22 stars 3 forks source link

feat: Create `DriftDetector` abstraction and introduce alibi-detect #555

Closed robinholzi closed 5 days ago

robinholzi commented 5 days ago

Motivation

We want to be able to use both evidently and alibi-detect as they support different drift detection metrics as well as differ in their runtime performance.

For that we need a clean abstraction that allows us to delegate drift detection to the respective library

Changes

Note:

As this is rather big already, I'd like to get this merged before trying to run drift detection pipelines, so we might need slight adjustments in a followup PR.

github-actions[bot] commented 5 days ago

Line Coverage: -% ( % to main) Branch Coverage: -% ( % to main)

MaxiBoether commented 5 days ago

changes LGTM

codecov[bot] commented 5 days ago

Codecov Report

Attention: Patch coverage is 85.95745% with 33 lines in your changes missing coverage. Please review.

Project coverage is 82.89%. Comparing base (b6284f8) to head (c6fe126).

Files Patch % Lines
...n/supervisor/internal/triggers/datadrifttrigger.py 40.54% 22 Missing :warning:
.../internal/pipeline_executor/evaluation_executor.py 0.00% 3 Missing :warning:
...onfig/schema/pipeline/trigger/drift/aggregation.py 87.50% 2 Missing :warning:
...pervisor/internal/triggers/drift/alibi_detector.py 95.23% 2 Missing :warning:
...pervisor/internal/triggers/drift/drift_detector.py 84.61% 2 Missing :warning:
...isor/internal/triggers/drift/evidently_detector.py 95.12% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #555 +/- ## ========================================== + Coverage 82.63% 82.89% +0.25% ========================================== Files 218 226 +8 Lines 10235 10419 +184 ========================================== + Hits 8458 8637 +179 - Misses 1777 1782 +5 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.