mitre / menelaus

Online and batch-based concept and data drift detection algorithms to monitor and maintain ML performance.
https://menelaus.readthedocs.io/en/latest/
Apache License 2.0
65 stars 7 forks source link

Add flow diagram to README on selecting drift detectors #37

Open tms-bananaquit opened 2 years ago

tms-bananaquit commented 2 years ago

We can have a "decision tree" on which setting each detector is appropriate for.

tms-bananaquit commented 2 years ago

Here's a draft of such a diagram, that excludes concerns like "how fast does it run" and "do you have >=10 features" that would make it an easier decision between e.g. HDDDM and kdq-tree, for example.

It should be as easy as switching this to the sphinx syntax for mermaid, putting that in an .rst file, and referencing that rst file in our index.rst.

graph TD
A[Do you have access to the model's predicted outcome?]
A -->|No| B
A -->|Yes| C
B[Do you have access to the data for each model feature?]
B -->|No| D
B -->|Yes| E
C[Can you easily/quickly obtain the true outcome?]
C -->|No| B
C -->|Yes| F
C -->|Eventually| G
D{{Access to either the data or the model output is required for drift detection.}}
E{{Data Drift Detector}}
F{{Concept Drift Detector}}
G{{Semi-supervised Detector}}

The sphinx-syntax is as above, but with semicolons terminating each line, and the ..mermaid:: directive, i.e..:

.. mermaid::

    graph TD;
        A[Do you have access to the model's predicted outcome?];
        A -->|No| B;
        A -->|Yes| C;
        B[Do you have access to the data for each model feature?];
        B -->|No| D;
        B -->|Yes| E;
        C[Can you easily/quickly obtain the true outcome?];
        C -->|No| B;
        C -->|Yes| F;
        C -->|Eventually| G;
        D{{Access to either the data or the model output is required for drift detection.}};
        E{{Data Drift Detector}};
        F{{Concept Drift Detector}};
        G{{Semi-supervised Detector}};