We are using the Evidently library for embedding drift detection in DataDriftTrigger. We would like to support hypothesis testing ("bootstrap" in Evidently language) for distance metrics (e.g., euclidean, cosine, MMD) because the range of the distance values varies with data and model, and we do not have a way to normalize the values.
Problem
Evidently's null-hypothesis in its statistical hypothesis testing for the "distance" and the "mmd" methods is problematic. It approximates the null distribution with reference data only. Thus, only resampling from the reference embedding matrix. So, for any current embedding matrix, as long as the reference embedding matrix is the same, the bootstrap distance values are the same. If the reference samples are sampled from the same distribution, then we always get close-to-zero bootstrap values (for mmd, predominantly max(negative, 0)), making the hypothesis test extremely sensitive, detecting drift all the time. The MMD paper suggests bootstrapping on the aggregated data.
Should also check the "bootstrap" code for the "model" (binary classifier) method. There is no evidently documentation about what bootstrapping technique or statistical test they are using.
Solution
Integrate the Alibi-Detect library. Alibi-Detect implements a permutation test on reference and current data combined. It also provides more kinds of embedding drift detection methods. It takes custom model in the binary classifier method.
Need to figure out the environment settings that can install both Evidently and Alibi-Detect. For example, fix the Python version to 3.11 because Alibi-Detect is not compatible with higher versions.
robinholzi
commented
1 month ago
Context
We are using the Evidently library for embedding drift detection in DataDriftTrigger. We would like to support hypothesis testing ("bootstrap" in Evidently language) for distance metrics (e.g., euclidean, cosine, MMD) because the range of the distance values varies with data and model, and we do not have a way to normalize the values.
Problem
Evidently's null-hypothesis in its statistical hypothesis testing for the "distance" and the "mmd" methods is problematic. It approximates the null distribution with reference data only. Thus, only resampling from the reference embedding matrix. So, for any current embedding matrix, as long as the reference embedding matrix is the same, the bootstrap distance values are the same. If the reference samples are sampled from the same distribution, then we always get close-to-zero bootstrap values (for mmd, predominantly max(negative, 0)), making the hypothesis test extremely sensitive, detecting drift all the time. The MMD paper suggests bootstrapping on the aggregated data. Should also check the "bootstrap" code for the "model" (binary classifier) method. There is no evidently documentation about what bootstrapping technique or statistical test they are using.
Solution
Integrate the Alibi-Detect library. Alibi-Detect implements a permutation test on reference and current data combined. It also provides more kinds of embedding drift detection methods. It takes custom model in the binary classifier method. Need to figure out the environment settings that can install both Evidently and Alibi-Detect. For example, fix the Python version to 3.11 because Alibi-Detect is not compatible with higher versions.