{Fast, Correct, Simple} - pick three
Boxkite is an instrumentation library designed from ground up for tracking concept drift in HA (Highly Available) model servers. It integrates well with existing DevOps tools (ie. Grafana, Prometheus, fluentd, kubeflow, etc.), and scales horizontally to multiple replicas with no code or infrastructure change.
nan
, None
, inf
, and negative valuesSome non-goals of this project are:
If you are interested in alternatives, please refer to our discussions in FAQ.
Follow one of our tutorials to easily get started and see how Boxkite works with other tools:
See Installation & User Guide for how to use Boxkite in any environment.
Prometheus has supported outlier detection in time series data since 2015. Once you've setup KL divergence and K-S test metrics, outlier detection can be configured on top using alerting rules. For a detailed example, refer to this tutorial: https://prometheus.io/blog/2015/06/18/practical-anomaly-detection/.
Adversarial detection concerns with identifying single OOD (Out Of Distribution) samples rather than comparing whole distributions. The algorithms are also highly model specific. For these reasons, we do not have plans to support them in boxkite at the moment. As an alternative, you may look into Seldon for such capabilities https://github.com/SeldonIO/alibi-detect#adversarial-detection.
Not yet. This is still an actively researched area that we are keeping an eye on.
Yes, our instrumentation library is framework agnostic. It expects input data to be a list
or np.array
regardless of how the model is trained.
The following people have contributed to the original concept and code
A full list of contributors, which includes individuals that have contributed entries, can be found here.
Boxkite is a project from BasisAI, who offer an MLOps Platform called Bedrock.
Bedrock helps data scientists own the end-to-end deployment of machine learning workflows. Boxkite was originally part of the Bedrock client library, but we've spun it out into an open source project so that it's useful for everyone!