So, we should validate that how I've adapted the calculators makes sense.
Additionally, we might want to do some prework and then rerun this test: I was getting significantly different results w.r.t. detected change points between our package and the calculators added here, as well as between the different calculator implementations themselves. It seems like it would make sense to ensure that they produce the same outputs, with some tolerance perhaps, before relying to much on these profiling results.
ancostas
commented
4 years ago
So, we should validate that how I've adapted the calculators makes sense.
Additionally, we might want to do some prework and then rerun this test: I was getting significantly different results w.r.t. detected change points between our package and the calculators added here, as well as between the different calculator implementations themselves. It seems like it would make sense to ensure that they produce the same outputs, with some tolerance perhaps, before relying to much on these profiling results.
The change point analysis package I've been using (
ruptures) has nice tooling for the validation bit of this: producing different kinds of change points and standard metrics to evaluate algorithms.