Open Matgrb opened 3 years ago
I see how training a resemblance model could help detect seasonality in the data. I think there should be more context provided before offering a tool like this:
Hmm, I see this throwing confusing outputs. There are two potential flaws Let me explain:
If you want to detect seasonality, maybe a better approach is to map your folds to a repetitive pattern. For example, if you have 3 years of data, why not define a fold mapping based on quarters? In that way, you train your resemblance on Q1,Q2 and Q3 folds, and test on Q4, and then you repeat for every shuffle.
I see the approach you propose hard to interpret. let me explain: assume the that you have a linear shift in your features with time. If you train on folds 1234 and 6789 and evaluate on fold 5, probably you would not detect anything, cause you are exactly on the average shift. A better approach might be to do a "rolling" fold, where you train on 123 and estimate on 4, and then you train on 234 and estimate on 5... this might be more complicated to implement, but I would expect this to yield better results.
Addressing your comments: @timvink
@sbjelogr
What do you think?
This blogpost identifies different types of datashift and provides some ways of tackling it.
TL;DR: There are 3 types of data shifts possible :
Co-variate shift :
Prior Probability Shift :
Concept Drift :
In terms of Probatus implementation this can form a module, datashift
which produces a data shift report . The report covers the above 3 aspects.
@anilkumarpanda That is a great proposal.
It seems like a very large feature though. If we split it into the tree parts and tackle each one separately, and then try to combine using some wrapper that would be doable. For the first point, most can be done using components already available in probatus. For the rest, we would need to use other libraries.
In order to work on it we would probably need involvement of multiple collaborators, and set a more structured way of working on it. Who would like to contribute to that?
I agree with the above points. The feature is a large one and needs to be separated. I will create a separate issues for this one. Linking them to this master issue. sample_similarity
module would be the obvious place for implementing this functionality. I can start with the Prior Probability shift.
One way to measure at which point in time we can observe a data shift.
One idea would be to split data into multiple folds based on time e.g. 10. Then one could train a resemblance model taking:
Another idea would be to again split the data into folds based on time. However, in this case we would perform cross-validation, where at each iteration only one fold belongs to class 1. This way we could point to e.g. months that significantly differ from other months in the past and the future.
This feature could be part of sample_similarity, and the user could specify which resemblance model to use: permutation or shap based (default could be SHAP).
The init should take as arguments the clf and resemblance_model type, and fit should take X dataset, and indication of number of bins, or split dates, and date column name.
The output of compute should be the report presenting the validation AUC for each iteration of the process, the information about current time split date and top feature of the resemblance model (top_n parameter?)
In the plot method we could plot the AUC over time, but also user should be able to plot resemblance model plots for a specific iteration, to analyse it.