Cross-validation using a list of k input sets and a corresponding list of k output sets

[fairread]

Hi I wonder if a single Scan execution could perform k-fold cross validation using a list of k input sets and a corresponding list of k output sets to obtain the average performance metric. Specifically, my classification problem uses time series data containing episodes recorded at different times and durations. Ideally I wish to pass those episodes as a list of training sets into the Scan function, and that each set would take turn to be treated as a held out fold while the other sets are being used together as a training set.

[github-actions[bot]]

Welcome to Talos community! Thanks so much for creating your first issue :)

[mikkokotila]

Do you mean that you would like to use k-fold approach for each permutation in the experiment?

[fairread]

@mikkokotila basically yes but with my own pre-splitted folds stored in a list.

[mikkokotila]

This is possible now. After you have completed the scan, you can do that now. Let's say you want to use val_acc and split your data into 5. Then make sure x_val is made of the 5 datasets you want to use and are equally sized. Now you just have to make sure that n_models is the number of permutations in the experiment and Shuffle is False. You will get cross-validation for every permutation.

scan_object.evaluate_models(x_val, y_val, task, n_models, metric='val_acc', folds=5, shuffle=False, asc=False)

Closing here. Feel free to open new issue if anything.