Closed ntraut closed 2 years ago
@kegl @albertcthomas @rth any thought on this?
Unfortunately, I am not familiar with the blending feature and the corresponding part of the code.
@kegl we need you here :)
As nobody seems to know I tried to understand by looking at the code.
In the function blend_submissions
:
https://github.com/paris-saclay-cds/ramp-workflow/blob/cbc3e831d6598d69810bd1bfe5866e6cbf670374/rampwf/utils/testing.py#L175-L178
the one which does not go with the other variables is the name of the csv, which is given to the function bag_submissions
with the argument score_f_name_prefix
.
So the real scores are the ones reported by the script and and the names of the csv files are wrong.
For the difference between foldwise and combined, from what I understand in foldwise we just take the most performant submission in each fold, whereas in combined the real blending occurs, i.e. in each fold we try to combine a set of submissions to obtain the best possible score, but doing so the combination is also foldwise, which may be the source of the confusion.
can you send a doc PR to document this? that will be useful in the future :) thx
I did a PR to fix the issue but I don't know where we should document the different outputs of ramp_blend_submissions
, do you have an idea for this?
We should document it here: https://paris-saclay-cds.github.io/ramp-docs/ramp-workflow/stable/command_line.html
Simply add a line to https://github.com/paris-saclay-cds/ramp-workflow/blob/master/doc/command_line.rst
@ntraut if you do it in the current PR #285 , we can merge it immediately.
Thanks a lot for spotting this!
When we run
ramp_blend_submissions
we get two different combined auc scores: "Combined bagged scores" and "Foldwise best bagged scores". For example:But when we compute the auc from the combined predictions which are in the folder
submissions/training_output
, we see that what is reported in "Combined bagged scores" corresponds to the auc fromy_pred_foldwise_best_bagged_valid.csv
and "Foldwise best bagged scores" corresponds to the auc fromy_pred_combined_bagged_test.csv
.So which one is right, and incidentally, is there a way of combining which should be preferred over the other?