Statistical Performance
K-fold cross validation is to be done on the HCI side instead of ML. The plots for the other statistical performance measures, namely:
Receiver-operating curves
Specificity-precision curves
Plots of feature importance could be done by either HCI or, upon request, by ML.
The original spec from Oggie specifically says:
• To ensure statistical robustness, the performance of the above should be underlain
by k-fold cross validation
And later:
More elaborate (but very common) statistical performance measures should also
be provided, such as the corresponding Receiver-Operating Curves (ROCs), and
specificity-precision curves.
Meanwhile, an email clarification from Oggie says:
My initial plan was to have the ML group create the plots and exports that you referred to. However, during the Q&A session this plan has changed in response to people's preferences. In other words, ML groups will now create the corresponding results (e.g. a list of values of the importance of different features) and pass these to other relevant groups to produce plots or files.
So from the email, ML clearly does not need to create any plots (I suppose people can do it as an extension if they want to, but it definitely shouldn't be on the spec otherwise everyone will think they need to do it), but they do still definitely need to perform the statistical analysis themselves. The only change from the original spec being that the data is left in raw CSV format for BE to export to different formats and for HCI to render.
The revised spec makes it sound like it's optional for ML to perform the statistical analysis and that HCI can do it instead, but that's not what is implied by any material we've received, nor is it something we really discussed.
To quote the recently revised ML spec:
The original spec from Oggie specifically says:
And later:
Meanwhile, an email clarification from Oggie says:
So from the email, ML clearly does not need to create any plots (I suppose people can do it as an extension if they want to, but it definitely shouldn't be on the spec otherwise everyone will think they need to do it), but they do still definitely need to perform the statistical analysis themselves. The only change from the original spec being that the data is left in raw CSV format for BE to export to different formats and for HCI to render.
The revised spec makes it sound like it's optional for ML to perform the statistical analysis and that HCI can do it instead, but that's not what is implied by any material we've received, nor is it something we really discussed.