search

slds-lmu / paper_2019_multiobjective_rfms

High Dimensional Restrictive Federated Model Selection with multi-objective Bayesian Optimization over shifted distributions
https://github.com/smilesun/tex_2018_intellisys2019_fmoboms
2 stars 0 forks source link

is it possible to have: when train on the whole dataset, mmce is 1, but the cv mmce score on the same can be 0.4? #16

Closed smilesun closed 5 years ago

pfistfl commented 5 years ago

It is possible for sure, Do you know how many observations are in each CV split?

smilesun commented 5 years ago

around 100 instances, actually when you source test.R , you will observe this when meas_openbox_cv is used.

smilesun commented 5 years ago

florian mentioned that in the measures I implemented, the labels got inverted, I have not find out where. Do you have some experience? @mllg

https://github.com/compstat-lmu/paper_2018_fmoms/blob/0f94470a6ca385c57d7690e0647ab6904c8c259e/R/bt_measures_objs.R#L18

smilesun commented 5 years ago

confirmed this is a bug from kernlab