searching valid metric is better than re-train valid metric in autoFIS?

[guvcolie]

Thank you for your excellent codes. I run your autoFIS codes (autoDeepFM), but found that the valid metric after searching (auc=0.7896248002165958) is better than the retrain valid metric (auc=0.7851094769386633) in the log. i.e. [searching valid metric] > [re-train valid metric] > [official valid metric], it looks like the performance improvement is achieved by BN but autofis damages the valid metric. Something wrong ?

the valid metric after searching: the valid metric after re-train:

[chenboability]

It seems that this may be the result of an overfit found in the search stage, resulting a little degradation in the re-train stage.

[guvcolie]

但是我用不同seed做了10次重复实验，大概率还是 search阶段valid指标 优于 re-train阶段的valid指标，也就是说 autofis整体带来的指标提升看起来是 search阶段的高阶特征后跟了BN且有alpha系数的加持 带来的，剔除掉一些高阶特征valid指标反倒降低了。那这样的话，autoFIS意义何在呢？