A few models predict the same number for all inputs (just for some seeds). What should I do?

[karolinachalupova]

@martinhronec I had a deeper look on the models by random seed for models trained on 12 years and models trained on 13 years. I discovered that a few of them (3/(592) = 3 percent of models) learned to predict the same number no matter what input. It appears in deep models in particular. The following models suffer from the issue:

• among models trained on 12 years:
  • architecture with 4 hidden layers, 5th seed
  • architecute with 5 hidden layers, 8th seed
• among models trained on 13 year:
  • architecture with 5 hidden layers, 9th seed

The predicted number is always very close to the mean of the training data.

What do you think I should do about these models?

[martinhronec]

@KarolinaChalupova Try using SGD instead of ADAM and try adding batchnorm.

[karolinachalupova]

@martinhronec batchnorm already there, will try SGD. Thanks a lot!