zeyiwen
commented
3 years ago The training should be stable. The only randomness is due to parallelism in finding min/max from an array. If there are multiple min/max values in an array, any one of the min/max values may be used in the training. However, the difference due to this parallelism is tiny. As you get large differences (?), I anticipate the way you use it may be incorrect.
Would you provide a simple example for us to reproduce the problem?
I am doing hyperparameter search using SVR class, I see that sometimes same parameter set generates different loss values. and it throws off the bayesian hyperparameter algorithm and it goes.
Any idea what might be going on? There is always some level of randomness when using GPU, but with 10-fold cross validation repeated 5 times, I expect the rmse to be within 1% for same parameter set, not more than 5%