search

eldar / pose-tensorflow

Human Pose estimation with TensorFlow framework
GNU Lesser General Public License v3.0
1.14k stars 384 forks source link

Make model deterministic #98

Closed sebo361 closed 5 years ago

sebo361 commented 5 years ago

Hello,

i am wondering if it is possible to make the model deterministic? I tried to use the TF_CUDNN_DETERMINISTIC flag and tf.set_random_seed(1) but I could not get deterministic behavior. Which part exactly brings stochasticity in?

eldar commented 5 years ago

Hi, It's a little trickier than that, this code uses numpy random generator to shuffle images: https://github.com/eldar/pose-tensorflow/blob/master/dataset/pose_dataset.py#L176-L181 So the random seed has to be specified for the numpy.random too. Can you try that?

sebo361 commented 5 years ago

Hi @eldar, thanks for helping! I tried it with DeepLabCut (same code as yours) https://github.com/AlexEMG/DeepLabCut/blob/20bb84dcdb740a73c8644d8abe70c5b20e64078f/deeplabcut/pose_estimation_tensorflow/dataset/pose_dataset.py#L140-L148 but unfortunately I don't get deterministic behavior as the loss value varies when starting training with the same settings.. So if input images and pretrained weights (i use the pretrained resnet101 on MPII) are the same, do you know what else can bring in stochasticity? Thanks so much for helping!

sebo361 commented 5 years ago

Issue solved by https://github.com/AlexEMG/DeepLabCut/pull/324. Thank you for helping @eldar!