Open mathpopo opened 4 years ago
looks like the train_loss does not drop too much. Have you follow the instruction to disable cudnn for batchnorm layer if you are using pytorch0.4.x?
i use titan xp ,can run well,this effect is in the v100,the two graphics cards are different?
disable cudnn for batchnorm layer: sed -i "1194s/torch.backends.cudnn.enabled/False/g"?i do this already,use pytorch 0.4.1,cuda 9.0
can you tell me your graphics cards hardware?
i use v100,resume from pre-trained model ,acc descend ,use titan xp , go
up
The gpu I used is titan xp, not v100. I guess some low-level difference b/w driver/cudnn caused the problem in v100.
thank you very mush.sorry ,other issue,if half body & side ,effect so bad,i descend threshold ,point error happen
this is .0.1 threshold effect
I try to run "python example/mpii.py -a hg --stacks 8 --blocks 1 --checkpoint checkpoint/hg_s8_b1/ " in your code , compare to your log.txt,loss can descend as same with you,but acc samll too mush,just "python example/mpii.py -a hg --stacks 8 --blocks 1 --checkpoint checkpoint/hg_s8_b1/ " is whole?![log](https://user-images.githubusercontent.com/21274466/66547260-6d5aa680-eb71-11e9-98ef-3e0b3b2dd8bc.png)