Closed htlpt closed 1 year ago
I noticed the NaN in the training, possibly FP16 overflow?
Train done, test.
0: 17.752699
8: 12.5911665
16: 12.909588
24: 9.543803
32: 11.67271
40: 12.495184
48: 11.199245
Mean psnr: 12.594914
low PSNR of course...
It's just the instablity of CUDA on Windows, WSL2 runs just fine...
I was building on Win11, torch 2.0.1, CUDA 11.8, RTX4090 training the example ngp_fox, seeing voxel artifacts, could it be the platform, the GPU(mix-precision ?), or just torch version..