Unexpected floating ScalarType in at::autocast::prioritize

[urbste]

I was just trying out the demo_fundamental.py demo and ran into the following error:

Unexpected floating ScalarType in at::autocast::prioritize

It occurs in the grid_sampler here (local_correlation.py, line 40)

local_window_coords = (coords[_,:,:,None]+local_window[:,None,None]).reshape(1,h,w*(2*r+1)**2,2).half()
window_feature = F.grid_sample(
                feature1[_:_+1], local_window_coords, padding_mode=padding_mode, align_corners=False, mode = sample_mode,)

It can be solved by removing the half precision of local_window_coords and converting feature1 to float(). However, I am not sure about any implications that might have.

local_window_coords = (coords[_,:,:,None]+local_window[:,None,None]).reshape(1,h,w*(2*r+1)**2,2)
window_feature = F.grid_sample(
                feature1[_:_+1].float(), local_window_coords, padding_mode=padding_mode, align_corners=False, mode = sample_mode,)

System/Env:

• Python 3.10
• Pytorch 2.0.1
• Cuda 11.7
• RTX 3090

[Parskatt]

I think its due to autocast not wanting to mix bfloat16 and float16, the .half() call I think I only used to try to save memory. If you dont have any memory issues then float should be fine, its probably also fine to not cast at all as I guess linspace is automatically float32.

[Parskatt]

Btw reason for bfloat16 is that its more stable during training, but I think I should make it optional as I think float16 can be faster at inference.

[urbste]

I will close this issue. Brought it up, if someone else has this problem. Thanks for the response :)

[Parskatt]

Thanks for bringing it up! I'll keep it in mind when cleaning up the code :)

[LooperzZ]

Hello, I have the same problem, it still does not work after modifying it according to your method, how to proceed? Unexpected floating ScalarType in at::autocast::prioritize

[Parskatt]

You could try explicitly casting everything to float32? Whats your pytorch/cuda version?

[LooperzZ]

python==3.7 pytorch==1.10.2 cuda==11.3

[mpizenberg]

Thanks @urbste this saved me from searching for hours what the issue is ^^.

[Parskatt]

As many people seem to be having this issue Ill try to fix it.

[Parskatt]

@mpizenberg @LooperzZ , are your issues resolved with https://github.com/Parskatt/RoMa/pull/3 ?

[mpizenberg]

This is my diff

[mpizenberg]

ah sorry I missinterpreted the question. I can check #3 and let you know.

[mpizenberg]

@mpizenberg @LooperzZ , are your issues resolved with #3 ?

Indeed, I don’t have this issue anymore in that branch so I’d say it’s solved.

[direct-up]

Thanks very much! By removing the half precision from local_window_coords and converting feature1 to float using float(), the issue of 'Unexpected floating ScalarType in at::autocast::prioritize' is resolved. System/Env:

• Python 3.9.9
• Pytorch 2.0.0
• Cuda 11.7
• GRID A100X（rev a1）

PS：I have already tried version #3, and the issue does not occur again.