Fix dtype casting in swinv2 and swinv2sr to allow non-FP32 inference

[aliencaocao]

What does this PR do?

The current implementation uses .float() in https://github.com/huggingface/transformers/blob/0f67ba1d741d65b07d549daf4ee157609ce4f9c1/src/transformers/models/swin2sr/modeling_swin2sr.py#L286-L287 which causes subsequent relative_coords_table to be always in torch.float32, not respecting whatever precision the other weights might be, e.g. torch.float16.

This PR adds a cast to the same dtype as the continuous_position_bias_mlp layer since relative_coords_table is being passed directly into the layer at https://github.com/huggingface/transformers/blob/0f67ba1d741d65b07d549daf4ee157609ce4f9c1/src/transformers/models/swin2sr/modeling_swin2sr.py#L349

Same issue & fix for swinv2

Prerequisite for https://github.com/huggingface/transformers/pull/31342 image to image pipeline FP16 test to pass.

Before submitting

• [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x] Did you read the contributor guideline, Pull Request section?
• [x] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• [x] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• [x] Did you write any new necessary tests?

Who can review?

@amyeroberts

[aliencaocao]

Added them, they pass locally

[amyeroberts]

@aliencaocao Great! Could you push an empty commit with the message: [run_slow] swin2sr, swinv2. I trust the tests are passing locally, but because of differences that can creep in because of hardware and env set-up, the logits can still be slightly different. So let's make sure the numbers match what's going to be running on the CI : )

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[aliencaocao]

@amyeroberts need your approval for slow tests

[aliencaocao]

ugh it seems the specific gpu used indeed has numerical difference... I ran the tests and got logit using RTX 3080Ti, torch2.3.1+cu121, nvidia 555.95 driver, Windows 10

[amyeroberts]

@aliencaocao As the change is simple, and look OK, updating the tests to use the results from the CI runs should be OK

[aliencaocao]

How do I get the CI outputs? Do I have to print in CI?

[amyeroberts]

@aliencaocao Good question! Indeed, they're not part of the console output. Let me see if I can ssh in

[amyeroberts]

@aliencaocao Running on the runners, I get the following logits

swin2sr

tensor([[0.5454, 0.5542, 0.5640],
        [0.5518, 0.5562, 0.5649],
        [0.5391, 0.5425, 0.5620]], device='cuda:0', dtype=torch.float16)

swinv2

tensor([-0.3938, -0.4290,  0.0020], device='cuda:0', dtype=torch.float16)

[aliencaocao]

Thanks, triggered again

[amyeroberts]

@aliencaocao Thanks! All looks good - we can merge 🤗