autocast for inference

[javierquincke]

Hi, I've been using this repo and https://github.com/plemeri/transparent-background excellent work!

Now I'm trying to use autocast to lower the inference times a bit, I couldn't get it to work correctly

What I did was replace this part of the code https://github.com/plemeri/transparent-background/blob/main/transparent_background/Remover.py#L111 with this:

with torch.no_grad():
            with autocast():
                pred = self.model(x)

the model runs apparently without errors but the result of pred is a vector of nan like this:



tensor([[[[nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          ...,
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan]]]], device='cuda:0',
       dtype=torch.float16)

Do you have any suggestion?

[plemeri]

I think the autocast that you are trying to do is to use lower precision, such as half precision. It is convenient but could throw some error like this.

I'm not sure about the result but you can try casting manually. I think you can use a wrapper to change the model itself to support lower precision.

Also, I personally do not recommend using half precision for all layers. Use half precision on backbone network only instead.

Thanks.