search

isl-org / DPT

Dense Prediction Transformers
MIT License
2.04k stars 260 forks source link

segmentation failure on Titan RTX #47

Open llbb3322 opened 3 years ago

llbb3322 commented 3 years ago

I tested segmentation on Titan RTX and it outputs black mask with some white noises.

It seems that "optimize" option caused that problem.

The problem is solved by deleting 2 lines about "memory_format=torch.channels_last"

but I did not figure out the reason.

ranftlr commented 3 years ago

Thanks for reporting. You can also disable the optimization by calling the script with the --no-optimize flag. The optimizations refer to common speed improvements and are completely optional.

As for the reason for this issue: I wasn't able to find a reason either. I don't have a Titan RTX to debug myself, unfortunately. If you really need the optimization, my only recommendation is to make sure to have the latest drivers/CUDA/Pytorch installed.

We'd be also happy to hear about any solutions that you might find.

llbb3322 commented 3 years ago

It seems that half accuracy method ( "model.half()" & "sample.half()" ) is helpful for optimization, but the "torch.channels_last" stuff will cause nonsense output on TITAN RTX or just slow it down on Geforce 1080.

By the way, I tested on CUDA 11.3, Pytorch 1.8.1, Python 3.9