snk4tr
commented
2 years ago Hi @v-haopt12, the error that you are getting is not a bug but an expected behaviour. What you are trying to do is to make a single shot computation of the metric on relatively large input tensors. As stated in the error message, the amount of available GPU memory is not enough for this operation. My suggestion would be to use smaller chunks of data several times and reduce your results afterwords. Feel free to re-open if you still think that there is something wrong with the library.
Describe the bug Got CUDA out of memory when calling PR()(feat1, feat2) where feat1 has torch.Size([49021, 2048]) and feat2 has torch.Size([50000, 2048]).
Expected behavior Able to compute PR without error
Screenshots
File "/lustre/scratch/client/vinai/users/haopt12/diffae_gitlab/pytorch_fid/pr.py", line 65, in calculate_pr_given_paths prec, rec = pr_metric(feat1, feat2) File "/lustre/scratch/client/vinai/users/haopt12/envs/ddpm/lib/python3.8/site-packages/torch /nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/lustre/scratch/client/vinai/users/haopt12/envs/ddpm/lib/python3.8/site-packages/piq/b ase.py", line 15, in forward return self.compute_metric(x_features, y_features) File "/lustre/scratch/client/vinai/users/haopt12/envs/ddpm/lib/python3.8/site-packages/piq/p r.py", line 110, in compute_metric torch.isclose(distance_real_fake, real_nearest_neighbour_distances) RuntimeError: CUDA out of memory. Tried to allocate 9.13 GiB (GPU 0; 39.59 GiB total capacity; 24.40 GiB already allocated; 7.09 GiB free; 30.65 GiB reserved in total by PyTorch) Testing DataLoader 0: 0%| | 0/1 [38:15<?, ?it/s]Additional context Evaluation of a generative model on FFHQ 256.