CUDA memory leak

[PaulHax]

Steps

1. Load up the OIRDS dataset from nrtk_explorer_datasets repo.
2. Compute embeddings for 300+
3. Select a chunk out of the cloud with box select
4. Error: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 983.00 MiB. GPU
5. Select a smaller set of images. All OK.
6. Select a slightly bigger set, another error indicating leak: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 314.00 MiB. GPU

[vicentebolea]

This is probably not a leak but a OOM after loading a big dataset, we have to adjust the batching parameter for larger datasets depending on one's hw.

[PaulHax]

The out of memory "Tried to allocate" number threshhold keeps going down (after a sucessfull selection of a smaller set)