RuntimeError: GET was unable to find an engine to execute this computation

[bobhagen]

Hi @gmberton,

Would you help me to fix the following error?

2024-06-05 21:50:30 main.py --method=mixvpr --backbone=ResNet50 --descriptors_dimension=512 --database_folder=../VPR-datasets-downloader/msls/val/database/ --queries_folder=../VPR-datasets-downloader/msls/val/queries/ --image_size 480 640 2024-06-05 21:50:30 Arguments: Namespace(positive_dist_threshold=25, method='mixvpr', backbone='ResNet50', descriptors_dimension=512, database_folder='../VPR-datasets-downloader/msls/val/database/', queries_folder='../VPR-datasets-downloader/msls/val/queries/', num_workers=4, batch_size=4, log_dir='default', device='cuda', recall_values=[1, 5, 10, 20], no_labels=False, num_preds_to_save=0, save_only_wrong_preds=False, image_size=[480, 640], use_labels=True) 2024-06-05 21:50:30 Testing with mixvpr with a ResNet50 backbone and descriptors dimension 512 2024-06-05 21:50:30 The outputs are being saved in logs/default/2024-06-05_23-50-30 Searching test images in ../VPR-datasets-downloader/msls/val/database/ with glob() Searching test images in ../VPR-datasets-downloader/msls/val/queries/ with glob() 2024-06-05 21:50:32 Testing on < #queries: 11084; #database: 18871 > 0%| | 0/4718 [00:01<?, ?it/s] 2024-06-05 21:50:33
Traceback (most recent call last): File "/home/robert/VPR-methods-evaluation/main.py", line 43, in descriptors = model(images.to(args.device)) File "/home/robert/.pyenv/versions/vpr_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/robert/VPR-methods-evaluation/vpr_models/mixvpr.py", line 118, in forward x = self.backbone(x) File "/home/robert/.pyenv/versions/vpr_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/robert/VPR-methods-evaluation/vpr_models/mixvpr.py", line 100, in forward x = self.model.conv1(x1) File "/home/robert/.pyenv/versions/vpr_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/robert/.pyenv/versions/vpr_env/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 463, in forward return self._conv_forward(input, self.weight, self.bias) File "/home/robert/.pyenv/versions/vpr_env/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward return F.conv2d(input, weight, bias, self.stride, RuntimeError: GET was unable to find an engine to execute this computation

Best, Robert

[gmberton]

Hi, it seems to be a CUDA issue. You can try to run with --device=cpu and see if that works

[bobhagen]

Thank you @gmberton. Yes, it works on CPU. I was able to reproduce the results reported in #7. It is weird though, I can run my other pytorch models on GPU. When you have time, you might still want to check if you can reproduce the error I mentioned. I will let you know if I can find what prevents me from running your code on GPU.

Best, Robert

[gmberton]

I can't reproduce it, the code is running smoothly on my end. Perhaps you could try reinstalling torch or creating a new cuda env? Not sure, just throwing ideas. Anyway, good to know that it works, please reply here if ever you manage to fix it so that others can follow your guide, thank you!