search

avirambh / ScopeFlow

Dynamic Scene Scoping for Optical Flow (CVPR 2020)
Apache License 2.0
95 stars 10 forks source link

Error during eval on sintal. #5

Closed vivasvan1 closed 4 years ago

vivasvan1 commented 4 years ago 
Validate:   0% 0/23 [00:00<?, ?it/s]error in correlation_forward_cuda_kernel: invalid device function

Traceback (most recent call last):
  File "evaluate.py", line 41, in <module>
    evaluate()
  File "evaluate.py", line 37, in evaluate
    validation.validate(args)
  File "/content/ScopeFlow/lib/validation.py", line 93, in validate
    res = train_and_val(args)
  File "/content/ScopeFlow/lib/pipeline_wrapper.py", line 87, in train_and_val
    validation_augmentation=validation_augmentation)
  File "/content/ScopeFlow/lib/runtime.py", line 676, in exec_runtime
    augmentation=validation_augmentation).run()
  File "/content/ScopeFlow/lib/runtime.py", line 528, in run
    loss_dict_per_step, output_dict, batch_size = self._step(example_dict)
  File "/content/ScopeFlow/lib/runtime.py", line 451, in _step
    loss_dict, output_dict = self._model_and_loss(example_dict)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/ScopeFlow/lib/configuration.py", line 85, in forward
    output_dict = self._model(example_dict)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/ScopeFlow/models/IRR_PWC_V2.py", line 120, in forward
    out_corr_relu_f = self._correlate(x1, x2_warp)
  File "/content/ScopeFlow/models/IRR_PWC_V2.py", line 275, in _correlate
    out_corr = self.correlation(x1, x2_warp)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/ScopeFlow/models/correlation_package/correlation.py", line 59, in forward
    result = CorrelationFunction(self.pad_size, self.kernel_size, self.max_displacement, self.stride1, self.stride2, self.corr_multiply)(input1, input2)
  File "/content/ScopeFlow/models/correlation_package/correlation.py", line 27, in forward
    self.pad_size, self.kernel_size, self.max_displacement,self.stride1, self.stride2, self.corr_multiply)
RuntimeError: CUDA call failed (correlation_forward_cuda at correlation_cuda.cc:79)

How do i fix this error?

avirambh commented 4 years ago

Hi @vivasvan1,

It seems like a problem with the correlation package installation. I found this error in a few public issues (i.e here), maybe you need to change some configurations during the installation.

I hope this will help, Aviram

vivasvan1 commented 4 years ago

Can you take a look here?

https://colab.research.google.com/drive/1kF2RbKHpegIYGtZ2qULbHjy0_5rwhu5w?usp=sharing

avirambh commented 4 years ago

@vivasvan1 it seems that your cuda version is 10.1, I don't think it's compatible with the correlation package installation.

vivasvan1 commented 4 years ago

i am installing cuda 8.0 on colab. you can see the output of nvcc --version

avirambh commented 4 years ago

Yes but the CUDA version at the bottom (in the output of nvidia-smi) still shows 10.1 so I'm not sure what's in use.