sniklaus / pytorch-pwc

a reimplementation of PWC-Net in PyTorch that matches the official Caffe version
GNU General Public License v3.0
608 stars 122 forks source link

Cannot inspect the model using TensorBoard #47

Closed lollows closed 3 years ago

lollows commented 3 years ago

I want to visualize the model using tensorboard. And some error happens:

C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\tensor.py:455: RuntimeWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  'incorrect results).', category=RuntimeWarning)
Traceback (most recent call last):
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 549, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends\cuda\libs\nvrtc.pyx", line 112, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends\cuda\libs\nvrtc.pyx", line 124, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends\cuda\libs\nvrtc.pyx", line 58, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 354, in <module>
    writer.add_graph(netNetwork, (tenPreprocessedFirst, tenPreprocessedSecond))
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\utils\tensorboard\writer.py", line 714, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose))
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\utils\tensorboard\_pytorch_graph.py", line 285, in graph
    trace = torch.jit.trace(model, args)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\jit\__init__.py", line 955, in trace
    check_tolerance, strict, _force_outplace, _module_class)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\jit\__init__.py", line 1109, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, strict, _force_outplace)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 720, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 704, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 307, in forward
    objEstimate = self.netSix(tenFirst[-1], tenSecond[-1], None)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 720, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 704, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 215, in forward
    input=correlation.FunctionCorrelation(tenFirst=tenFirst, tenSecond=tenSecond),
  File "./correlation\correlation.py", line 386, in FunctionCorrelation
    return _FunctionCorrelation.apply(tenFirst, tenSecond)
  File "./correlation\correlation.py", line 295, in forward
    'output': rbot0
  File "cupy\_util.pyx", line 59, in cupy._util.memoize.decorator.ret
  File "./correlation\correlation.py", line 275, in cupy_launch
    return cupy.cuda.compile_with_cache(strKernel).get_function(strFunction)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 373, in compile_with_cache
    cache_in_memory)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 437, in _compile_with_cache_cuda
    log_stream, cache_in_memory)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 212, in compile_using_nvrtc
    name_expressions, log_stream)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 195, in _compile
    ptx, mapping = prog.compile(options, log_stream)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 561, in compile
    'nvrtc' if not runtime.is_hip else 'hiprtc')
cupy.cuda.compiler.CompileException: C:\Users\ADMINI~1\AppData\Local\Temp\tmpthz_eabd\256b2afebe918404edfb6592392d441b_2.cubin.cu(16): error: identifier "tensor" is undefined

1 error detected in the compilation of "C:\Users\ADMINI~1\AppData\Local\Temp\tmpthz_eabd\256b2afebe918404edfb6592392d441b_2.cubin.cu".

Here is some of the code I used.

from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter('runs/test')
writer.add_graph(netNetwork, (tenPreprocessedFirst, tenPreprocessedSecond))
writer.close()

I have tested the run.py, it works well. Please let me know if I missed something. Thank you.

sniklaus commented 3 years ago

I am afraid that my time is limited and I am unable to handle such support requests, my apologies.

sniklaus commented 2 years ago

Seems like thanks to @Etienne66 this might work now?

Etienne66 commented 2 years ago

Here is one of the Graphs that I was able to create.

2022-03-09T030431flow_pwc_test_epoch_0

sniklaus commented 2 years ago

Looks awesome, thanks for sharing!