Can't run burgers - gradient calculation failure

[LupusTheCanine]

When trying to run burgers I get the following error:

(eventOF) lupus@DESKTOP-IFAQBRF:~/event_based_optical_flow$ python ./main.py --config_file ./configs/mvsec_indoor_burgers.yaml
Torch scatter needs to run some special interpolation.
2022-12-02 11:49:33,906 - src.data_loader.base - INFO - Loading directory in /home/lupus/event_based_optical_flow/datasets/MVSEC/hdf5/MVSEC
2022-12-02 11:49:33,906 - src.data_loader.base - INFO - No undistortion.
2022-12-02 11:49:33,906 - src.data_loader.mvsec - INFO - Use sequence indoor_flying1
2022-12-02 11:49:33,907 - src.data_loader.mvsec - INFO - Undistort events = False
2022-12-02 11:49:36,851 - src.data_loader.mvsec - INFO - Loading ground truth flow /home/lupus/event_based_optical_flow/datasets/MVSEC/gt_flow/indoor_flying1_gt_flow_dist.npz
2022-12-02 11:49:45,186 - src.data_loader.mvsec - INFO - Use only valid frames.
2022-12-02 11:49:45,770 - src.data_loader.mvsec - WARNING - directly load calib_param is not implemented!! please use rectify instead.
2022-12-02 11:49:45,778 - src.solver.patch_contrast_pyramid - INFO - Pyramidal patch.
2022-12-02 11:49:45,934 - src.solver.base - INFO - Load hybrid cost
2022-12-02 11:49:45,945 - src.costs.hybrid - INFO - Log functions are mix of {'multi_focal_normalized_gradient_magnitude': 1.0, 'total_variation': 0.01}
2022-12-02 11:49:46,129 - src.warp - INFO - Set camera matrix K.
2022-12-02 11:49:46,132 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,132 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,139 - src.solver.base - INFO - Setup time-aware parameters
2022-12-02 11:49:46,140 - src.solver.base - INFO - No scaling before upwind
2022-12-02 11:49:46,141 - src.solver.base - INFO - Configuration:
    self.normalize_t_in_batch = True
2022-12-02 11:49:46,142 - src.solver.base - INFO - Configuration:
    {'n_iter': 40, 'method': 'Newton-CG', 'max_iter': 25, 'parameters': {'trans_x': {'min': -150, 'max': 150}, 'trans_y': {'min': -150, 'max': 150}}}
    {'method': 'pyramidal_patch_contrast_maximization', 'time_aware': True, 'time_bin': 10, 'flow_interpolation': 'burgers', 't0_flow_location': 'middle', 'patch': {'initialize': 'random', 'scale': 5, 'crop_height': 256, 'crop_width': 336, 'filter_type': 'bilinear'}, 'motion_model': '2d-translation', 'warp_direction': 'first', 'parameters': ['trans_x', 'trans_y'], 'cost': 'hybrid', 'outer_padding': 0, 'cost_with_weight': {'multi_focal_normalized_gradient_magnitude': 1.0, 'total_variation': 0.01}, 'iwe': {'method': 'bilinear_vote', 'blur_sigma': 1}}
2022-12-02 11:49:46,162 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,162 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,165 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,165 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,175 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,175 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,176 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,177 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,183 - __main__ - INFO - Single-frame optimization
2022-12-02 11:49:46,426 - src.solver.patch_contrast_pyramid - INFO - Start optimization.
2022-12-02 11:49:46,426 - src.solver.patch_contrast_pyramid - INFO - DoF is 680
2022-12-02 11:49:46,438 - src.solver.patch_contrast_pyramid - INFO - Scale 1
2022-12-02 11:49:46,439 - src.solver.patch_contrast_base - INFO - random initialization
/home/lupus/event_based_optical_flow/src/solver/scipy_autograd/scipy_minimize.py:100: OptimizeWarning: Unknown solver options: gtol
  optim_res = sopt.minimize(
2022-12-02 11:49:47,653 - src.solver.patch_contrast_pyramid - INFO - loss = tensor(4.7517, dtype=torch.float64, grad_fn=<AddBackward0>)
Traceback (most recent call last):
  File "/home/lupus/event_based_optical_flow/./main.py", line 179, in <module>
    best_motion: np.ndarray = solv.optimize(batch)
  File "/home/lupus/event_based_optical_flow/src/solver/patch_contrast_pyramid.py", line 161, in optimize
    best_motion_per_scale, opt_result = self.run_scipy_over_scale(events)
  File "/home/lupus/event_based_optical_flow/src/utils/misc.py", line 127, in wrapper
    retval = func(*args, **kwargs)
  File "/home/lupus/event_based_optical_flow/src/solver/patch_contrast_pyramid.py", line 194, in run_scipy_over_scale
    opt_result = self.run_scipy(events, best_motion_per_scale)
  File "/home/lupus/event_based_optical_flow/src/solver/patch_contrast_pyramid.py", line 303, in run_scipy
    result = scipy_autograd.minimize(
  File "/home/lupus/event_based_optical_flow/src/solver/scipy_autograd/scipy_minimize.py", line 100, in minimize
    optim_res = sopt.minimize(
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_minimize.py", line 696, in minimize
    res = _minimize_newtoncg(fun, x0, args, jac, hess, hessp, callback,
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 1849, in _minimize_newtoncg
    sf = _prepare_scalar_function(
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 263, in _prepare_scalar_function
    sf = ScalarFunction(fun, x0, args, grad, hess,
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 158, in __init__
    self._update_fun()
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 251, in _update_fun
    self._update_fun_impl()
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 155, in update_fun
    self.f = fun_wrapped(self.x)
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in fun_wrapped
    fx = fun(np.copy(x), *args)
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 76, in __call__
    self._compute_if_needed(x, *args)
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 70, in _compute_if_needed
    fg = self.fun(x, *args)
  File "/home/lupus/event_based_optical_flow/src/solver/scipy_autograd/torch_wrapper.py", line 40, in get_value_and_grad
    grads = torch.autograd.grad(loss, input_var_grad)
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/torch/autograd/__init__.py", line 300, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: could not compute gradients for some functions
(eventOF) lupus@DESKTOP-IFAQBRF:~/event_based_optical_flow$ python ./main.py --config_file ./configs/mvsec_indoo

[shiba24]

Hi, can you tell me the version of python, torch, and scipy? And are you using CUDA?

[LupusTheCanine]

python 3.10.8 torch 1.13.0 scipy 1.9.3 No CUDA.

[Zehaoc]

same issue: python 3.8.15 torch 1.13.0+cu116 scipy 1.8.1

[shiba24]

Hmm, thank you for reporting.

My environment is as followings:

Python 3.9.14
scipy==1.9.1
torch==1.12.1
torchvision==0.13.1

And it works.

I just tested with torch 1.13.0 and got the same error that you faced.

Can you try installing torch 1.12.1 (or any version before 1.13)?

[shiba24]

Please let me know if downgrading pytorch version solves the error in your environment - then I will add it to the dependency.

[shiba24]

@LupusTheCanine @Zehaoc Hi, any update on this?

[shiba24]

Hey, I will close this issue if you don't respond. Please check the torch version and reopen this issue if you still have issues. Thanks!

[LupusTheCanine]

Sorry for late reply. Downgrading torch helps.

[shiba24]

By the way, I quickly checked torch 2.0.0 worked. Maybe torch 1.13.0 has some bugs.