RuntimeError: The backward pass for this operation requires the 'self' tensor to be strided, but a sparse tensor was given instead.

🐛 Bugs / Unexpected behaviors

When I try to use Pytorch3d as a differential operator to do some operation like the taubin_smoothing, if the vertices of the mesh require the grad, then the error ouccrs.

Instructions To Reproduce the Issue:

The code to Reproduce the Issue is uploaded as BUG.zip.

And I also write it here:

'''
ENV:
torch = 1.12.1+cu113
pytorch3d = 0.7.0 , I install it from https://anaconda.org/pytorch3d/pytorch3d/files
CUDA = 11.4
RTX 3080
'''
import torch
import numpy as np
import trimesh
import pytorch3d as p3d
from pytorch3d.structures import Meshes
from pytorch3d.ops import taubin_smoothing

def load_mesh_pml(mesh_file):
    meshset = pml.MeshSet()
    meshset.load_new_mesh(mesh_file)
    return meshset

if __name__ == '__main__':
    mesh_file = "tshirt.ply"
    # 参数; taubin_smooth parameters
    lam = 0.5
    mu = -0.53
    num_iter = 10
    # 加载mesh, 获取基本信息; Load a trimesh
    mesh = trimesh.load_mesh(mesh_file)
    edges = mesh.edges
    faces = np.asarray(mesh.faces)
    verts = np.asarray(mesh.vertices)
    # to tensor; Please note When .requires_grad_(True) is set, the bug occurs.
    verts_t = torch.tensor(verts).cuda().requires_grad_(True)
    faces_t = torch.tensor(faces).cuda()

    mesh_t = Meshes(verts=[verts_t], faces=[faces_t])
    lap_single = taubin_smoothing(mesh_t, num_iter=num_iter, lambd=lam, mu=mu).verts_list()[0]

The bug logs:

Traceback (most recent call last):
  File "xxxx/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3508, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-606074cc5014>", line 1, in <module>
    runfile('/laplacian.py', ')
  File "/snap/pycharm-community/383/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/snap/pycharm-community/383/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/laplacian.py", line 47, in <module>
    lap_single = taubin_smoothing(mesh_t, num_iter=num_iter, lambd=lam, mu=mu).verts_list()[0]
  File "xxxx/lib/python3.8/site-packages/pytorch3d/ops/mesh_filtering.py", line 50, in taubin_smoothing
    verts = (1 - lambd) * verts + lambd * torch.mm(L, verts) / total_weight
**RuntimeError: The backward pass for this operation requires the 'self' tensor to be strided, but a sparse tensor was given instead. Please either use a strided tensor or set requires_grad=False for 'self'**

Solution

I look the code in pytorch3d/ops/mesh_filtering.py taubin_smoothing.

    for _ in range(num_iter):
        L = norm_laplacian(verts, edges)
        total_weight = torch.sparse.sum(L, dim=1).to_dense().view(-1, 1)
        verts = (1 - lambd) * verts + lambd * torch.mm(L, verts) / total_weight

        L = norm_laplacian(verts, edges)
        total_weight = torch.sparse.sum(L, dim=1).to_dense().view(-1, 1)
        verts = (1 - mu) * verts + mu * torch.mm(L, verts) / total_weight

It seems that view(-1, 1) causes the error. So I make a small change as follows:

    for _ in range(num_iter):
        L = norm_laplacian(verts, edges)
        total_weight = torch.sparse.sum(L, dim=1)
        total_weight = total_weight.unsqueeze(-1).to_dense()
        verts = (1 - lambd) * verts + lambd * torch.sparse.mm(L, verts)
        verts = verts/total_weight

        L = norm_laplacian(verts, edges)
        total_weight = torch.sparse.sum(L, dim=1)
        total_weight = total_weight.unsqueeze(-1).to_dense()
        verts = (1 - mu) * verts + mu * torch.sparse.mm(L, verts) / total_weight

Then it works.

May it can help you.

Another Bug during Backward():

When I try to do the backward() as the following:

lap_single = lap_single.norm()
lap_single.backward()

It reports:

Traceback (most recent call last):
  File "xxxx//IPython/core/interactiveshell.py", line 3508, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-606074cc5014>", line 1, in <module>
    runfile('xxxx/laplacian.py', wdir='xxxx')
  File "/snap/pycharm-community/383/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/snap/pycharm-community/383/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "xxxx/laplacian.py", line 50, in <module>
    lap_single.norm().sum().backward()
  File "xxxx//torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "xxxx//torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Tensors of type SparseTensorImpl do not have strides

solution

I find that it is the total_weight causes this error. So I have to give up the usage of taubin_smoothing. I replace the taubin_smoothing with the original Laplacian smoothing.

Again, go to pytorch3d/ops/mesh_filtering.py taubin_smoothing. ADD:

from pytorch3d.ops import laplacian

and Change the loop code in def taubin_smoothing:

for _ in range(num_iter):
  L = laplacian(verts, edges)
  verts = verts + torch.sparse.mm(L, verts)

Note that, the num_iter should set to 3 or 5, not to large.

facebookresearch / pytorch3d