Out of memory error on Multi-ICP with open3d tensor point clouds

Checklist

[X] I have searched for similar issues.
[X] For Python issues, I have tested with the latest development wheel.
[X] I have checked the release documentation and the latest documentation (for master branch).

Describe the issue

I have been running multi ICP in the tensor form and whenever I reach a certain sample it runs to out of memory:

RuntimeError: [Open3D Error] (void open3d::core::__OPEN3D_CUDA_CHECK(cudaError_t, const char*, int)) /root/Open3D/cpp/open3d/core/CUDAUtils.cpp:289: /root/Open3D/cpp/open3d/core/MemoryManagerCUDA.cpp:24 CUDA runtime error: out of memory

I used this example as help: http://www.open3d.org/docs/release/tutorial/t_pipelines/t_icp_registration.html#Multi-Scale-ICP-Example

I have two point clouds loaded as a torch tensor and can correctly create a open3d.t Point cloud and multi-ICP is working with 1.9GB GPU memory occupied but after 100 point cloud pair predictions the memory jumps to 3GB and after the 250th pair it runs out of memory.

I also tried to specificaly del source_pcd, target_pcd without help.

I hope you can help me.

Steps to reproduce the bug

import open3d as o3d

RT_pred = o3d_multi_icp_tensor(source, target)

def o3d_multi_icp_tensor(source: torch.Tensor,
                                                 target: torch.Tensor,
                                                 trans_init: Union[torch.Tensor, np.ndarray] = torch.eye(4),
                                                 voxel_sizes = [0.5, 0.3, 0.1],
                                                 corr_distances = [1.5, 0.9, 0.3],
                                                 max_iter = [20, 15, 10],
                                                 rmse_fit = [0.001, 0.0001, 0.00001]):
if isinstance(trans_init, torch.Tensor):
    trans_init = trans_init.cpu().numpy()
voxel_sizes = o3d.utility.DoubleVector(voxel_sizes)
criteria_list = []
for i in range(len(max_iter)):
    criteria_list.append(o3d.t.pipelines.registration.ICPConvergenceCriteria(relative_fitness=rmse_fit[i],
                                             relative_rmse=rmse_fit[i],
                                             max_iteration=max_iter[i]))

    max_correspondence_distances = o3d.utility.DoubleVector(corr_distances)
    init_source_to_target = o3d.core.Tensor.eye(4, o3d.core.Dtype.Float32)
    estimation = o3d.t.pipelines.registration.TransformationEstimationPointToPlane()

    reg_p2p = o3d.t.pipelines.registration.multi_scale_icp(source, target, voxel_sizes,
    criteria_list,
    max_correspondence_distances,
    init_source_to_target, estimation)

return reg_p2p

Error message

RuntimeError: [Open3D Error] (void open3d::core::__OPEN3D_CUDA_CHECK(cudaError_t, const char*, int)) /root/Open3D/cpp/open3d/core/CUDAUtils.cpp:289: /root/Open3D/cpp/open3d/core/MemoryManagerCUDA.cpp:24 CUDA runtime error: out of memory

Expected behavior

No response

Open3D, Python and System information

- Operating system: Ubuntu 20.04 
- Python version: Python 3.9
- Open3D version: 0.17.0 (same problem in 0.16.0)
- System architecture: x86
- Is this a remote workstation?: yes
- How did you install Open3D?: pip

Additional information

No response

isl-org / Open3D