meet cuda runtime error : out of memory when using gpu function, such as voxel_down_sample

xiebinghua commented 1 year ago

Checklist

[X] I have searched for similar issues.
[X] For Python issues, I have tested with the latest development wheel.
[X] I have checked the release documentation and the latest documentation (for master branch).

Describe the issue

the open3d edition is 0.16.0 The situation is like this: first, I read the point cloud data onto the GPU, and then I perform downsampling operations on it, such as functions such as voxel_down_sample(voxel_size=0.01).remove_radius_outliers(10, 0.05)[0]. After the above operation is completed, I use the to_legacy() function to move it to the CPU, and I will perform some calculations on the CPU in the future. I will use loops to perform the above operations on each different point cloud data, and as the number of operations increases or loops, there will be problems with cuda runtime error : out of memory 。

Steps to reproduce the bug

item = self.data_info[index]
anno_path = item['anno']
lidar_path = item['color_bin']
adrn = Path(anno_path).stem.replace('lidar.concat.parking', 'bev')

annotation = json.load(open(anno_path, 'r'))
pcd = read_color_bin(fp=lidar_path,     # 已经是多帧结果
                     lidar_type='color', 
                     crop_range=[-10.0, -10.0, -0.5, 10.0, 10.0, 2.0], 
                     device=self.device)

pcd = pcd.voxel_down_sample(voxel_size=0.01).remove_radius_outliers(10, 0.05)[0] # 下采样 + 去噪
data = self.obstacle_height_extractor(pcd, annotation['points'], annotation['category'])

def read_color_bin(fp, lidar_type='color_bin', crop_range=[-10.0, -10.0, -0.1, 10.0, 10.0, 2.0], device='cpu'):
    if re.search('color', lidar_type):  # 有时候'color'表示存储的不一定是着色点云
        col_num = 6
    elif re.search('slam', lidar_type):
        col_num = 5

    if '5' in Path(fp).stem:
        col_num = 5
    elif '6' in Path(fp).stem:
        col_num = 6

    color_bin = np.fromfile(fp, dtype=np.float32).reshape(-1, col_num)
    #* open3d的点云crop似乎更快一些
    # color_bin = remove_points_according_range(color_bin, 
    #                                           remove_range=crop_range, #* 范围截断 
    #                                           save_inside=True)
    if re.search('cpu', device):
        pcd = o3d.geometry.PointCloud()
        pcd.points = o3d.utility.Vector3dVector(color_bin[:, :3])
        # 给颜色赋值
        if col_num == 6:
            pcd.colors = o3d.utility.Vector3dVector(color_bin[:, 3:] / 255.0)
        elif col_num == 5:
            intensity = np.expand_dims(color_bin[:, 3], axis=-1)
            intensity = np.repeat(intensity, repeats=3, axis=-1) 
            pcd.colors = o3d.utility.Vector3dVector(intensity / 255.0)
        #* 范围截断
        pcd = pcd.crop(o3d.geometry.AxisAlignedBoundingBox(
                            min_bound=np.array(crop_range[:3]).reshape(3,1),
                            max_bound=np.array(crop_range[3:]).reshape(3,1)
                    ))

    elif re.search('cuda', device):
        pcd = o3d.t.geometry.PointCloud(device=o3c.Device(device)) # 有.cpu(), .cuda()的成员函数
        pcd.point.positions = o3c.Tensor(color_bin[:, :3], dtype=o3c.float32, device=o3c.Device(device))
        # 给颜色赋值
        if col_num == 6:
            pcd.point.colors = o3c.Tensor(color_bin[:, 3:] / 255.0, dtype=o3c.float32, device=o3c.Device(device))
        elif col_num == 5:
            intensity = np.expand_dims(color_bin[:, 3], axis=-1)
            intensity = np.repeat(intensity, repeats=3, axis=-1) 
            pcd.point.colors =  o3c.Tensor(intensity / 255.0, dtype=o3c.float32, device=o3c.Device(device))
        #* 范围截断
        pcd = pcd.crop(o3d.t.geometry.AxisAlignedBoundingBox(
            o3c.Tensor(crop_range[:3], device=o3c.Device(device), dtype=o3c.float32),
            o3c.Tensor(crop_range[3:], device=o3c.Device(device), dtype=o3c.float32)
        ))

    return pcd

Error message

RuntimeError: [Open3D Error] (void open3d::core::__OPEN3D_CUDA_CHECK(cudaError_t, const char*, int)) /root/Open3D/cpp/open3d/core/CUDAUtils.cpp:289: /root/Open3D/cpp/open3d/core/MemoryManagerCUDA.cpp:24 CUDA run time error: out of memory

Expected behavior

we wish problem mentioned above is fixed, we don't meet cuda out of memory

Open3D, Python and System information

- Operating system: Ubuntu 20.04
- Python version: Python 3.8 
- Open3D version: 0.16.0
- System architecture: x86 
- Is this a remote workstation?: no
- How did you install Open3D?: pip 
- Compiler version (if built from source): not built from source

Additional information

No response

feendres commented 1 year ago

Hi,

i had a similar problem when using open3d:t:geometry:pointcloud functions on the GPU in C++ and looping over some dataset. I watched the GPU Memory and it slowly got more and more used. My solution was to add open3d::core::MemoryManagerCached::ReleaseCache(device); at the end of the loop to free up the GPU Memory (see) . I didn't find any python binding to that C++ class/function, so i am not sure it helps, but maybe i gives some additional information.

Kind regards!

wj-data commented 9 months ago

open3d::core::MemoryManagerCached::ReleaseCache(device) can not find in python, but o3c.cuda.release_cache() can release cuda memory. I test it, and solve my problem.

xiebinghua commented 9 months ago

open3d::core::MemoryManagerCached::ReleaseCache(device) can not find in python, but o3c.cuda.release_cache() can release cuda memory. I test it, and solve my problem.

It perfectly solve my problem, thanks.

isl-org / Open3D