How to infer one folder of large data(eg: one folder of scannet) or new data?

[ThePassedWind]

I would like to infer my own data instead of tesing standard data(eg: scannet...) Could you please provide this script?

[filaPro]

Hi @ThePassedWind , You can try demo/pcd_demo.py script.

[ThePassedWind]

It worked! Thanks! But I'd like to infer and visualize my own 3D-data, how could I preprocess my data?

[ThePassedWind]

Is there any code for process one .ply of data? It seems that all codes are for processing whole dataset(such as scannet...)

[filaPro]

You only need a single array of n x 6 for the model input. 6 is for (x, y, z, r, g, b). I think you can easily make it from your .ply. Just be sure that r, g, b are normalized in the same way as in scannet. Also z axis is orthogonal to the floor, x and y axis are parallel to walls. Also coordinates should be in meters.

[ThePassedWind]

OK! Thanks, I will try as soon as possible.

[ThePassedWind]

Sorry, some issues still occurred. I have process own data by the same way as in scannet. My own input data: cloud.ply Loading ./ply data: pcd = o3d.io.read_point_cloud('./cloud.ply') xyz_load = np.asarray(pcd.points) colors_load = np.asarray(pcd.colors) points2 = np.hstack((xyz_load, colors_load * 255)) points2[:,0] = (points2[:,0] - min(points2[:,0])) / (max(points2[:,0] - min(points2[:,0]))) * 9 points2[:,1] = (points2[:,1] - min(points2[:,1])) / (max(points2[:,1] - min(points2[:,1]))) * 9 points2[:,2] = (points2[:,2] - min(points2[:,2])) / (max(points2[:,2] - min(points2[:,2]))) * 3 Saving .bin data: points2.tofile(osp.join('/mnt/data1/mmdetection3d/data/scannet/','cloud.bin')) Therefore, my data has a similar distribution with scannet dataset. However, I use python demo/pcd_demo.py /mnt/data1/mmdetection3d/data/scannet/cloud.bin configs/votenet/votenet_8x8_scannet-3d-18class.py checkpoints/votenet_8x8_scannet-3d-18class_20210823_234503-cf8134fa.pth --out-dir demo/scene0000_00/ --show to infer own data, with the inference output 'None'. Could you please give me a hand to explain the reason?

PS: I have tested python demo/pcd_demo.py /mnt/data1/mmdetection3d/data/scannet/scene0000_00.bin configs/votenet/votenet_8x8_scannet-3d-18class.py checkpoints/votenet_8x8_scannet-3d-18class_20210823_234503-cf8134fa.pth --out-dir demo/scene0000_00/ --score-thr 0.7 --show, which is correct to visualize.

[filaPro]

As I remember the model inputs are normalized from 0 to 1, not to 255. Also are you sure in (max(points2[:,0] - min(points2[:,0]))) * 9? Cooridnates should be in meters.

[ThePassedWind]

I have just solved this problem which is about float32 and float64. I made a mistake to create a float64 data but this model input needs float32 data. Thanks for your sincere help!

[ThePassedWind]

I've succeeded in running this command to infer 3D detection results of my own data, eg: tr3d and votenet model:

1. python demo/pcd_demo.py /mnt/data1/mmdetection3d/data/scannet/cloud_2hu.bin \ configs/tr3d/tr3d_scannet-3d-18class.py \ checkpoints/tr3d_scannet.pth \ --out-dir demo/scene0000_00/ --show --score-thr 0.6
2. python demo/pcd_demo.py /mnt/data1/mmdetection3d/data/scannet/cloud_2hu.bin configs/votenet/votenet_8x8_scannet-3d-18class.py checkpoints/votenet_8x8_scannet-3d-18class_20210823_234503-cf8134fa.pth \ --out-dir demo/scene0000_00/ --show --score-thr 0.5 However, I was likely to face some problems when running this command by using fcaf3d model, due to the None output: python demo/pcd_demo.py /mnt/data1/mmdetection3d/data/scannet/cloud_2hu.bin \ configs/fcaf3d/fcaf3d_scannet-3d-18class.py \ checkpoints/fcaf3d_8x2_scannet-3d-18class_20220805_084956.pth \ --out-dir demo/scene0000_00/ --show --score-thr 0.5

_The output is None as follow: load checkpoint from local path: checkpoints/fcaf3d_8x2_scannet-3d-18class_20220805_084956.pth The model and loaded state dict do not match exactly unexpected key in source state_dict: head.conv_center.kernel, head.conv_reg.kernel, head.conv_cls.kernel, head.conv_cls.bias missing keys in source state_dict: head.center_conv.kernel, head.reg_conv.kernel, head.cls_conv.kernel, head.cls_conv.bias Traceback (most recent call last): File "demo/pcd_demo.py", line 44, in main() File "demo/pcd_demo.py", line 33, in main show_result_meshlab( File "/mnt/data1/tr3d/mmdet3d/apis/inference.py", line 527, in show_result_meshlab file_name = show_det_result_meshlab(data, result, out_dir, score_thr, File "/mnt/data1/tr3d/mmdet3d/apis/inference.py", line 370, in show_det_result_meshlab show_result( File "/mnt/data1/tr3d/mmdet3d/core/visualizer/show_result.py", line 129, in show_result 0, 255, size=(predlabels.max() + 1, 3)) / 256 **RuntimeError: max(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.**

Could you please help me find the reason why there is nothing by fcaf3d model when inferring the same data as the former model(tr3d and votenet model).

[filaPro]

Is it possible that FCAF3D simply doesn't find any objects in this scene with score_thr=0.5? Can you try with lower threshold?

[filaPro]

Oh I see, the problem is The model and loaded state dict do not match exactly. You need to rename the keys in the checkpoint or in the model to make them similar. Or run the model in latest open-mmlab/mmdetection3d codebase (not samsunglabs/tr3d).

[ThePassedWind]

Ohhh, I've tried to run the model in lastest open-mmlab/mmdetection3d, but it failed. It was so strange...

[ThePassedWind]

It was still The model and loaded state dict do not match exactly

[filaPro]

Please use something like this. May be modify a little until all The model and loaded state dict do not match exactly are fixed.

import torch
from argparse import ArgumentParser

mapping = {
    'head.conv_cls': 'head.cls_conv',
    'head.conv_reg': 'head.bbox_conv',
    'head.conv_center': 'head.center_conv'
}

if __name__ == '__main__':
    parser = ArgumentParser()
    parser.add_argument('--in-path', type=str)
    parser.add_argument('--out-path', type=str)
    args = parser.parse_args()

    checkpoint = torch.load(args.in_path)
    meta = checkpoint['meta']
    state_dict = checkpoint['state_dict']
    new_dict = dict()

    for state_key, state_value in state_dict.items():
        replace_flag = False
        for key, value in mapping.items():
            if key in state_key:
                new_key = state_key.replace(key, value)
                print(f'replace {state_key} to {new_key}')
                new_dict[new_key] = state_value
                replace_flag = True
                break
        if not replace_flag:
            new_dict[state_key] = state_value

    torch.save({'state_dict': new_dict, 'meta': meta}, args.out_path)