Details of Tips for Reproducing Study Results

[eunzihong]

Dear the authors of SECAD-Net,

Thank you for your great work and clean code!

I am currently working on a project that aligns closely with your research, and I believe that reproducing your study's results could greatly enhance our understanding of your important findings. However, I encountered a significant discrepancy: my chamfer distance was 21.75, far from the 0.330 reported in your paper (in Table 1). Given this discrepancy, I would greatly appreciate any advice or insights you could share to guide our reproduction efforts more accurately.

I've followed the paper's guidelines closely and these are the steps I've gone through.

Note that the paper said 50 CAD shapes are used for the quantitative evaluation but the list of the 50 CAD models is not been published, thus I used the first 500 shapes from the full ABC test set (1000 shapes). (5.1 Setup - Dataset preparation)

1. Fine-tune the network for each test shape for 300 epochs using the given fine-tuning script (taking 3 mins per shape) (4.4 Implementation Details)
2. Extract the mesh using Marching cubes with 256 resolution using the given test script (5.2 Comparison on CAD Reconstruction)
3. Sample 8,192 points on the surface. (5.1 Setup - Dataset preparation)
4. Meausre two-way chamfer distance between the point cloud from the GT mesh and the reconstructed mesh. (Supplementary Materials, 1. Details of Evaluation Metrics) ※ As the evaluation code is not provided, I implemented it by myself, and the code is attached on the comment below.
5. Multiply 1000 to the chamfer distance output number. (Supplementary Materials, 1. Details of Evaluation Metrics).

However, the chamfer distance number I got is 21.75 which is far different from 0.330, the number reported in Table 1. In this context, I would be very grateful if you could provide some guidance or additional information that could assist in accurately replicating your work.

In particular, I'm seeking:

• The list of 50 CAD models used for quantitative evaluation in both the ABC and Fusion360 datasets.
• The evaluation code for measuring chamfer distance, edge chamfer distance, and normal consistency.
• The preprocessed Fusion360 dataset and the list of the CAD IDs, as only the ABC dataset's data are currently available.

Thank you very much for considering my request. I look forward to your response.

Best, Eunji

[eunzihong]

Here is the code I implemented for the evaluation above. (only chamfer distance)

import open3d as o3d
import numpy as np
import os
import point_cloud_utils as pcu
from glob import glob
from tqdm import tqdm
import torch

if __name__ == "__main__":
    # load meshes
    pred_dir = './exp_log/ABC/Reconstructions/MC/*.obj'
    gt_dir = './data/meshes'

    pred_mesh_paths = glob(pred_dir)
    pred_mesh_paths.sort()

    mean_chamfer_dist = 0

    batch = pred_mesh_paths

    for pred_mesh_path in tqdm(batch):
        name = os.path.basename(pred_mesh_path).split(".")[0]
        gt_mesh_path = os.path.join(gt_dir, name + ".off")

        assert os.path.exists(pred_mesh_path)
        assert os.path.exists(gt_mesh_path)

        pred_mesh = o3d.io.read_triangle_mesh(pred_mesh_path)
        gt_mesh = o3d.io.read_triangle_mesh(gt_mesh_path)

        pred_pc = pred_mesh.sample_points_uniformly(number_of_points=8192)
        pred_pc = np.asarray(pred_pc.points)
        gt_pc = gt_mesh.sample_points_uniformly(number_of_points=8192)
        gt_pc = np.asarray(gt_pc.points)

        chamfer_dist = pcu.chamfer_distance(pred_pc, gt_pc)
        mean_chamfer_dist += chamfer_dist

    mean_chamfer_dist /= len(batch)
    print(len(batch), mean_chamfer_dist, mean_chamfer_dist*1000)

[BunnySoCrazy]

Hello Eunji,

I will provide the content you've requested, though it will take some time.

The logic of your code seems correct, even though I haven't run it yet.

Regarding the poor numerical results you're currently getting, my guess is that there is a mismatch between the mesh files in the meshes directory and the shapes in the hdf5 file provided by CAPRI-Net. In other words, the pred_mesh and gt_mesh in your code do not correspond to the same shape. You can verify my guess by visualizing them. I am considering fixing this issue and providing the correct data.

My suggestion is to obtain the gt_pc from the hdf5 file instead of sampling from the mesh under the meshes folder. Alternatively, you could wait for me to release my evaluation code and more information.

As a final note, as mentioned in the README, aside from posting issues, the most efficient way to communicate with me is by sending an email to pli142857@gmail.com. Contacting others may not help in resolving the matter.

Best regards, Pu

[eunzihong]

Hi Pu,

Thank you for your swift reply! And I'm sorry that I couldn't check that the README file was updated 😅.

I updated my evaluation code by loading the GT point cloud from a file named 'points2mesh.hdf5' as follows

import open3d as o3d
import numpy as np
import os
import point_cloud_utils as pcu
from glob import glob
from tqdm import tqdm
import torch
import h5py

if __name__ == "__main__":
    # load meshes
    pred_dir = './exp_log/ABC/Reconstructions/MC/*.obj'

    with h5py.File('./data/points2mesh.hdf5', 'r') as f:
        gt_points = f['points'][:]

    gt_names = np.load('./data/test_names.npz')['test_names']

    gt_points_dict = {}
    for i, gt_name in enumerate(gt_names):
        gt_points_dict[gt_name] = gt_points[i]

    pred_mesh_paths = glob(pred_dir)
    pred_mesh_paths.sort()

    mean_chamfer_dist = 0
    batch = pred_mesh_paths

    for pred_mesh_path in tqdm(batch):
        name = os.path.basename(pred_mesh_path).split(".")[0]
        gt_pc = gt_points_dict[name][:, :3]

        assert os.path.exists(pred_mesh_path)

        pred_mesh = o3d.io.read_triangle_mesh(pred_mesh_path)

        pred_pc = pred_mesh.sample_points_uniformly(number_of_points=8192)
        pred_pc = np.asarray(pred_pc.points).astype(np.float32)
        gt_pc = np.asarray(gt_pc).astype(np.float32)

        chamfer_dist = pcu.chamfer_distance(pred_pc, gt_pc)
        mean_chamfer_dist += chamfer_dist

    mean_chamfer_dist /= len(batch)
    print(len(batch), mean_chamfer_dist, mean_chamfer_dist*1000)

However, the chamfer distance output is still about 22.xxx like the previous result using the point clouds sampling from the GT meshes. Could you please confirm if it is appropriate to load the files named 'points2mesh.hdf5' and 'test_names.npz' for this process?

I have taken the liberty of visualizing both the ground truth (GT) meshes located in the ‘meshes’ directory provided and the predicted mesh outputs I have obtained. I have compiled a gallery accessible via the following link: https://drive.google.com/drive/folders/15koNGHhfTnoYRJzPV8easUZmuly5IAJa?usp=sharing, which includes images of the GT meshes in blue and the predicted meshes in orange. To my observation, it appears that each predicted mesh closely resembles its corresponding GT mesh in shape. I would greatly appreciate your insights or feedback on this comparison.

Thank you once again for your fascinating work and the thoughtful response you provided. I would be immensely appreciative if you could share the evaluation code and dataset at your earliest convenience.

Warm regards, Eunji

[BunnySoCrazy]

Hello Eunji,

Firstly, selecting 'points2mesh.hdf5' and 'test_names.npz' is appropriate.

Secondly, based on the visualization results you provided, the correspondence between pred_mesh and gt_mesh is correct. However, some of the pred_mesh seem to be worse than what I have obtained. Even so, the evaluation results should be better than they currently are.

I am a bit confused now. I will start to investigate this issue more deeply, and I will update here as soon as I find anything.

Best regards, Pu

[eunzihong]

Hi, Pu.

Thank you for your care. We're looking forward to your answer.

Best, Eunji

[BunnySoCrazy]

Hello Eunji,

I've noticed that the implementation of the Chamfer distance in point_cloud_utils is using the mean of Euclidean distances, while our implementation uses the mean of squared Euclidean distances.

Therefore, to align your results with those in the paper, you should square the distance first, and then multiply by 1000.

I hope this helps.

Best regards, Pu

[eunzihong]

It's right! I changed the chamfer distance function to pytorch3d.loss.chamfer_distance and the scale seems now okay.

Thank you for your answer! It was really helpful.

Best, Eunji

[eunzihong]

Hi Pu,

I wanted to follow up on my previous inquiry regarding the availability of the dataset associated with your project.

I understand that you might be busy with various commitments, and I appreciate the effort and time you put into maintaining this project. If there are any updates or alternatives you could share, I would be very grateful.

Thank you for your time and consideration. We look forward to hearing from you.

Best, Pu

[BunnySoCrazy]

I'm still working on it. Thanks for waiting!

[BunnySoCrazy]

Hello Eunji,

Apologies for the delay. I've just uploaded the evaluation code and ABC test list here. If you have any questions, please let me know.

Best regards, Pu