How to use the benchmark?

[raphaelsulzer]

Hi,

thank you for putting this valuable dataset together.

How can I use the benchmark, i.e. evaluate my own reconstructions?

Let's say I have your entry level dataset pointclouds and ground truth wireframes, and my own reconstructed wireframes. How do I evaluate my wireframes for the whole dataset on all evaluation metrics? Is there a high-level function for this?

Kind regards

[shangfenghuang]

You can submit your results to https://huggingface.co/spaces/Building3D/USM3D.

The submission systems of Building3D website will be released in the end of this month.

You can use the building3D github code to evaluate the training set.

[raphaelsulzer]

Hi, thank you for the reply.

You can use the building3D github code to evaluate the training set.

How? Which function to call?

do I need to organise my data like the example and call APCalculator?

batch = {'batch_size': 1,
         'predicted_corners': np.array(
             [[[1, 2, 3], [7, 8, 9], [4, 5, 1], [7, 8, 9], [5, 3, 2], [1, 2, 4], [2, 5, 7], [1, 1, 1]]]),
         'wf_vertices': np.array([[[2, 3, 4], [5, 6, 4], [6, 7, 8], [-10, -10, -10], [-10, -10, -10]]]),
         'predicted_edges': np.array([[[1, 2], [1, 5], [5, 6], [2, 4]]]),
         'pred_edges_vertices': np.array([[[7, 8, 9], [4, 5, 1]],
                                          [[7, 8, 9], [1, 2, 4]],
                                          [[4, 5, 1], [5, 3, 2]]]),
         'wf_edges': np.array([[[0, 1], [0, 2], [1, 2], [3, 4], [4, 5], [2, 4], [1, 3], [-1, -1], [-1, -1]]]),
         'predicted_score': np.array([[0.8, 0.8, 0.2, 1]]),
         'label_edges_vertices': np.array(...),
         'centroid': np.array([[2, 2, 2]]),
         'max_distance': np.array([[1]])}

APCalculator.compute_metrics(batch)

What is centroid and max_distance?

[shangfenghuang]

yeah. The centroid and max_distance is calculated by dataset/building3d.py. It is used to noramlize the dataset. So when you calculate the average corner offset (ACO), you need to inverse normalize the data to the original coordinates and size.

But if you don't use the provided data processing code, you are also required to use the original coordinates to calculate the ACO.

I suggest that you organise your data like the example and call APCalculate.

By the way, I will upload the ap.ipynb file to introduce how to use the APCalculate tomorrow.