search

marrlab / SHAPR_torch

SHAPR: Code for "Capturing Shape Information with Multi-Scale Topological Loss Terms for 3D Reconstruction"
https://shapr.topology.rocks
BSD 3-Clause "New" or "Revised" License
39 stars 10 forks source link

How to reproduce results? #5

Closed IlyaTrofimov closed 9 months ago

IlyaTrofimov commented 10 months ago

Hello!

Dear colleagues, I'm trying to reproduce results from your paper. I have downloaded data and executed scripts, assuming that the first one is SHAPR and the second one is SHAPR+topological loss:

python run_train_script.py -p config/red-blood-cell.json python run_train_script.py -p config/red-blood-cell-2D.json

Training has finished, but numbers on a test set which I see in WanDB are different from numbers in your paper (https://arxiv.org/pdf/2203.01703.pdf, Table 1). In fact the list of metrics in WanDB (IoU_Error, dice_error, volume_error) are different from the errors presented in the paper (1 − IoU, Volume, Surface area, Surface roughness). The script "shapr/scripts/evaluation.py" only outputs figures.

Can you please give a hint how to reproduce results?

Pseudomanifold commented 10 months ago

Hi!

The script needs to be modified to result in tabular output. The lead author @DLMLDW also used a Jupyter notebook with which to tabulate the results—maybe you could share this notebook, Dominik? Otherwise, evaluation.py produces a data frame that can be stored; you have to change the call to make_df accordingly. Hope that helps!

IlyaTrofimov commented 9 months ago

I have modified evaluation.py to calculate average of 1 − IoU, Volume, Surface area, Surface roughness. Now results match Table 1 in your paper. Thanks.

Pseudomanifold commented 9 months ago

Thanks so much! If you want, we'd be happy to accept this as a pull request!