search

chinmay5 / vesselformer

11 stars 2 forks source link

Very low score for test set #4

Open RomStriker opened 7 months ago

RomStriker commented 7 months ago

Hi,

I have trained vesselformer on the synthetic dataset using the default hyperparameters. During training I get the following best validations scores:

Validation Max Det 40
mAP_IoU: 0.2965481494636376
AP_IoU_5: 0.6506306071682731
AP_IoU_75: 0.23343406057003702
mAR_IoU: 0.3899653763743117
AR_IoU_5: 0.7214829325675964
AR_IoU_75: 0.3675396367907524

but when running the vesselformer_inference.py script for testing using this trained model, I get very low scores, and jugding from the plot the predictions look almost random. Test stats are following.

Betti-error: [46.5578]
Mean SMD: tensor(0.0302)
Mean L1 radius regression: tensor(10.3417)
Test Max Det 200
Nodes
mAP_IoU_1 = 0.0026091254648875264
AP_IoU_5_1 = 0.005493092728723394
AP_IoU_75_1 = 0.00022674023662463274
mAR_IoU_1 = 0.006909224083384248
AR_IoU_5_1 = 0.023025307895504988
AR_IoU_75_1 = 0.0029932900264156483
Edges
mAP_IoU_2 = 4.3443222095069505e-06
AP_IoU_5_2 = 2.428608264147055e-05
AP_IoU_75_2 = 2.646739802111198e-07
mAR_IoU_2 = 0.0013259698470660544
AR_IoU_5_2 = 0.007606023355869699
AR_IoU_75_2 = 6.327054484977765e-05

Do you know what could be causing this issue?

Coincidance-Levi commented 7 months ago

I‘ve met similar situation. Very puzzled with the result. Did the problem come from the possible wrong correspondence between the image and node & edge? Have you ever visualized the data?