Not able to replicate results on FlyingThings++ noted in the paper

[ayshrv]

Hi,

Great work and thank you for releasing your code and documenting it.

I was trying to replicate numbers on FlyingThings++ shown in this table.

I had setup the conda environment and FlyingThings++ exactly as told in the README. I am using the model checkpoint provided in get_reference_model.sh.

I run the command python test_on_flt.py --modeltype pips and python test_on_flt.py --modeltype raft to evaluate on FlyingThings++.

Here are the results that I get.

PIPS:

1_8_16_pips_flt_17:07:18; step 002542/2542; rtime 0.00; itime 0.09, ate_vis = 10.42, ate_occ = 27.04

RAFT:

1_8_16_raft_flt_18:46:56; step 002542/2542; rtime 0.00; itime 1.03, ate_vis = 15.04, ate_occ = 39 

Do you have any idea why they might be different from the results provided in the paper?

[aharley]

Interesting! Thanks for the heads up. At least you're doing better than the paper, instead of worse!

My first thought is that maybe the data you generated is slightly different, due to some randomness in that process. What I should probably do is put my data on google drive. If you want/need that, let me know. I'll keep thinking, in case this isn't it.

[ayshrv]

Thank you for your response. Yes, that will be great if you could provide the data that you generated. I agree that there could be some randomness in the process.

[ayshrv]

Hey @aharley , please let me know by when you will be able to share the data you generated.

[aharley]

Thanks for the push. Today I'm tarring, downloading, and uploading to Google drive... Should be ready in 24h maybe.

[aharley]

Maybe more time actually. The upload time is reading 55h right now, and that's assuming no interruptions. Anyway, at the end of this we should have a very good gdrive link. I'll let you know @ayshrv!

[aharley]

Update: I think the problem is basically on my side, to do with last-minute updates to the dataset, after putting results into the paper. Re-running things locally I'm actually getting values very close to yours.

I will update the repo soon with new reference results in the README.

If you still want to download my exact flyingthings data you can get it here: https://drive.google.com/drive/folders/1zzWkGGFgJPyHpVaSA19zpYlux1Mf6wGC?usp=sharing

[ayshrv]

Great, thanks for checking and providing me your version of the dataset.

Looks like you are indeed able to get better results than reported in the paper 🙂 .