Reproduce article results

[farhadinima75]

Hi @PatrickTUM. First of all, thank you for the nice repo and data.

I'm trying to reproduce your mentioned results in the article but after several runs from test.py, I couldn't achieve that. The only change that I've made was changing the data directory.

This is the results for SiamUnet_conc after 50 epochs:

pr_c, rec_c, f_meas, pr_nc, rec_nc
[0.5586626676076217, 0.3887575910376428, 0.4584752138873591, 0.9686968866198238, 0.9840231425152476]
{'train_loss': 0.13158418238162994, 'train_accuracy': 97.97806035271284, 'train_nochange_accuracy': 99.0765609741211, 'train_change_accuracy': 46.83369064331055, 'test_loss': 0.32782667875289917, 'test_accuracy': 95.45872523406716, 'test_nochange_accuracy': 98.40231323242188, 'test_change_accuracy': 38.87575912475586}
Elapsed time:
963.6851334571838

The measured value for F is 0.4584752138873591, but the article mentioned it's equal to 0.519. Am I doing something wrong?

[PatrickTUM]

Hi @farhadinima75,

thanks for your kind words! Your observed discrepancy in performance is beyond what one would expect. So I gave it a check and the cause of this should be due to the code‘s default parameters: the patches in our experiments were samples with a stride of 1, as reported in the paper. Please use this smaller stride in place of the larger stride defined here https://github.com/PatrickTUM/multimodalCD_ISPRS21/blob/77d50958fcaf6488a0224375ab60b647bf0c7f4e/code/data_loader.py#L40

These default values come from the original paper's code. However, we experienced better outcomes with more patches sliced. I suppose this is also because of the random augmentation pipeline. You may want to reduce training to around 10-15 epochs (as there‘ll be many more samples now). We didn’t optimize this much further, but considering additional protocols may be beneficial. Hoping this helps!

[farhadinima75]

I'll inform you of the new results. thanks again for your time.

[Jason-Study]

Dear, @farhadinima75,

I am a freshman and I am trying to run this code, but there seems lack of the files named "SiamUnet_diff_multi" and "FresUNet_multi". Could you provide me the relevant files or methods to find these files?

elif TYPE == 4:
        if NET == 'Unet': net, net_name = Unet(2*13+2*2, 2), 'FC-EF-multi' # same architecture as the other network
        if NET == 'SiamUnet_conc-simple': net, net_name = SiamUnet_conc(13+2, 2), 'FC-Siam-conc-simple'
        if NET == 'SiamUnet_conc': net, net_name = SiamUnet_conc_multi((13, 2), 2), 'FC-Siam-conc-complex'
        if NET == 'SiamUnet_diff': net, net_name = SiamUnet_diff_multi(13, 2, 2), 'FC-Siam-diff'
        if NET == 'FresUNet': net, net_name = FresUNet_multi(2 * 13 + 2*13, 2), 'FresUNet'

Thank you very much!

[farhadinima75]

Hi @Jason-Study As far as I know, the paper didn't use any of these methods. But you can find SiamUnet_diff (with setting 13+2 as input channels) in this file: https://github.com/likyoo/Siam-NestedUNet/blob/master/models/siamunet_dif.py Let me know if you need further help.

[Jason-Study]

Hi @Jason-Study As far as I know, the paper didn't use any of these methods. But you can find SiamUnet_diff (with setting 13+2 as input channels) in this file: https://github.com/likyoo/Siam-NestedUNet/blob/master/models/siamunet_dif.py Let me know if you need further help.

Hi @farhadinima75,

Thanks for your reply.

The files named SiamUnet_diff_multi and FresUNet_multi existing in the file of test.py (Line 414 and 415). If using the multimodal data(4-All bands Sentinel-2 & Sentinel-1), need to run these function.

[farhadinima75]

Yeah, I've seen them and I used 'SiamUnet_conc-simple' instead. This mode is the mode that is used in the paper. If you wanna use FresUNet_multi, @PatrickTUM can answer better.

[Jason-Study]

OK. Thank you very much!

[PatrickTUM]

Hi all, please see #4 for this matter.