Unable to reproduce the results

[guyleaf]

Hi I try to reproduce the results of PlanIGN and Google Mismatch. The results are much worse than the results in the paper.

I follow the implementation details in the supplementary material.

Thanks.

PlanIGN

python train.py --dataroot ~/data/StegoGAN/PlanIGN \
                --name plan_ign_200_epochs \
                --model stego_gan \
                --gpu_ids 0 \
                --lambda_reg 0.25 \
                --lambda_consistency 1 \
                --resnet_layer 1 \
                --batch_size 1  \
                --no_flip \
                --n_epochs 200 \
                --display_env stegogan

Result (epoch which has the lowest FID)

{
   'FID': 74.82682986695903,
   'KID': 4.056192402742358,
   'RMSE': 23.52806436030369,
   'acc2': 58.65092298719618,
   'acc5': 70.50994194878473,
   'mIOU(mask)': 31.18319348202519,
   'method': 'StegoGAN',
   'precision(mask)': 32.174854834601355,
   'recall(mask)': 80.05126802578572
}

Google Mismatch

python train.py --dataroot ~/data/StegoGAN/Google_mismatch/0.65 \
                --name google_stego_0.65_200_epochs \
                --model stego_gan \
                --gpu_ids 0 \
                --lambda_reg 0.3 \
                --lambda_consistency 1 \
                --resnet_layer 8 \
                --batch_size 1  \
                --no_flip \
                --n_epochs 200

Result (epoch which has the lowest FID)

{
   'FID': 174.4486028830649,
   'KID': 21.07417390784439,
   'RMSE': 25.83185946549272,
   'acc10': 58.46950981293064,
   'acc5': 31.75560116502384,
   'iFPR': 0.0,
   'method': 'StegoGAN',
   'pFPR100': 0.0
}

[sian-wusidi]

sorry to not have replied earlier. Can you try the following configurations? python train.py --dataroot ~/data/StegoGAN/PlanIGN \ --name plan_ign_200_epochs \ --model stego_gan \ --gpu_ids 0 \ --lambda_reg 0.25 \ --lambda_consistency 1 \ --load_size 256 \ --resnet_layer 1 \ --batch_size 1 \ --fusionblock\ --n_epochs 200 \ --display_env stegogan

python train.py --dataroot ~/data/StegoGAN/Google_mismatch/0.65 \ --name google_stego_0.65_200_epochs \ --model stego_gan \ --gpu_ids 0 \ --lambda_reg 0.3 \ --lambda_consistency 1 \ --resnet_layer 8 \ --batch_size 1 \ --fusionblock \ --n_epochs 200

[guyleaf]

Hi, thanks for reply. I have tried these configs. The result of PlanIGN is worse than the previous one. BTW, it seems most of masks are zero. The result of Google mismatch is much better.

But I still cannot reproduce similar results in the paper.

PlanIGN

python train.py --dataroot ~/data/StegoGAN/PlanIGN \
                --name plan_ign_200_epochs_fusion \
                --model stego_gan \
                --gpu_ids 0 \
                --lambda_reg 0.25 \
                --lambda_consistency 1 \
                --load_size 256 \
                --resnet_layer 1 \
                --batch_size 1 \
                --fusionblock \
                --n_epochs 200 \
                --display_env plan_ign_stegogan_200_epochs_fusion

Result (epoch which has the lowest FID)

{
   'FID': 99.38335153936549,
   'KID': 5.844608458227034,
   'RMSE': 29.634924396781187,
   'acc2': 30.998048570421005,
   'acc5': 57.876825968424484,
   'mIOU(mask)': 0.16216485954573465,
   'method': 'StegoGAN',
   'precision(mask)': 0.27900964732867667,
   'recall(mask)': 0.21460123200571884
}

Google Mismatch

python train.py --dataroot ~/data/StegoGAN/Google_mismatch/0.65 \
                --name google_stego_0.65_200_epochs_fusion \
                --model stego_gan \
                --gpu_ids 0 \
                --lambda_reg 0.3 \
                --lambda_consistency 1 \
                --resnet_layer 8 \
                --batch_size 1 \
                --fusionblock \
                --n_epochs 200 \
                --display_env google_stego_0.65_stegogan_200_epochs_fusion

Result (epoch which has the lowest FID)

{
   'FID': 162.1324984722084,
   'KID': 19.607546675701528,
   'RMSE': 25.56585017451607,
   'acc10': 60.07601561684386,
   'acc5': 32.06470379054148,
   'iFPR': 0.0,
   'method': 'StegoGAN',
   'pFPR100': 0.0
}

[sian-wusidi]

For PlanIGN, can you reduce the --lambda_reg to 0.2? --n_epochs should be 100 instead of 200, sorry for the error in previous configurations.