search

SunnyHaze / IML-ViT

Official repository of paper “IML-ViT: Benchmarking Image manipulation localization by Vision Transformer”
MIT License
184 stars 23 forks source link

test datasets size #24

Open talha-alam opened 1 month ago

talha-alam commented 1 month ago

Hi,

Thank you for the code and your prompt responses to the issues.

Could you please provide the number of images used for testing from the CASIA V1 dataset? I am currently experiencing difficulties reproducing the results reported in the manuscript, as I am obtaining notably high F1 scores for the CASIA V1 test dataset. Are these F1 scores the pixel-level ones or the general ones mentioned in Table 3 of the paper? Additionally, I’ve noticed that the AUC code has not yet been updated in the repository. Could you please advise on this so that I can align my results with those presented in the manuscript?

Moreover, when changing the norm type, I encounter an error: "Unexpected key(s) in state_dict: 'predict_head.norm.running_mean', 'predict_head.norm.running_var', 'predict_head.norm.num_batches_tracked'." Could you provide guidance on how to resolve this issue?

SunnyHaze commented 1 month ago

Hi, thanks for your attention to our work, and sorry for the delay. Here are our responses:

Could you please provide the number of images used for testing from the CASIA V1 dataset?

For testing the pixel-F1 score, we only applied 920 manipulated images since pixel-F1 for the authentic images is meaningless (see this comment.

I am currently experiencing difficulties reproducing the results reported in the manuscript, as I am obtaining notably high F1 scores for the CASIA V1 test dataset. Are these F1 scores the pixel-level ones or the general ones mentioned in Table 3 of the paper?

Since this work only focuses on localization, only pixel-level metrics are reported. The reason why your results seem very high is that we thoroughly searched optimal settings months after we released the paper and adjusted this part of the settings in our code. Thus, you may experience some performance improvement with the code. We will revise the paper to clarify this gap in the future.

I’ve noticed that the AUC code has not yet been updated in the repository. Could you please advise on this so that I can align my results with those presented in the manuscript?

You may check this issue #8, but I recommend trying out our latest benchmark, IMDL-BenCo, to calculate the metrics for a more robust solution. The documentations are also provided.

Moreover, when changing the norm type, I encounter an error: "Unexpected key(s) in state_dict: 'predict_head.norm.running_mean', 'predict_head.norm.running_var', 'predict_head.norm.num_batches_tracked'." Could you provide guidance on how to resolve this issue?

When you change the type of norm layer, the model structure is also changed. So, you cannot directly load a checkpoint that was trained with a different normalization layer. You must re-train the model with the norm layer you like and load it to the corresponding model.

If you have further questions, please feel free to reach out.