Performance differences

[julietaumpierrez]

Hello!

Me and some colleagues came across your work a few months ago and found it very interesting. Since last week we've been running the provided code with the wrapper_75_new checkpoint. We wanted to reproduce your results on the Columbia Dataset. Firstly, we encountered that doing the evaluation with the full dataset (using forged and pristine images) returns a pmAP of 0.45 so we assumed that the reported results is only with forged images. In the latter case we achieved a pmAP of 0.9156 which is a bit different from the 0.94 that is reported on this Github. We are running it on a MacBook M1 pro 2021. Do you know what would be the reason behind this difference in performance?

Another issue is regarding the evaluation of the full dataset in which we found out that image 105 (canonxt_35_sub_03.tif) returns this error:

/Users/julietaumpierrez/Desktop/Exif_as_language/exif-as-language/.venv/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/TensorShape.cpp:3550.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] /Users/julietaumpierrez/Desktop/Exif_as_language/exif-as-language/.venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py:3464: RuntimeWarning: Mean of empty slice. return _methods._mean(a, axis=axis, dtype=dtype, /Users/julietaumpierrez/Desktop/Exif_as_language/exif-as-language/.venv/lib/python3.10/site-packages/numpy/core/_methods.py:184: RuntimeWarning: invalid value encountered in divide ret = um.true_divide( 0%| | 0/363 [00:12<?, ?it/s] Traceback (most recent call last): File "/Users/julietaumpierrez/Desktop/Exif_as_language/exif-as-language/splice_evaluator.py", line 199, in evaluator.evaluate() File "/Users/julietaumpierrez/Desktop/Exif_as_language/exif-as-language/splice_evaluator.py", line 96, in evaluate pred = self.exif_sc.predict(data["img"]) File "/Users/julietaumpierrez/Desktop/Exif_as_language/exif-as-language/eval/image_splice/exif_sc.py", line 192, in predict ms = mean_shift( File "/Users/julietaumpierrez/Desktop/Exif_as_language/exif-as-language/eval/image_splice/exif_sc.py", line 55, in mean_shift nearest_inds = kdt.query_ball_point(point, r=eps_5) File "_ckdtree.pyx", line 948, in scipy.spatial._ckdtree.cKDTree.query_ball_point ValueError: 'x' must be finite, check for nan or inf values

We believe is because the parameter eps_5 is equal to zero. Did you encounter the same problem? And do you have any idea on how to fix it?

Thanks in advance and once again fantastic work!

[hellomuffin]

Thank you for trying our code! Yes for the splice localization task, the dataset is for forged images since its target is for localization. Actually very recently I mainly grabbed the dataset resource from here, in which the authors for ImageForensicsOSN paper gather most of the common-used forensics datasets there and is pretty convenient to use.

For the buggy performance you got, I don't really see it in my machine (linux system). I doubt it may come to half-precision training and over-regularized, is it possible for you to try on linux machine? Or if you can't access to Linux system, did you observe the same problem in here?

[julietaumpierrez]

Thank you for your answer!

With that pth file I still get the same error on image canonxt_35_sub_03.tif which is a pristine image, did you run the code with the full Columbia dataset? I am going to try to ran only the forged images to see if the pmAP performance increases and I am going to try it on a linux machine too.

[julietaumpierrez]

With the new weights i am achieving pmAP of 0.932. However as I said on the previous message the problem with canonxt_35_sub_03.tif persists. I'll let you know if running it on a linux machine fixes anything, if you have any idea of what might be wrong I would appreciate it. Thank you again!