`example.py` throws error : erroneous infinity returned

[aneesh-joshi]

I am trying to run example.py on a video of my own. The only modification made by me is for the path to the folder with the videos : https://github.com/DariusAf/MesoNet/blob/61fe5cde61b2b6f531f55e7007976806fddbbf7a/example.py#L31

I have the same python and keras versions as mentioned in the readme.

The error trace:

Using TensorFlow backend.
2019-05-03 14:32:26.071187: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Found 4 images belonging to 2 classes.
Predicted : [[0.99782073]] 
Real class : [1.]
Dealing with video  fadg0-fram1-roi93.mov
Traceback (most recent call last):
  File "/home/aneesh/Work/MesoNet/example.py", line 31, in <module>
    predictions = compute_accuracy(classifier, '/home/aneesh/Work/data/deepfaketimit/DeepfakeTIMIT/')
  File "/home/aneesh/Work/MesoNet/pipeline.py", line 305, in compute_accuracy
    skipstep = max(floor(face_finder.length / frame_subsample_count), 0)
OverflowError: cannot convert float infinity to integer
Fatal Python error: could not acquire lock for <_io.BufferedReader name=8> at interpreter shutdown, possibly due to daemon threads

Thread 0x00007f08e77fe700 (most recent call first):
  File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/site-packages/imageio_ffmpeg/_parsing.py", line 61 in run
  File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007f0949657740 (most recent call first):
  File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/subprocess.py", line 1557 in _communicate
  File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/subprocess.py", line 863 in communicate
  File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/site-packages/imageio_ffmpeg/_io.py", line 193 in read_frames
  File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/site-packages/imageio/plugins/ffmpeg.py", line 342 in _close
  File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/site-packages/imageio/core/format.py", line 252 in close
  File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/site-packages/imageio/core/format.py", line 241 in __del__

I've tried debugging it. It seems to be stemming from https://github.com/DariusAf/MesoNet/blob/61fe5cde61b2b6f531f55e7007976806fddbbf7a/pipeline.py#L305

where face_finder.length is returning infinity.

I found a potential solution here : https://stackoverflow.com/questions/54778001/how-to-to-tackle-overflowerror-cannot-convert-float-infinity-to-integer

I may have fixed it in my fork : https://github.com/aneesh-joshi/MesoNet/blob/19749db10561901b0622009d5266046bc967bb10/pipeline.py#L25

But on running it with this "fix", I get 100% confidence of fakeness in real and fake videos. I suspect I'm doing something wrong.

Could you help me out @DariusAf

Thanks!

[DariusAf]

Hi,

I haven't seen this error on the length before with ffmpeg, I couldn't tell you what's going on exactly with your video. But judging from the errors it doesn't seem that fixing the frame count is enough.

Reading your fork, it seems you have tried to display the video frames, did it work right ? Is it working with other videos ? (ie. is linked to your configuration of ffmpeg, or to a badly encoded video)

In any case:

• the skipstep parameter is not essential, if you know your video length you could manually set it, let's say to 24 to extract a frame each second ;
• the pipeline code is super messy and definitely not robust to error handling, I've shared it because some people requested the code for the face extraction, but you could find another face extractor and directly read your images using keras' ImageDataGenerator ;
• I have no clue how imageio and ffmpeg work internally, I can only advise you to manually debug the imageio.get_reader and see if you get the frames correctly ;
• a 100% confidence of fakeness does sound like a bug, typically that the read images are all null.

[aneesh-joshi]

@DariusAf I think a possible reason could be normalization.

https://github.com/DariusAf/MesoNet/blob/61fe5cde61b2b6f531f55e7007976806fddbbf7a/example.py#L15

You normalize the images for the DF model but don't do the same the same for the Face2Face model which takes a video folder as the input.

Is this intentional?

(I divided by 255. and the confidence was more sane (between 0.7 and 0.9))

I'm trying to just run some sanity checks: it detects almost all videos as fake. I generated this video from the vidTIMIT dataset which is not fake, but it gets predicted as always fake.

Could you help me in just cross checking it?

[aneesh-joshi]

Reading your fork, it seems you have tried to display the video frames, did it work right ? Is it working with other videos ?

I am able to see the images. I was initially having problems matplotlib can't show float images. It showed them after converting to int.

[aneesh-joshi]

Additionally, I've been using this script for videop prediction: https://github.com/aneesh-joshi/MesoNet/blob/master/video_predictor.py

Not sure if I'm missing anything

[DariusAf]

The only way to be sure is to print your read images and check if the range is between 0 and 1 (that's how the model was trained).

matplotlib can show float images, only if their range is in [0,1], so that probably means that there was indeed a problem of normalisation.

I am surprised this happens, those are the main function of the pipeline code I have used for evaluation, so the normalisation should be ok. I'll take a look at your script and video.

Ultimately, the detection is not absolute, you may have found a video that is a false positive. I must admit that would be a bit disappointing for your first try.

[aneesh-joshi]

The only way to be sure is to print your read images and check if the range is between 0 and 1 (that's how the model was trained).

On printing, I got pixel values like ..., 40. , 80. , ... (i.,e floats of integers) Which probably explains why the displayed image was weird (A white image with some streaks of yellow). May also explain the 100% confidence values.

Ultimately, the detection is not absolute, you may have found a video that is a false positive

I ran it on several(5 to 6) videos from the VidTIMIT and DeepFakeTimit dataset. The results were either always fakes or fake/pristine with a small level of confidence(0.51, 0.49). Again, it might be because of some bug on my side.

I tried varying the window around the face.

https://github.com/aneesh-joshi/MesoNet/blob/83990cc59d86c0aabba044e1a52bc6d783275f92/video_predictor.py#L79

But it didn't help. A higher PADDING made it worse.

I am using the same face extraction as you:

https://github.com/aneesh-joshi/MesoNet/blob/83990cc59d86c0aabba044e1a52bc6d783275f92/video_predictor.py#L71

[abnishstha]

I also faced similar issue as mentioned by @aneesh-joshi

---

from pipeline import compute_accuracy from classifiers import Meso4

classifier = Meso4() classifier.load('weights/Meso4_DF')

predictions = compute_accuracy(classifier, 'test_videos1') for video_name in predictions: print('{} video class prediction :'.format(video_name), predictions[video_name][0])

---

line 305, in compute_accuracy skipstep = max(floor(face_finder.length / frame_subsample_count), 0)

OverflowError: cannot convert float infinity to integer

In [24]: face_finder.length Out[24]: inf

[markzampoglou]

It seems the problem is linked to this bug: https://stackoverflow.com/questions/54778001/how-to-to-tackle-overflowerror-cannot-convert-float-infinity-to-integer

The solution is probably to replace line 21 of pipeline.py: Instead of self.length = self.container.get_length() put self.length = self.container.count_frames()

[aneesh-joshi]

@markzampoglou I've made that change in my fork:

https://github.com/aneesh-joshi/MesoNet/blob/19749db10561901b0622009d5266046bc967bb10/pipeline.py#L25

But the other problem is 100% certainty in fakeness. Something's off and needs to be foxed.

[nviable]

@aneesh-joshi, it's probably better to create a different issue for the 100% fakeness problem the title seems misleading now.

@markzampoglou, thanks for the solution.

The imageio 2.5.0 release notes suggest the exact same thing.

The reader of the ffmpeg plugin now always reports inf as the number of frames. Use reader.count_frames() to get the actual number, or estimate it from the fps and duration in the meta data.

[aneesh-joshi]

@nviable makes sense. If @DariusAf could verify this change, I'll make a PR and close this issue.