Bug: 'builtin_function_or_method' object is not iterable. && AUC on CelebDF is far away from paper's.

[zkxcongming]

I follow your instruction to prepare CelebDF test dataset, trying to validate lipforensics_ff.pth on CelebDF. However I came across:

Traceback (most recent call last):
  File "evaluate.py", line 179, in <module>
    main()
  File "evaluate.py", line 174, in main
    auc = validate_video_level(model, loader, args)
  File "evaluate.py", line 109, in validate_video_level
    for data in tqdm(loader):
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/users/kaixuanzhang/repos/LipForensics/data/dataset_clips.py", line 169, in __getitem__
    sample = self.transform(sample)
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 70, in __call__
    img = t(img)
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 247, in __call__
    return F.center_crop(img, self.size)
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 382, in center_crop
    image_width, image_height = img.size
TypeError: 'builtin_function_or_method' object is not iterable

I noticed that the transform input is np.ndarray but the PIL Image is required, so I substitute the CenterCrop with VideoCenterCrop:

class VideoCenterCrop:
    def __init__(self, outputsize):
        self.output_size = outputsize

    def __call__(self, img):
        """
        param img (numpy.ndarray): THWC 
        """
        if isinstance(self.output_size, numbers.Number):
            output_size = (int(self.output_size), int(self.output_size))
        #print("img shape", img.shape)
        image_width, image_height = img.shape[2], img.shape[1]
        crop_height, crop_width = self.output_size
        crop_top = int(round((image_height - crop_height) / 2.))
        crop_left = int(round((image_width - crop_width) / 2.))
        return img[:, crop_top:crop_top + crop_height, crop_left:crop_left + crop_width, :]

Then run

python evaluate.py --dataset CelebDF --weights_forgery ./models/weights/lipforensics_ff.pth

It works, but I got only AUC of 57.7% on CelebDF. I am wondering which step did I get wrong. I really appreciate it if you could give me some advice. Thank you!

[ahaliassos]

In the CelebDFClips class' __getitem__ method, the sample is cast to torch.Tensor type (line 165 in data/dataset_clips.py) before the transform is applied, and torchvision.transforms.CenterCrop can handle torch.Tensor inputs. Perhaps, the torchvision library version that you have installed is older? You can install using the requirements.txt file to make sure.

Also, have you preprocessed the mouth crops according to the instructions? Do your mouth crops look similar to the examples in the examples directory?

[zkxcongming]

In the CelebDFClips class' __getitem__ method, the sample is cast to torch.Tensor type (line 165 in data/dataset_clips.py) before the transform is applied, and torchvision.transforms.CenterCrop can handle torch.Tensor inputs. Perhaps, the torchvision library version that you have installed is older? You can install using the requirements.txt file to make sure.

Also, have you preprocessed the mouth crops according to the instructions? Do your mouth crops look similar to the examples in the examples directory?

I updated torchvision and everything works fine now, and got 80.8% on CelebDF test set. It was preprocessing problem. I got landmarks on the cropped faces so it looks different from your examples. Problem solved. Thank you!