problems of extract_faces.py

hey thanks for your work but I have some question when I try to run the extract_faces.py I seems to see the faces have been extracted,but the list have nothing

Loading video DataFrame Loading face extractor <function main.. at 0x7fd95923fea0> Extracting faces: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:00<00:00, 246.73it/s] path height width frames class label source name original nfaces 0 Celeb-synthesis/id3_id1_0001.mp4 500 528 380 Celeb-synthesis True id3_id1_0001.mp4 id3_id1_0001 -1 0 1 Celeb-synthesis/id9_id1_0008.mp4 500 944 461 Celeb-synthesis True id9_id1_0008.mp4 id9_id1_0008 -1 0 2 Celeb-synthesis/id0_id9_0006.mp4 500 944 534 Celeb-synthesis True id0_id9_0006.mp4 id0_id9_0006 -1 0 3 Celeb-synthesis/id9_id4_0007.mp4 500 944 456 Celeb-synthesis True id9_id4_0007.mp4 id9_id4_0007 -1 0 4 Celeb-synthesis/id7_id12_0005.mp4 500 944 161 Celeb-synthesis True id7_id12_0005.mp4 id7_id12_0005 -1 0 ... ... ... ... ... ... ... ... ... ... ... 1198 YouTube-real/00098.mp4 500 892 453 YouTube-real False 00098.mp4 00098 -1 0 1199 YouTube-real/00009.mp4 500 892 575 YouTube-real False 00009.mp4 00009 -1 0 1200 YouTube-real/00207.mp4 500 892 126 YouTube-real False 00207.mp4 00207 -1 0 1201 YouTube-real/00174.mp4 500 892 465 YouTube-real False 00174.mp4 00174 -1 0 1202 YouTube-real/00040.mp4 500 892 381 YouTube-real False 00040.mp4 00040 -1 0

[1203 rows x 10 columns] Collecting faces results: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1203/1203 [00:00<00:00, 12715.78it/s] Saving videos DataFrame to data/celebdf_videos.pkl Saving faces DataFrame to faces/celeb/output_from_video_0_to_video_0.pkl Traceback (most recent call last): File "/home/ubuntu/project/icpr2020dfdc-master/extract_faces.py", line 317, in main() File "/home/ubuntu/project/icpr2020dfdc-master/extract_faces.py", line 164, in main df_faces = pd.concat(faces_dataset, axis=0, ) File "/home/ubuntu/.conda/envs/icpr2020/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 284, in concat sort=sort, File "/home/ubuntu/.conda/envs/icpr2020/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 331, in init raise ValueError("No objects to concatenate") ValueError: No objects to concatenate

Hi @IItaly, It's hard to tell without seeing the other modification you did on the code, but the problem you are reporting seems the same as #11.

I can suggest you to check after if video_face_checkpoint_path.exists():. Probably you fail to enter that if due to a missing pkl. Try to investigate why that file do not exist.

Hi @IItaly, It's hard to tell without seeing the other modification you did on the code, but the problem you are reporting seems the same as #11.

I can suggest you to check after if video_face_checkpoint_path.exists():. Probably you fail to enter that if due to a missing pkl. Try to investigate why that file do not exist.

tosave_list = list(p.map(partial(process_video, source_dir=source_dir, facedestination_dir=facedestination_dir, checkpoint_folder=checkpoint_folder, face_size=face_size, face_extractor=face_extractor, lazycheck=lazycheck, deepcheck=deepcheck, ), df_videos_process.iloc[batch_idx0:batch_idx0 + batch_size].iterrows())

url Thanks for your suggestion，I have a question earlier that the video_face_checkpoint_path's exists. Just here ,the list return None.I wonder whether the value should be None?

If the list is None and you don't see explicit Exceptions printed out, it means the video reader returned 0 frames. My guess is that this is a problem of dataset, since you are trying to run the pipeline on Celeb DF which is not one of the two tested dataset (DFDC and FF++).

We can do our best for helping you anyway, would you mind attaching your modified version of extract_faces.py and the parameters you give as input?

Thank you very much！Cuz I want to try it on Celeb-DF.This is the extract_faces_py.

this is the code

import argparse
import sys
import traceback
from concurrent.futures import ThreadPoolExecutor
from functools import partial
from pathlib import Path
from typing import Tuple, List

import numpy as np
import pandas as pd
import torch
import torch.cuda
from PIL import Image
from tqdm import tqdm

import blazeface
from blazeface import BlazeFace, VideoReader, FaceExtractor
from isplutils.utils import adapt_bb
import os

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--source', type=Path, default='Celeb_DF',help='Videos root directory', required=False)
    parser.add_argument('--videodf', type=Path, default='data/celebdf_videos.pkl',help='Path to read the videos DataFrame', required=False)
    parser.add_argument('--facesfolder', type=Path,default='faces/celeb/output', help='Faces output root directory', required=False)
    parser.add_argument('--facesdf', type=Path, default='faces/celeb/output',help='Path to save the output DataFrame of faces', required=False)
    parser.add_argument('--checkpoint', type=Path, default='/tmp/per/video/outputs',help='Path to save the temporary per-video outputs', required=False)

    parser.add_argument('--fpv', type=int, default=32, help='Frames per video')
    parser.add_argument('--device', type=torch.device,
                        default=torch.device('cuda:0' if torch.cuda.is_available() else 'cpu'),
                        help='Device to use for face extraction')
    parser.add_argument('--collateonly', help='Only perform collation of pre-existing results', action='store_true')
    parser.add_argument('--noindex', help='Do not rebuild the index', action='store_false')
    parser.add_argument('--batch', type=int, help='Batch size', default=16)
    parser.add_argument('--threads', type=int, help='Number of threads', default=8)
    parser.add_argument('--offset', type=int, help='Offset to start extraction', default=0)
    parser.add_argument('--num', type=int, help='Number of videos to process', default=0)
    parser.add_argument('--lazycheck', action='store_true', help='Lazy check of existing video indexes')
    parser.add_argument('--deepcheck', action='store_true', help='Try to open every image')

    args = parser.parse_args()

    ## Parameters parsing
    device: torch.device = args.device
    source_dir: Path = args.source
    facedestination_dir: Path = args.facesfolder
    frames_per_video: int = args.fpv
    videodataset_path: Path = args.videodf
    facesdataset_path: Path = args.facesdf
    collateonly: bool = args.collateonly
    batch_size: int = args.batch
    threads: int = args.threads
    offset: int = args.offset
    num: int = args.num
    lazycheck: bool = args.lazycheck
    deepcheck: bool = args.deepcheck
    checkpoint_folder: Path = args.checkpoint
    index_enable: bool = args.noindex

    ## Parameters
    face_size = 512

    print('Loading video DataFrame')
    df_videos = pd.read_pickle(videodataset_path)

    if num > 0:
        df_videos_process = df_videos.iloc[offset:offset + num]
    else:
        df_videos_process = df_videos.iloc[offset:]

    if not collateonly:

        ## Blazeface loading
        print('Loading face extractor')
        facedet = BlazeFace().to(device)
        facedet.load_weights("blazeface/blazeface.pth")
        facedet.load_anchors("blazeface/anchors.npy")
        videoreader = VideoReader(verbose=False)
        video_read_fn = lambda x: videoreader.read_frames(x, num_frames=frames_per_video)

        face_extractor = FaceExtractor(video_read_fn, facedet)

        ## Face extraction
        with ThreadPoolExecutor(threads) as p:
            for batch_idx0 in tqdm(np.arange(start=0, stop=len(df_videos_process), step=batch_size),
                                   desc='Extracting faces'):
                tosave_list = list(p.map(partial(process_video,
                                                 source_dir=source_dir,
                                                 facedestination_dir=facedestination_dir,
                                                 checkpoint_folder=checkpoint_folder,
                                                 face_size=face_size,
                                                 face_extractor=face_extractor,
                                                 lazycheck=lazycheck,
                                                 deepcheck=deepcheck,
                                                 ),
                                         df_videos_process.iloc[batch_idx0:batch_idx0 + batch_size].iterrows()))

                for tosave in tosave_list:
                    if tosave is not None:
                        if len(tosave[2]):
                            list(p.map(save_jpg, tosave[2]))
                        tosave[1].parent.mkdir(parents=True, exist_ok=True)
                        tosave[0].to_pickle(str(tosave[1]))

    if index_enable:
        # Collect checkpoints
        df_videos['nfaces'] = np.zeros(len(df_videos), np.uint8)
        faces_dataset = []
        for idx, record in tqdm(df_videos.iterrows(), total=len(df_videos), desc='Collecting faces results'):
            # Checkpoint
            video_face_checkpoint_path = checkpoint_folder.joinpath(record['path']).with_suffix('.faces.pkl')
            if video_face_checkpoint_path.exists():
                try:
                    print(video_face_checkpoint_path)
                    df_video_faces = pd.read_pickle(str(video_face_checkpoint_path))
                    # Fix same attribute issue
                    df_video_faces = df_video_faces.rename(columns={'subject': 'videosubject'}, errors='ignore')
                    nfaces = len(
                        np.unique(df_video_faces.index.map(lambda x: int(x.split('_subj')[1].split('.jpg')[0]))))
                    df_videos.loc[idx, 'nfaces'] = nfaces
                    faces_dataset.append(df_video_faces)
                except Exception as e:
                    print('Error while reading: {}'.format(video_face_checkpoint_path))
                    print(e)
                    video_face_checkpoint_path.unlink()

        # Save videos with updated faces
        print('Saving videos DataFrame to {}'.format(videodataset_path))
        df_videos.to_pickle(str(videodataset_path))

        if offset is not None:
            if num is not None:
                facesdataset_path = facesdataset_path.parent.joinpath(str(facesdataset_path.parts[-1]).split('.')[0]+'_from_video_{}_to_video_{}.pkl'.format(offset, num+offset))
            else:
                facesdataset_path = facesdataset_path.parent.joinpath(str(facesdataset_path.parts[-1]).split('.')[0] + '_from_video_{}.pkl'.format(offset))
        elif num is not None:
            facesdataset_path = facesdataset_path.parent.joinpath(str(facesdataset_path.parts[-1]).split('.')[0] + '_from_video_{}_to_video_{}.pkl'.format(0, num))

        # Creates directory (if doesn't exist)
        facesdataset_path.parent.mkdir(parents=True, exist_ok=True)
        print('Saving faces DataFrame to {}'.format(facesdataset_path))

        df_faces = pd.concat(faces_dataset, axis=0, )
        df_faces['video'] = df_faces['video'].astype('category')
        for key in ['kp1x', 'kp1y', 'kp2x', 'kp2y', 'kp3x',
                    'kp3y', 'kp4x', 'kp4y', 'kp5x', 'kp5y', 'kp6x', 'kp6y', 'left',
                    'top', 'right', 'bottom', ]:
            df_faces[key] = df_faces[key].astype(np.int16)
        df_faces['videosubject'] = df_faces['videosubject'].astype(np.int8)
        # Eventually remove duplicates
        df_faces = df_faces.loc[~df_faces.index.duplicated(keep='first')]
        fields_to_preserve_from_video = [i for i in ['folder', 'subject', 'scene', 'cluster', 'nfaces'] if
                                         i in df_videos]
        df_faces = pd.merge(df_faces, df_videos[fields_to_preserve_from_video], left_on='video',
                            right_index=True)
        df_faces.to_pickle(str(facesdataset_path))

    print('Completed!')

def save_jpg(args: Tuple[Image.Image, Path or str]):
    image, path = args
    image.save(path, quality=95, subsampling='4:4:4')

def process_video(item: Tuple[pd.Index, pd.Series],
                  source_dir: Path,
                  facedestination_dir: Path,
                  checkpoint_folder: Path,
                  face_size: int,
                  face_extractor: FaceExtractor,
                  lazycheck: bool = False,
                  deepcheck: bool = False,
                  ) -> (pd.DataFrame, Path, pd.DataFrame, Path, List[Tuple[Image.Image, Path]]) or None:

    # Instatiate Index and Series
    idx, record = item

    # Checkpoint
    video_faces_checkpoint_path = checkpoint_folder.joinpath(record['path']).with_suffix('.faces.pkl')

    if not lazycheck:
        if video_faces_checkpoint_path.exists():
            try:
                df_video_faces = pd.read_pickle(str(video_faces_checkpoint_path))
                for _, r in df_video_faces.iterrows():
                    face_path = facedestination_dir.joinpath(r.name)
                    assert (face_path.exists())
                    if deepcheck:
                        img = Image.open(face_path)
                        img_arr = np.asarray(img)
                        assert (img_arr.ndim == 3)
                        assert (np.prod(img_arr.shape) > 0)
            except Exception as e:
                print('Error while checking: {}'.format(video_faces_checkpoint_path))
                print(e)
                video_faces_checkpoint_path.unlink()

    if not (video_faces_checkpoint_path.exists()):

        try:

            video_face_dict_list = []

            # Load faces
            frames = face_extractor.process_video(source_dir.joinpath(record['path']))

            if len(frames) == 0:
                return

            face_extractor.keep_only_best_face(frames)
            for frame_idx, frame in enumerate(frames):
                frames[frame_idx]['subjects'] = [0] * len(frames[frame_idx]['detections'])

            # Extract and save faces, bounding boxes, keypoints
            images_to_save: List[Tuple[Image.Image, Path]] = []
            for frame_idx, frame in enumerate(frames):
                if len(frames[frame_idx]['detections']):
                    fullframe = Image.fromarray(frames[frame_idx]['frame'])

                    # Preserve the only found face even if not a good one, otherwise preserve only clusters > -1
                    subjects = np.unique(frames[frame_idx]['subjects'])
                    if len(subjects) > 1:
                        subjects = np.asarray([s for s in subjects if s > -1])

                    for face_idx, _ in enumerate(frame['faces']):
                        subj_id = frames[frame_idx]['subjects'][face_idx]
                        if subj_id in subjects:  # Exclude outliers if other faces detected
                            face_path = facedestination_dir.joinpath(record['path'], 'fr{:03d}_subj{:1d}.jpg'.format(
                                frames[frame_idx]['frame_idx'], subj_id))

                            face_dict = {'facepath': str(face_path.relative_to(facedestination_dir)), 'video': idx,
                                         'label': record['label'], 'videosubject': subj_id, 'class': record['class'],
                                         'source': record['source'], 'quality': record['quality'], 'original': record['original']}
                            for field_idx, key in enumerate(blazeface.BlazeFace.detection_keys):
                                face_dict[key] = frames[frame_idx]['detections'][face_idx][field_idx]

                            cropping_bb = adapt_bb(frame_height=fullframe.height,
                                                   frame_width=fullframe.width,
                                                   bb_height=face_size,
                                                   bb_width=face_size,
                                                   left=face_dict['xmin'],
                                                   top=face_dict['ymin'],
                                                   right=face_dict['xmax'],
                                                   bottom=face_dict['ymax'])
                            face = fullframe.crop(cropping_bb)

                            for key in blazeface.BlazeFace.detection_keys:
                                if (key[0] == 'k' and key[-1] == 'x') or (key[0] == 'x'):
                                    face_dict[key] -= cropping_bb[0]
                                elif (key[0] == 'k' and key[-1] == 'y') or (key[0] == 'y'):
                                    face_dict[key] -= cropping_bb[1]

                            face_dict['left'] = face_dict.pop('xmin')
                            face_dict['top'] = face_dict.pop('ymin')
                            face_dict['right'] = face_dict.pop('xmax')
                            face_dict['bottom'] = face_dict.pop('ymax')

                            face_path.parent.mkdir(parents=True, exist_ok=True)
                            images_to_save.append((face, face_path))

                            video_face_dict_list.append(face_dict)

            if len(video_face_dict_list) > 0:

                df_video_faces = pd.DataFrame(video_face_dict_list)
                df_video_faces.index = df_video_faces['facepath']
                del df_video_faces['facepath']

                # type conversions
                for key in ['kp1x', 'kp1y', 'kp2x', 'kp2y', 'kp3x', 'kp3y',
                            'kp4x', 'kp4y', 'kp5x', 'kp5y', 'kp6x', 'kp6y', 'left', 'top',
                            'right', 'bottom']:
                    df_video_faces[key] = df_video_faces[key].astype(np.int16)
                df_video_faces['conf'] = df_video_faces['conf'].astype(np.float32)
                df_video_faces['video'] = df_video_faces['video'].astype('category')

                video_faces_checkpoint_path.parent.mkdir(parents=True, exist_ok=True)

            else:
                print('No faces extracted for video {}'.format(record['path']))
                df_video_faces = pd.DataFrame()

            return df_video_faces, video_faces_checkpoint_path, images_to_save

        except Exception as e:
            print('Error while processing: {}'.format(record['path']))
            print("-" * 60)
            traceback.print_exc(file=sys.stdout, limit=5)
            print("-" * 60)
            return

if __name__ == '__main__':
    main()

The code you pasted is exactly the same of the original script, but from the first error message of this thread:

File "/home/ubuntu/project/icpr2020dfdc-master/extract_faces.py", line 164, in main df_faces = pd.concat(faces_dataset, axis=0, )

I can see that the line df_faces = pd.concat(faces_dataset, axis=0, ) moved from line 154 (in the orignal script) to line 164 (your modified script).

I need your modified script to try to understand what's going on.

The code you pasted is exactly the same of the original script, but from the first error message of this thread:

File "/home/ubuntu/project/icpr2020dfdc-master/extract_faces.py", line 164, in main df_faces = pd.concat(faces_dataset, axis=0, )

I can see that the line df_faces = pd.concat(faces_dataset, axis=0, ) moved from line 154 (in the orignal script) to line 164 (your modified script).

I need your modified script to try to understand what's going on.

ok.this is the index_ffpp.py and extract_faces.py. Thank you I have check it and rerun it

index_ffpp.py https://gist.github.com/IItaly/b03ab0bcec336b39b4eb7542b6e3ba5a

extract_faces.py https://gist.github.com/IItaly/23e5a3222af02ce3d6e14f3ab1237f1c

the results: Loading video DataFrame Loading face extractor Extracting faces: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:00<00:00, 250.13it/s] Collecting faces results: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1203/1203 [00:00<00:00, 12612.64it/s] Saving videos DataFrame to data/celebdf_videos.pkl Saving faces DataFrame to faces/celeb/output_from_video_0_to_video_0.pkl Traceback (most recent call last): File "/home/ubuntu/project/icpr2020dfdc-master/extract_faces.py", line 306, in main() File "/home/ubuntu/project/icpr2020dfdc-master/extract_faces.py", line 154, in main df_faces = pd.concat(faces_dataset, axis=0, ) File "/home/ubuntu/.conda/envs/icpr2020/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 284, in concat sort=sort, File "/home/ubuntu/.conda/envs/icpr2020/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 331, in init raise ValueError("No objects to concatenate") ValueError: No objects to concatenate

I replicate your case and I managed to extract the faces.

▶ python extract_faces.py --source /nas/public/dataset/celeb-df --facesfolder data/facecache/celeb_df/ --videodf data/celebdf_videos.pkl --facesdf data/celeb_df_faces.pkl --checkpoint tmp/celeb_df_prep/ --fpv 5
Loading video DataFrame
Loading face extractor
Extracting faces: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [04:37<00:00,  3.65s/it]
Collecting faces results: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1203/1203 [00:05<00:00, 210.15it/s]
Saving videos DataFrame to data/celebdf_videos.pkl
Saving faces DataFrame to data/celeb_df_faces_from_video_0_to_video_0.pkl
Completed!

I indexed the celeb by running your modified version of index_ffpp.py and then I just run extract_faces.py with 5 frames per video to save time. Are you sure you are not messing with the dataset/dataframe/saving paths?

By the way, we just pushed a bug fix for a problem with dataframe creation (#20). You can try to re-run your code after a pull, I can't exclude the two problems might be related.

I replicate your case and I managed to extract the faces.

▶ python extract_faces.py --source /nas/public/dataset/celeb-df --facesfolder data/facecache/celeb_df/ --videodf data/celebdf_videos.pkl --facesdf data/celeb_df_faces.pkl --checkpoint tmp/celeb_df_prep/ --fpv 5
Loading video DataFrame
Loading face extractor
Extracting faces: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [04:37<00:00,  3.65s/it]
Collecting faces results: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1203/1203 [00:05<00:00, 210.15it/s]
Saving videos DataFrame to data/celebdf_videos.pkl
Saving faces DataFrame to data/celeb_df_faces_from_video_0_to_video_0.pkl
Completed!

By the way, we just pushed a bug fix for a problem with dataframe creation (#20). You can try to re-run your code after a pull, I can't exclude the two problems might be related.

Thanks for your help.I guess that the environment is not configured cuz I found that the runtime error of the cudnn. Since the code have no problem I will re-try it.

We just released an "official" way to index the Celeb-DF (v2) dataset, have a look at the script index_celebdf.py.

We just released an "official" way to index the Celeb-DF (v2) dataset, have a look at the script index_celebdf.py.

Greate Help! Thank you

I'm closing this for inactivity. Feel free to reopen if you need it.

polimi-ispl / icpr2020dfdc

problems of extract_faces.py #18