Getting same csv generated by api for different video files

[rashutyagi]

Hey, I am facing an issue that while using the api in the code, when I run "run_example.py" in the api folder then it is generating me exact same csv for even different video files. What could be the possible reason for it ?

[wtomin]

What do you mean by "exact same csv for different video files"? Is it the csv contains the same predictions for different videos, or the csv file gets overwritten because the the video files have the same video name?

[rashutyagi]

By exact same CSV I mean that The CSV contains the same predictions for different videos.

[wtomin]

The emotion_api accepts the video path. Then it uses OpenFace to extract faces from video frames, and save them to opface_output_dir.

    def run(self, video_file , csv_output = None):
        video_cap = cv2.VideoCapture(video_file)
        video_name = os.path.basename(video_file).split('.')[0]
        # first input video is processed using OpenFace
        opface_output_dir = os.path.join(os.path.dirname(video_file), 
                video_name+"_opface")
        if not os.path.exists(opface_output_dir):
            self.video_processor.process(video_file, opface_output_dir)
        assert len(os.listdir(opface_output_dir)) >0 , "The OpenFace output directory should not be empty: {}".format(opface_output_dir)

Can you check if you have different opface_output_dir for different videos?

[rashutyagi]

Yes different opface directories are being created for different videos, as you can see below

Aligned faces folder in the opface directory contains the faces of that person only which is there in video so I don't think there is issue from open face side.

But when I compare the model's predicted csv's for two different files this is what I am getting, the exact same csv as shown below

[wtomin]

I have a suggestion for you, in Emotion_API.test_ensemble line 203 to 215, you can print out the single_model's predictions and the ensemble_model's predictions of different videos to check if they were identical.

    def test_ensemble(self, dataloader):
        models_preds = {}
        for i_model, model in enumerate(self.ensemble):
            preds = self.test_single_model(model, dataloader)
            models_preds[i_model] = preds # print this
        ensemble_outputs = {}
        frames_ids = models_preds[0][TASKS[0]]['frames_ids']
        for task in TASKS:
            ensemble_outputs[task] = torch.stack([models_preds[i_model][task]['outputs'] for i_model in range(len(self.ensemble))], dim=0).mean(0)
            assert all([(models_preds[i_model][task]['frames_ids']==frames_ids).all() for i_model in range(len(self.ensemble))])
        ensemble_outputs = self._format_estimates(ensemble_outputs) #print this
        ensemble_outputs.update({"frames_ids": frames_ids})
        return ensemble_outputs

[rashutyagi]

Okay sure I will do it.

One more information I would like to convey is that. Instead of api I tried using the command as mentioned in root directory readme :

For CNN model: python run_pretrained_model.py --image_dir directory-containing-sequence-of-face-images --model_type CNN --batch_size 12 --eval_with_teacher --eval_with_students --save_dir save-directory --workers 8 --ensemble

For the above command, I used the batch size of 4 (as I have a smaller GPU in my system) and using the pretrained student and teacher weights and for all the 5 student models here are the predictions for Valence and Arousal (you can see that the predictions are repeating after every 4 frames in this ) and batch size was also 4. There might be some connection?

(The below 5 student models results are on one single video file named "random_expressions.mp4")

And the below results are for a different video file named as "s11_trial_09.avi" (for same batch size of 4 only)

Below images show the comparative predicted values by each of the student model for two different video files :

Student_0 :

Student_1 :

Student_2 :

Student_3 :

Student_4 :

Teacher :

Different student models are giving different outputs but the outputs are same for even different videos here . And outputs are also repeating after every 4 frames here and batch size is also 4. What can be the possible reason if any in case you feel ?

[rashutyagi]

Is the above issue due to this --> https://github.com/huggingface/datasets/issues/3423

[wtomin]

If data duplicates were caused by multi-processing, then you will not see the duplicates when num_workers=0.

Another suggestion from me is, since you ran "run_pretrained_model.py", you can set a break point at line 548 in run_pretrained_model.py.

    mask = np.zeros_like(frames_ids, dtype=bool)
    mask[np.unique(frames_ids, return_index=True)[1]] = True
    frames_ids = frames_ids[mask]
    predictions = predictions[mask]
    assert len(frames_ids) == len(predictions)

If there were repetitive frames loaded in the dataloader, they can be identified by frames_ids, after all, the mask here is used to filter repetitive frames.

[rashutyagi]

Thank you for the help ma'am. The above issue has been rectified. This was happening due to improper working of Cuda with the GPU of 30 series of Nvidia. After changing the versions of the Cuda toolkit this issue was resolved. Thankyou