search

ubicomplab / rPPG-Toolbox

rPPG-Toolbox: Deep Remote PPG Toolbox (NeurIPS 2023)
https://arxiv.org/abs/2210.00716
Other
442 stars 106 forks source link

Testing custom video on pre-trained model #221

Closed hakansndkc5 closed 9 months ago

hakansndkc5 commented 10 months ago

Hey! I've seen the issue about this problem but i could not fix it out. My problem is if i have a video with 5 secs included human face (like in datasets) how can i generate a ground_truth file and use it on DataLoader? I hope my question is clear. If you can help i'll be appreciate.

yahskapar commented 10 months ago

As it stands now, this toolbox offers the ability to use pseudo-labels instead of existing ground truth labels with a variety of dataloaders, for example with the UBFC-rPPG dataloader as shown below:

https://github.com/ubicomplab/rPPG-Toolbox/blob/d992de556fbbd6c865e38d475e25ec4eccebb55e/dataset/data_loader/UBFCrPPGLoader.py#L84-L89

Refer to any of the config examples involving BP4D, such as this one, for more details on how to setup a config file to use pseudo PPG labels. One thing you could do is create a custom dataloader for your video, and similar custom videos you might have later on, by following the instructions here. At that point, you can look at the UBFC-rPPG dataloader or other similar dataloaders to use pseudo-labels within the context of the toolbox.

If you just want to generate a ground truth file, such as a .csv, to use consistently as a part of a dataset you might be trying to create with pseudo-labels, I recommend using the lines of code here in the generate_pos_psuedo_labels() function in your own code and just adding some simple code to write the resulting pseudo-label out to a .csv file.

Hope this helps!

hakansndkc5 commented 10 months ago

Actually i was using the this data loader and i already applied all the steps but i could not get any metrics (MAE,RMSE etc.) What should i do to see them, (here) [https://github.com/ubicomplab/rPPG-Toolbox/issues/191] the user says that he/she can see results and i can not. Pls help me :))

yahskapar commented 10 months ago

Can you reply back with an image of the exact error you might be running into or give more details related to what happens when you try to run inference and get metrics on your video?

It's possible that, because you're running on a single custom video, you might find that some of the metrics look off because of how they're calculated (especially the Pearson correlation). A few more details can be found in @girishvn's reply here, and #220 itself may be related to the issue you are running into.

hakansndkc5 commented 10 months ago

When i use custom videos with CustomDataLoader.py as i mentioned before, i can preprocess datasets but after the test i can not get any valid error values. Screenshot in below. image

yahskapar commented 10 months ago

My guess is that there is some bug in your dataloader somewhere, assuming you haven't modified anything else. What's the length of your custom video? If your config is using data chunking and has a chunk size of 180, your preprocessed dataset length indicates that you ended up with just a single chunk of 180 frames (6 seconds of video given the video is 30 FPS). Does that sound correct? If not, definitely check your dataloader carefully.

Aside from that, it could also be that somehow you're able to predict the heart rate of your custom video perfectly. One way to check this is to print out the inputs to a given metric calculation. For example in the below line of code:

https://github.com/ubicomplab/rPPG-Toolbox/blob/d992de556fbbd6c865e38d475e25ec4eccebb55e/evaluation/metrics.py#L111

Print out gt_hr_fft_all and predict_hr_fft_all and check their values. If you have heart rate values there with the GT matching the predicted precisely, it makes sense why you would get results like you have gotten. You may want to record more custom videos, perhaps with varied heart rates, in that case and try those.

yahskapar commented 9 months ago

I'm going to go ahead and close this issue due to the lack of follow-up, but please feel free to re-open or create a new issue if you continue having problems @hakansndkc5.