search

ubicomplab / rPPG-Toolbox

rPPG-Toolbox: Deep Remote PPG Toolbox (NeurIPS 2023)
https://arxiv.org/abs/2210.00716
Other
504 stars 129 forks source link

MR-NIRP Dataset support #244

Closed koutsd closed 7 months ago

koutsd commented 9 months ago
  1. Created dataloader for the MR-NIRP dataset
  2. Added Sliding window overlap config setting for inference
  3. Edited the NN model trainers so that they can load and use pretrained weights on cpu
yahskapar commented 8 months ago

Hi @koutsd,

Thanks for this PR, there's quite a few changes so I won't have time to review it fully just yet. For the time being, can you note here what the latest results you managed to get with MR-NIRP were? When I get around to looking into this (apologies, I've been too occupied with a variety of things including moving across the country), the first thing I'd like to do is try to reproduce your results.

koutsd commented 8 months ago

Hi @yahskapar,

Sorry that the PR is a bit all over the place. For some reason I thought that subsequent commits will not update the PR. Here is a list of what I changed to hopefully make reviewing the PR easier:

  1. Created dataloader for the MR-NIRP dataset
  2. Added Sliding window overlap config setting for inference
  3. Edited the NN model trainers so that they can load and use pretrained weights on cpu
  4. Added ROI extraction preprocessing option using mediapipe facemesh (needs some polishing)
  5. Added save output feature to usupervised methods
  6. Restructured the evaluation code because there was a lot of repeated code and was making some of my tests difficult
  7. Edited the PhysFormer trainer to save the outputs on the cpu like it is done on the other trainers

Here are some inference results using the pretrained models you provide in the toolbox:

image

There are some videos from the dataset that are excluded from the tests. You can find the config files used here: https://github.com/koutsd/rPPG-Toolbox/tree/main/configs/infer_configs They are the ones named MR-NIRP{model name}{train dataset}.yaml where train dataset is PURE, UBFC-rPPG or SCAMPS.

I am also working on some training results but there have been some issues regarding the loss function as you can see here: https://github.com/ubicomplab/rPPG-Toolbox/issues/254#issuecomment-1986452147

yahskapar commented 8 months ago

Thanks @koutsd, this is a great summary! I will try to get to this and taking a deeper look at #254 toward the tail-end of this week.

Regarding the pre-trained models you tried already, can you also try one of the motion augmented ones? For example, MA-UBFC_tscan.pth.

koutsd commented 8 months ago

Hi @yahskapar,

Here are some the results with from MA-UBFC_tscan.pth. The same config file as before was used. I assumed this model also takes diff-normalized input.

image

yahskapar commented 7 months ago

Hi @koutsd,

Did you mean to close this? Sorry, I still haven't gotten around to reviewing it fully yet but can plan to take a thorough look on Friday.

koutsd commented 7 months ago

Hi @yahskapar,

I apologize about this. I had to make the repo private for the moment. I do plan on making another PR with trained models when everything is cleared out.

yahskapar commented 7 months ago

No problem - sounds good, feel free to ping me when it's back up for me to take a look.