About the generalization ability of which train on MMPD (not mini-MMPD) and test on UBFC

[pengfeiZhao1993]

Config file : MMPD_MMPD_UBFC_TSCAN_BASIC.yaml

TRAIN: \ BATCH_SIZE: 4 \ EPOCHS: 30 \ LR: 9e-3 \ MODEL_FILE_NAME: MMPD_MMPD_UBFC_tscan \ DATA: \ INFO: \ LIGHT: [1, 2, 3] # 1 - LED-Low, 2 - LED-high, 3 - Incandescent, 4 - Nature \ MOTION: [1] # 1 - Stationary, 2 - Rotation, 3 - Talking, 4 - Walking \ EXERCISE: [2] # 1 - True, 2 - False \ SKIN_COLOR: [3] # Fitzpatrick Scale Skin Types - 3, 4, 5, 6 \ GENDER: [1, 2] # 1 - Male, 2 - Female \ GLASSER: [1, 2] # 1 - True, 2 - False \ HAIR_COVER: [1, 2] # 1 - True, 2 - False \ MAKEUP: [1, 2] # 1 - True, 2 - False`

Finally I got the Test result as follow:

FFT MAE (FFT Label):18.10546875 \ FFT RMSE (FFT Label):28.37179621284168 \ FFT MAPE (FFT Label):16.860544848875065 \ FFT Pearson (FFT Label):0.33862065963207777

I have checked the data, label, filename, but all of these seem right. Can you help me find what's going on here?

[yahskapar]

Hi @pengfeiZhao1993,

I'm in the process of downloading the full-resolution version of the MMPD dataset, and will get back to you with a direct comparison within the next day or two if nobody else does.

I should note that does look like an unusual result, and even with Mini-MMPD I didn't see results that poor in the past when using Mini-MMPD for inter-dataset testing (including on UBFC-rPPG). When you say you "checked the data, label, filename", are you sure the file sizes themselves look reasonable? You can compare your local file sizes to the file sizes shown in the OneDrive download link that the MMPD authors distribute, for example. Also, you should have 20 files per a subject, leading to 660 .mat files total in your download of the dataset. I can give you some additional pointers if that all looks good, but let me also try to reproduce the result in question once I download the full-resolution version of the dataset myself.

[pengfeiZhao1993]

Thank you for your reply @yahskapar .

I have double-checked the dataset, it consists of 33 subjects and each contains 20 .mat files corresponding to 20 senarios. I also debug into code to check inputs, labels, network, but find nothing wrong.

Besides, I download mini-MMPD, train on mini-MMPD, and then test on UBFC, the result is a little bit better, but still bad. For this test, the config file is as follow,

BASE: [''] \ TOOLBOX_MODE: "train_and_test" # "train_and_test" or "only_test" \ TRAIN: \ BATCH_SIZE: 4 \ EPOCHS: 30 \ LR: 9e-3 \ MODEL_FILE_NAME: miniMMPD_miniMMPD_UBFC_tscan \ DATA: INFO : \ LIGHT: [1, 2, 3] # 1 - LED-Low, 2 - LED-high, 3 - Incandescent, 4 - Nature \ MOTION: [1] # 1 - Stationary, 2 - Rotation, 3 - Talking, 4 - Walking \ EXERCISE: [2] # 1 - True, 2 - False \ SKIN_COLOR: [3] # Fitzpatrick Scale Skin Types - 3, 4, 5, 6 \ GENDER: [1, 2] # 1 - Male, 2 - Female \ GLASSER: [1, 2] # 1 - True, 2 - False \ HAIR_COVER: [1, 2] # 1 - True, 2 - False \ MAKEUP: [1, 2] # 1 - True, 2 - False \ FS: 30 \ DATASET: MMPD \ DO_PREPROCESS: True # if first time, should be true \ DATA_FORMAT: NDCHW \ DATA_PATH: "/data/rPPG-Datasets/mini_MMPD" # Raw dataset path, need to be updated \ CACHED_PATH: "/data/rPPG-Datasets/PreprocessedData" # Processed dataset save path, need to be updated

The result shows below:

===Testing=== \ Testing uses best epoch selected using model selection as non-pretrained model! \ PreTrainedModels/MMPD_SizeW72_SizeH72_ClipLength180_DataTypeDiffNormalized_Standardized_LabelTypeDiffNormalized_Large_boxTrue_Large_size1.5_Dyamic_DetFalse_det_len180/miniMMPD_miniMMPD_UBFC_tscan_Epoch24.pth \ FFT MAE (FFT Label):18.78662109375 \ FFT RMSE (FFT Label):28.438763433204006 \ FFT MAPE (FFT Label):17.431503774446146 \ FFT Pearson (FFT Label):0.3137744093377981

[McJackTang]

To be clear, we didn't give any official results about models trained on MMPD. All the data is trained on PURE/UBFC/SCAMPS and tested on MMPD. The pre-trained model trained on MMPD would be public with the following papers.

[yahskapar]

Hi @pengfeiZhao1993,

I got the below results with the full-res version of the dataset.

Training on MMPD without validation set, test on UBFC:

FFT MAE (FFT Label):17.347935267857142 FFT RMSE (FFT Label):28.862467124904665 FFT MAPE (FFT Label):16.028916001349657 FFT Pearson (FFT Label):0.32945984607956735

Training on MMPD with validation set, test on UBFC:

FFT MAE (FFT Label):16.950334821428573 FFT RMSE (FFT Label):27.372707681550523 FFT MAPE (FFT Label):15.620547307264458 FFT Pearson (FFT Label):0.353869973491664

Pretty similar results to what you got using the same config settings. I think there could be a variety of reasons for this poor performance, and it may be worth investigating if fiddling with the exclusion criteria (e.g., skin tones, lighting, and motion type) could help troubleshoot if certain kinds of MMPD training data are more problematic than others. I happen to be using parts of the MMPD dataset for my research, so I'll try to update you on this issue if I find anything interesting. Sorry for not being of more help, the saner looking results I was referring to in my earlier reply might've just been with the natural lighting videos and skin type 3, or perhaps some other combination that didn't involve more than one or two lighting types.

[pengfeiZhao1993]

@yahskapar OKay， I got it， Thank you!

Then I will also work on this issue, and share to you if there is some findings.

[Promisery]

I noticed that MMPD is collected at 1280x720 and then compressed to 320x240, which is not the same aspect ratio. Therefore, I guess we should first resize the frames to the original aspect ratio. In my own experiments, I can make MMPD work by (1) resize the images to 320x180 before further preprocessing using skimage.transform.resize, and (2) keep only ['Stationary', 'Stationary (after exercise)'] motions. Hopefully this can help you.

[yahskapar]

I noticed that MMPD is collected at 1280x720 and then compressed to 320x240, which is not the same aspect ratio. Therefore, I guess we should first resize the frames to the original aspect ratio. In my own experiments, I can make MMPD work by (1) resize the images to 320x180 before further preprocessing using skimage.transform.resize, and (2) keep only ['Stationary', 'Stationary (after exercise)'] motions. Hopefully this can help you.

Interesting observations, @Promisery, thanks for sharing. Were you unable to train with any other motion types aside from stationary? If you were to visualize the preprocessed data when using non-stationary motion types, I wonder if you would notice any issues that would likely stem from face cropping not being robust enough when motion is present. This is something I plan to explore myself a bit when I have time in the coming two weeks or so.

[yahskapar]

I'm closing this issue for the time being as it seems no further discussion is taking place. Feel free to re-open the issue or create a new one in case any interesting points of discussion regarding the MMPD dataset come up.