yahskapar
commented
1 year ago Hi @Arktis2022,
I just tried reproducing the result and was able to get an MAE of 1.76 and a MAPE of 1.96 just now using an NVIDIA RTX 6000. Unfortunately, I don't currently have access to an NVIDIA RTX 4500, which was the GPU I used to train PhysFormer for this toolbox (i.e., the pre-trained model you mentioned). Regardless, what you're getting (MAE around 7) sounds way off, so let's troubleshoot this:
- What are your preprocessed dataset lengths, as reported by the toolbox when you run
main.py? Can you confirm your dataset lengths match mine below:
Cached Data Path /playpen-nas-ssd/akshay/UNC_Google_Physio/preprocessed_gold/train/PURE_SizeW128_SizeH128_ClipLength160_DataTypeDiffNorm
alized_DataAugNone_LabelTypeDiffNormalized_Crop_faceTrue_Large_boxTrue_Large_size1.5_Dyamic_DetFalse_det_len30_Median_face_boxFalse
File List Path /playpen-nas-ssd/akshay/UNC_Google_Physio/preprocessed_gold/train/DataFileLists/PURE_SizeW128_SizeH128_ClipLength160_Data
TypeDiffNormalized_DataAugNone_LabelTypeDiffNormalized_Crop_faceTrue_Large_boxTrue_Large_size1.5_Dyamic_DetFalse_det_len30_Median_face_b
oxFalse_0.0_0.8.csv
train Preprocessed Dataset Length: 596
Cached Data Path /playpen-nas-ssd/akshay/UNC_Google_Physio/preprocessed_gold/val/PURE_SizeW128_SizeH128_ClipLength160_DataTypeDiffNormal
ized_DataAugNone_LabelTypeDiffNormalized_Crop_faceTrue_Large_boxTrue_Large_size1.5_Dyamic_DetFalse_det_len30_Median_face_boxFalse
File List Path /playpen-nas-ssd/akshay/UNC_Google_Physio/preprocessed_gold/val/DataFileLists/PURE_SizeW128_SizeH128_ClipLength160_DataTy
peDiffNormalized_DataAugNone_LabelTypeDiffNormalized_Crop_faceTrue_Large_boxTrue_Large_size1.5_Dyamic_DetFalse_det_len30_Median_face_box
False_0.8_1.0.csv
valid Preprocessed Dataset Length: 154
Cached Data Path /playpen-nas-ssd/akshay/UNC_Google_Physio/preprocessed_gold/test/UBFC-rPPG_SizeW128_SizeH128_ClipLength160_DataTypeDiff
Normalized_DataAugNone_LabelTypeDiffNormalized_Crop_faceTrue_Large_boxTrue_Large_size1.5_Dyamic_DetFalse_det_len30_Median_face_boxFalse
File List Path /playpen-nas-ssd/akshay/UNC_Google_Physio/preprocessed_gold/test/DataFileLists/UBFC-rPPG_SizeW128_SizeH128_ClipLength160_
DataTypeDiffNormalized_DataAugNone_LabelTypeDiffNormalized_Crop_faceTrue_Large_boxTrue_Large_size1.5_Dyamic_DetFalse_det_len30_Median_fa
ce_boxFalse_0.0_1.0.csv
test Preprocessed Dataset Length: 483If they don't match, let's troubleshoot the datasets you downloaded further. I'd also be curious if they somehow matched, but you also got strange results when training with PURE and testing on UBFC-rPPG with another architecture (e.g., TS-CAN).
-
Can you share more details regarding your GPU, your version of PyTorch, and what version of CUDA you compiled PyTorch with? This will vary from toolbox user to toolbox user depending on what type of GPUs they have access to. In my case, since I mainly use RTX A4500s and RTX A6000s, I used
torch 1.8.2withcu111(CUDA 11.1). -
Are you using the default config settings in PURE_PURE_UBFC-rPPG_PHYSFORMER_BASIC.yaml without changing any of the hyperparameters, and without changing other aspects of the toolbox (e.g., dataloading)? The train config as is should produce a result reasonably close to the pre-trained model result that you were able to verify.
My guess is that there is some issue with the datasets you downloaded (e.g., incomplete download, corrupted download, etc) as that is an issue I've seen quite a few times in the past. Also, if it ends up not being the datasets you downloaded having some kind of issue or some difference in your version of the code (e.g., a branch on a fork) that wasn't accounted for, I'd also appreciate it if you shared with us your complete terminal output when training.
Arktis2022
zizheng-guo
lishubing17
I used the PURE_PURE_UBFC-rPPG_PHYSFORMER_BASIC.yaml file in the train_config folder to train physformer on PURE and test it on UBFC-rPPG, but I found differences between the results and those reported in the table. Is this normal, and do I need to change some hyperparameters? The results of the pre-trained model: PURE_PhysFormer_DiffNormalized_UBFC-rPPG_FFT_BlandAltman_ScatterPlot.pdf
The results of my own training model: PURE_PURE_UBFC-rPPG_physformer_Epoch7_UBFC-rPPG_FFT_BlandAltman_ScatterPlot.pdf
My training loss (I trained twice, and in the second attempt, I changed the epoch to 50. However, I used the model trained with 7 epochs for testing in both attempts, and the results of the two tests were the same.): PURE_PURE_UBFC-rPPG_physformer_losses.pdf
Using the given pre-trained model indeed yields the same metrics as those in the table (MAE: 1.44/MAPE: 1.66). However, when training the model using PURE_PURE_UBFC-rPPG_PHYSFORMER_BASIC.yaml as the configuration, both metrics are around 7.