Bland-Altman of UBFC-rppg

[Harry2345]

Thank you very much for your excellent work, but I recently encountered some problems in drawing the Bland-Altman plot during the UBFC-rPPG verification. Why is it only 12 data points? Does one point represent a subject?

[yahskapar]

Hi @Harry2345,

The plot is formed as a part of metric calculations after chunks are reformed into individual videos (which in the case of some datasets, corresponds to individual subjects).

I'm assuming you are doing intra-dataset experiments (i.e., using only UBFC-rPPG videos for train, val, and test splits) based on your results. How many videos are in your test split? Feel free to share your .yaml config file, as well as how you organized your dataset folder, so that we can help verify. For reference, you should have 42 UBFC-rPPG videos if you downloaded the whole dataset.

[Harry2345]

Yes, I have a total of 42 rppg videos. I use the first 30 subjects as the training set, the last 12 subjects as the test set and the verification set to select the best epoch. I also have a question, when the progress bar shows n/22 for training, n/9 for validation, and n/12 for testing, why is the progress bar different for validation and test sets? BASE: [''] TOOLBOX_MODE: "train_and_test" # "train_and_test" or "only_test" TRAIN: BATCH_SIZE: 16 EPOCHS: 30 LR: 3e-4 MODEL_FILE_NAME: UBFC_UBFC_UBFC_MyRhythmMamba AUG: 1 DATA: FS: 30 DATASET: UBFC-rPPG DO_PREPROCESS: False # if first time, should be true DATA_FORMAT: NDCHW DATA_PATH: "/mnt/d/yzx/rPPGProject/rPPGDataset/UBFC-rppg/dataset2/" # Raw dataset path, need to be updated CACHED_PATH: "/mnt/d/yzx/rPPGProject/rPPGDataset/UBFC-rppg/UBFC_mamba/" # Processed dataset save path, need to be updated EXP_DATA_NAME: "" BEGIN: 0.0 END: 0.72 PREPROCESS : DATA_TYPE: [ 'Standardized' ] LABEL_TYPE: Standardized DO_CHUNK: True CHUNK_LENGTH: 160 CROP_FACE: DO_CROP_FACE: True USE_LARGE_FACE_BOX: True LARGE_BOX_COEF: 1.5 DETECTION: DO_DYNAMIC_DETECTION: False DYNAMIC_DETECTION_FREQUENCY : 30 USE_MEDIAN_FACE_BOX: False # This should be used ONLY if dynamic detection is used RESIZE: H: 128 W: 128 VALID: DATA: FS: 30 DATASET: UBFC-rPPG DO_PREPROCESS: False # if first time, should be true DATA_FORMAT: NDCHW DATA_PATH: "/mnt/d/yzx/rPPGProject/rPPGDataset/UBFC-rppg/dataset2/" # Raw dataset path, need to be updated CACHED_PATH: "/mnt/d/yzx/rPPGProject/rPPGDataset/UBFC-rppg/UBFC_mamba/" # Processed dataset save path, need to be updated EXP_DATA_NAME: "" BEGIN: 0.72 END: 1.0 PREPROCESS: DATA_TYPE: [ 'Standardized' ] LABEL_TYPE: Standardized DO_CHUNK: True CHUNK_LENGTH: 160 CROP_FACE: DO_CROP_FACE: True USE_LARGE_FACE_BOX: True LARGE_BOX_COEF: 1.5 DETECTION: DO_DYNAMIC_DETECTION: False DYNAMIC_DETECTION_FREQUENCY : 30 USE_MEDIAN_FACE_BOX: False # This should be used ONLY if dynamic detection is used RESIZE: H: 128 W: 128 TEST: METRICS: ['MAE','RMSE','MAPE','Pearson','SNR','BA'] USE_LAST_EPOCH: False # to use provided validation dataset to find the best epoch, should be false DATA: FS: 30 DATASET: UBFC-rPPG DO_PREPROCESS: False # if first time, should be true DATA_FORMAT: NDCHW DATA_PATH: "/mnt/d/yzx/rPPGProject/rPPGDataset/UBFC-rppg/dataset2/" # Raw dataset path, need to be updated CACHED_PATH: "/mnt/d/yzx/rPPGProject/rPPGDataset/UBFC-rppg/UBFC_mamba/" # Processed dataset save path, need to be updated EXP_DATA_NAME: "" BEGIN: 0.72 END: 1.0 PREPROCESS: DATA_TYPE: [ 'Standardized' ] LABEL_TYPE: Standardized DO_CHUNK: True CHUNK_LENGTH: 160 CROP_FACE: DO_CROP_FACE: True USE_LARGE_FACE_BOX: True LARGE_BOX_COEF: 1.5 DETECTION: DO_DYNAMIC_DETECTION: False DYNAMIC_DETECTION_FREQUENCY : 30 USE_MEDIAN_FACE_BOX: False # This should be used ONLY if dynamic detection is used RESIZE: H: 128 W: 128 DEVICE: cuda:0 NUM_OF_GPU_TRAIN: 1 LOG: PATH: runs/exp MODEL: DROP_RATE: 0.2 NAME: MyRhythmMamba3 MODEL_DIR: ./output/fastslowmamba3_v1 INFERENCE: BATCH_SIZE: 2 EVALUATION_METHOD: "FFT" # "FFT" or "peak detection" EVALUATION_WINDOW: USE_SMALLER_WINDOW: False # Change this if you'd like an evaluation window smaller than the test video length WINDOW_SIZE: 10 # In seconds MODEL_PATH: "/home/zhixin_yan/project/RhythmMamba/output/mamba2/UBFC-rPPG_SizeW128_SizeH128_ClipLength160_DataTypeStandardized_DataAugNone_LabelTypeStandardized_Crop_faceTrue_Large_boxTrue_Large_size1.5_Dyamic_DetFalse_det_len30_Median_face_boxFalse/UBFC_UBFC_UBFC_MyRhythmMamba_Epoch29.pth"

[yahskapar]

Hi @Harry2345,

Yeah, definitely seems like there are potentially a couple things wrong in your situation.

1. Are you sure there are 42 videos in your UBFC-rPPG/dataset2 folder and that, when preprocessed, no errors are returned and all the videos are successfully preprocessed? Can you perhaps provide terminal output of what the preprocessing looks like and double-check this? I've noticed some folks in the past had corrupted videos and had to redownload them to properly preprocess the whole dataset.
2. Even with your train split, the numbers look off - 0.0 to 0.72 for 42 UBFC-rPPG videos should correspond to 30 videos, not 22. This seems very confusing - how about you try to avoid using validation (e.g., set USE_LAST_EPOCH back to True) and use a train split of 0.0 to 0.8 and a test split of 0.8 to 1.0 as suggested by the default configs? You should have 34 videos loaded for train and 8 videos loaded for test at that point . It would be good to confirm this works as expected before we proceed with debugging. If this doesn't work, can you share any changes you might've made to the dataloaders or preprocessing code?

[yahskapar]

@Harry2345 any update on checking 1 and/or 2 above?

[yahskapar]

I'll go ahead and close this issue due to the inactivity, but feel free to re-open if you're still running into issues @Harry2345.