search

ubicomplab / rPPG-Toolbox

rPPG-Toolbox: Deep Remote PPG Toolbox (NeurIPS 2023)
https://arxiv.org/abs/2210.00716
Other
442 stars 106 forks source link

UBFC-Phys pre-procssing saves only S56 and S24 #213

Closed Dylan-H-Wang closed 11 months ago

Dylan-H-Wang commented 11 months ago

Hi,

I used this official config to pre-process UBFC-Phys, but I can only get preprocessed data files for S56 and S24.

BASE: ['']
TOOLBOX_MODE: "only_test"      # "train_and_test"  or "only_test"
TEST:
  METRICS: ['MAE', 'RMSE', 'MAPE', 'Pearson', 'SNR', 'BA']
  USE_LAST_EPOCH: True
  DATA:
    FILTERING:
      USE_EXCLUSION_LIST: True
      EXCLUSION_LIST: [
      's3_T1', 's8_T1', 's9_T1', 's26_T1', 's28_T1', 's30_T1', 's31_T1', 's32_T1',
      's33_T1', 's40_T1', 's52_T1', 's53_T1', 's54_T1', 's56_T1', 's1_T2', 's4_T2',
      's6_T2', 's8_T2', 's9_T2', 's11_T2', 's12_T2', 's13_T2', 's14_T2', 's19_T2',
      's21_T2', 's22_T2', 's25_T2', 's26_T2', 's27_T2', 's28_T2', 's31_T2', 's32_T2',
      's33_T2', 's35_T2', 's38_T2', 's39_T2', 's41_T2', 's42_T2', 's45_T2', 's47_T2',
      's48_T2', 's52_T2', 's53_T2', 's55_T2', 's5_T3', 's8_T3', 's9_T3', 's10_T3',
      's13_T3', 's14_T3', 's17_T3', 's22_T3', 's25_T3', 's26_T3', 's28_T3', 's30_T3',
      's32_T3', 's33_T3', 's35_T3', 's37_T3', 's40_T3', 's47_T3', 's48_T3', 's49_T3',
      's50_T3', 's52_T3', 's53_T3']
      SELECT_TASKS: True
      TASK_LIST: ['T1', 'T2', 'T3']
    FS: 35
    DATASET: UBFC-PHYS
    DO_PREPROCESS: False                    # if first time, should be true
    DATA_FORMAT: NDCHW
    DATA_PATH: "/gscratch/ubicomp/xliu0/data3/mnt/Datasets/UBFC-PHYS/RawData"                     # Raw dataset path, need to be updated
    CACHED_PATH: "/gscratch/ubicomp/xliu0/data3/mnt/Datasets/rppg_toolbox/PreprocessedData"    # Processed dataset save path, need to be updated
    EXP_DATA_NAME: ""
    BEGIN: 0.0
    END: 1.0
    PREPROCESS:
      DATA_TYPE: [ 'DiffNormalized','Standardized' ]
      LABEL_TYPE: DiffNormalized
      DO_CHUNK: True
      CHUNK_LENGTH: 210
      CROP_FACE:
        DO_CROP_FACE: True
        USE_LARGE_FACE_BOX: True
        LARGE_BOX_COEF: 1.5
        DETECTION:
          DO_DYNAMIC_DETECTION: True
          DYNAMIC_DETECTION_FREQUENCY : 35
          USE_MEDIAN_FACE_BOX: True    # This should be used ONLY if dynamic detection is used
      RESIZE:
        H: 72
        W: 72
DEVICE: cuda:0
NUM_OF_GPU_TRAIN: 1
LOG:
  PATH: runs/exp
MODEL:
  DROP_RATE: 0.2
  NAME: Tscan
  TSCAN:
    FRAME_DEPTH: 10
INFERENCE:
  BATCH_SIZE: 4
  EVALUATION_METHOD: FFT        # "FFT" or "peak detection"
  EVALUATION_WINDOW:
    USE_SMALLER_WINDOW: False        # Change this if you'd like an evaluation window smaller than the test video length
    WINDOW_SIZE: 10        # In seconds
  MODEL_PATH:   "./final_model_release/UBFC-rPPG_TSCAN.pth"

There are warnings and errors when running the program

ERROR: No Face Detected
Warning: More than one faces are detected(Only cropping the biggest one.)

Is this the reason?

yahskapar commented 11 months ago

I don't think the warnings or errors are the reason. If no face is detected, cropping simply isn't performed and the whole video is used for re-sizing to, in this case, 72x72. In the case of the warning you got, that's simply a case of multiple faces being detected (even if there is literally only one face in the video) because of poor face detection. I plan on updating this toolbox's face detection significantly sometime by late November after some other conference deadlines of mine, hopefully then many of these errors and warnings that pop up will disappear completely due to a stronger face detection approach being present.

Can you copy/paste the exact config you're using (e.g., with your file paths and all)? Also, how is the UBFC-PHYS folder you are pointing to organized? Does it look like this below?

    RawData/
    |   |-- s1/
    |       |-- vid_s1_T1.avi
    |       |-- vid_s1_T2.avi
    |       |...
    |       |-- bvp_s1_T1.csv
    |       |-- bvp_s1_T2.csv
    |   |-- s2/
    |       |-- vid_s2_T1.avi
    |       |-- vid_s2_T2.avi
    |       |...
    |       |-- bvp_s2_T1.csv
    |       |-- bvp_s2_T2.csv
    |...
    |   |-- sn/
    |       |-- vid_sn_T1.avi
    |       |-- vid_sn_T2.avi
    |       |...
    |       |-- bvp_sn_T1.csv
    |       |-- bvp_sn_T2.csv

If your UBFC-PHYS dataset is organized in the same way and you are sure you downloaded the UBFC-PHYS videos correctly (e.g., you can open and view the folders aside from s56 and s24, as well as their ground truth files aren't empty), I would recommend double-checking your terminal output. The only other possible thing I can think of at the moment is limited CPU or memory causing threads to somehow silently crash, but I would expect your progress bars in the terminal would look off then too and your program itself would crash at some point.

Dylan-H-Wang commented 11 months ago

Here is the config I am using

# use default train/val split with ratio 8/2
BASE: ['']
TOOLBOX_MODE: "train_and_test"
TEST:
  USE_LAST_EPOCH: False
  DATA:
    FS: 35
    DATASET: UBFC-PHYS
    DO_PREPROCESS: True                    # if first time, should be true
    DATA_FORMAT: NCDHW
    DATA_PATH: "./data/UBFC-Phys/raw_data"                     # Raw dataset path, need to be updated
    CACHED_PATH: "./data/UBFC-Phys/preprocessed_data"    # Processed dataset save path, need to be updated
    FILE_LIST_PATH: "./data/UBFC-Phys/preprocessed_data/data_file_list"    # Path to store file lists, needs to be updated
    EXP_DATA_NAME: ""
    BEGIN: 0.0
    END: 1.0
    PREPROCESS:
      DATA_TYPE: ['Standardized'] 
      DATA_AUG: ['None']
      LABEL_TYPE: Standardized
      DO_CHUNK: True
      CHUNK_LENGTH: 180
      CROP_FACE:
        DO_CROP_FACE: True
        USE_LARGE_FACE_BOX: True
        LARGE_BOX_COEF: 1.5
        DETECTION:
          DO_DYNAMIC_DETECTION: True
          DYNAMIC_DETECTION_FREQUENCY : 35
          USE_MEDIAN_FACE_BOX: True    # This should be used ONLY if dynamic detection is used
      RESIZE:
        H: 72
        W: 72

and I just downloaded the dataset a week before. Below is the data structure which I believe is correct and contents are non-empty.

./data/UBFC-Phys/raw_data
├── s1
│   ├── bvp_s1_T1.csv
│   ├── bvp_s1_T2.csv
│   ├── bvp_s1_T3.csv
│   ├── eda_s1_T1.csv
│   ├── eda_s1_T2.csv
│   ├── eda_s1_T3.csv
│   ├── info_s1.txt
│   ├── selfReportedAnx_s1.csv
│   ├── vid_s1_T1.avi
│   ├── vid_s1_T2.avi
│   └── vid_s1_T3.avi
├── s10
│   ├── bvp_s10_T1.csv
│   ├── bvp_s10_T2.csv
│   ├── bvp_s10_T3.csv
│   ├── eda_s10_T1.csv
│   ├── eda_s10_T2.csv
│   ├── eda_s10_T3.csv
│   ├── info_s10.txt
│   ├── selfReportedAnx_s10.csv
│   ├── vid_s10_T1.avi
│   ├── vid_s10_T2.avi
│   └── vid_s10_T3.avi
├── s11
│   ├── bvp_s11_T1.csv
│   ├── bvp_s11_T2.csv
│   ├── bvp_s11_T3.csv
│   ├── eda_s11_T1.csv
│   ├── eda_s11_T2.csv
│   ├── eda_s11_T3.csv
│   ├── info_s11.txt
│   ├── selfReportedAnx_s11.csv
│   ├── vid_s11_T1.avi
│   ├── vid_s11_T2.avi
│   └── vid_s11_T3.avi
├── s12
│   ├── bvp_s12_T1.csv
│   ├── bvp_s12_T2.csv
│   ├── bvp_s12_T3.csv
│   ├── eda_s12_T1.csv
│   ├── eda_s12_T2.csv
│   ├── eda_s12_T3.csv
│   ├── info_s12.txt
│   ├── selfReportedAnx_s12.csv
│   ├── vid_s12_T1.avi
│   ├── vid_s12_T2.avi
│   └── vid_s12_T3.avi
.....

I have also checked the terminal outputs line by line, the above warning and errors messages were all I get.... and the program did not crash, it iterated all files. However, there is a difference compared to preprocessing other datasets. when processing other datasets, the outputs are

26%|█████████████████████████████████████████▋| 44/168 [58:59<2:46:13, 80.43s/it]
Warning: More than one faces are detected(Only cropping the biggest one.)
26%|█████████████████████████████████████████▋| 45/168 [58:59<2:46:13, 80.43s/it]

but UBFC-Phys procssing gave

Warning: More than one faces are detected(Only cropping the biggest one.)
Warning: More than one faces are detected(Only cropping the biggest one.)
...
Warning: More than one faces are detected(Only cropping the biggest one.)
100%|█████████████████████████████████████| 168/168 [58:59<2:46:13, 80.43s/it]

not sure if this can help to identify the issue?

yahskapar commented 11 months ago

Your dataset layout seems fine to me, try this config of yours that I modified below:

BASE: ['']
TOOLBOX_MODE: "train_and_test"
TEST:
  USE_LAST_EPOCH: False
  DATA:
    FS: 35
    DATASET: UBFC-PHYS
    DO_PREPROCESS: True                    # if first time, should be true
    DATA_FORMAT: NCDHW
    DATA_PATH: "./data/UBFC-Phys/raw_data"                     # Raw dataset path, need to be updated
    CACHED_PATH: "./data/UBFC-Phys/preprocessed_data"    # Processed dataset save path, need to be updated
    EXP_DATA_NAME: ""
    BEGIN: 0.0
    END: 1.0
    PREPROCESS:
      DATA_TYPE: ['Standardized'] 
      DATA_AUG: ['None']
      LABEL_TYPE: Standardized
      DO_CHUNK: True
      CHUNK_LENGTH: 210
      CROP_FACE:
        DO_CROP_FACE: True
        USE_LARGE_FACE_BOX: True
        LARGE_BOX_COEF: 1.5
        DETECTION:
          DO_DYNAMIC_DETECTION: True
          DYNAMIC_DETECTION_FREQUENCY : 35
          USE_MEDIAN_FACE_BOX: True    # This should be used ONLY if dynamic detection is used
      RESIZE:
        H: 72
        W: 72

All I did was remove the explicit file list path specification and changed the chunk length back to 210. Neither of those changes should matter, but I'm more curious if this would work on your end with these changes that make the situation the same as the original config. If it does work out with the above config, maybe there is a bug of some kind that I can start looking into soon when I have time.

Also, assuming you're using multi-processing, modify the multi_process_quota in the below line from 8 to 1:

https://github.com/ubicomplab/rPPG-Toolbox/blob/ff597bf8eb55c9db961709ec8b32fb142663fd78/dataset/data_loader/BaseLoader.py#L419

That will be a good sanity check to see if it's something related to the default allocation of multi-processing threads, which in this case might've only caused issues for you with UBFC-PHYS since the videos are quite longer than most other supported datasets (each UBFC-PHYS video is 3 minutes long).

yahskapar commented 11 months ago

Lastly, I should stress, if you're running into this issue even after trying the two recommendations in my previous reply, 1) double-check your compute resources and see if anything strange might be going on there even when running a single thread (maybe by just monitoring htop to begin with), and 2) please make sure you're trying this with the main version of the rPPG-Toolbox (e.g., not a heavily modified version of this toolbox that you might have locally or on another fork).

If you are testing this out on a heavily modified version of this toolbox as a part of your own codebase or some other fork, it will be very difficult to help you narrow the issue down and it's best you first test with the main, unaugmented version of the toolbox first.

Dylan-H-Wang commented 11 months ago

Thank you for the suggestions. I only tried to change the multi_process_quota from 8 to 1 and all outputs became normal. I guess there must be something wrong with the the multi-processing method.

yahskapar commented 11 months ago

There's definitely improvements we can make with multi-processing, especially with datasets that may require it on certain machines (e.g., UBFC-PHYS) due to their longer raw video lengths. We'll keep that in mind for future improvements where we may handle longer raw video lengths differently when using multi-processing.