Bad training results on MR-NIRP

[koutsd]

Hi,

I have been working on training DeepPhys and TSCAN on the MR-NIRP dataset. Howerver validation loss seems to remain quite high.

After plotting the predicted signal vs the ground truth it seems that the predicted one is between -1 and 1. I am wondering if the ground truth requires some additional preprocessing to because none of the models seems to be able to predict values outside [-1,1]

Here is the train config file. I am basically using the basic configuration similar to the other conigs. ` BASE: [''] TOOLBOX_MODE: "train_and_test" # "train_and_test" or "only_test" TRAIN: BATCH_SIZE: 4 EPOCHS: 30 LR: 9e-3 MODEL_FILE_NAME: MR-NIRP_TSCAN PLOT_LOSSES_AND_LR: True DATA: FILTERING: USE_EXCLUSION_LIST: True EXCLUSION_LIST: [

Face detection failed - Defective videos - Defective PPG signal:

      'subject7_driving_small_motion_975',

      'subject2_driving_still_940', 'subject7_driving_still_975', 'subject16_driving_still_940',

      'subject2_garage_small_motion_940', 'subject6_garage_small_motion_975',

      'subject5_garage_still_975', 'subject6_garage_still_975',

      # 'subject12'
    ]
    SELECT_TASKS: True
    TASK_LIST: ['subject1', 'subject2', 'subject3', 'subject4', 'subject5', 'subject6', 'subject7', 'subject8', 'subject9', 'subject10', 'subject11']
  FS: 30
  DATASET: MR-NIRP
  DO_PREPROCESS: True                    # if first time, should be true
  DATA_FORMAT: NDCHW
  DATA_PATH: "/mimer/NOBACKUP/groups/naiss2024-23-123/MR-NIRP"                     # Raw dataset path, need to be updated
  CACHED_PATH: "/mimer/NOBACKUP/groups/naiss2024-23-123/Dimitrios_Salvija/data/PreprocessedData"    # Processed dataset save path, need to be updated
  EXP_DATA_NAME: "MR-NIRP_DiffNormalized-Standardized_DiffNormalized"
  BEGIN: 0.0
  END: 1.0
  PREPROCESS:
    DATA_TYPE: [ 'DiffNormalized', 'Standardized' ]
    LABEL_TYPE: DiffNormalized
    DO_CHUNK: True
    CHUNK_LENGTH: 900
    CROP_FACE:
      DO_CROP_FACE: True
      BACKEND: 'RF'    # HC for Haar Cascade, RF for RetinaFace
      USE_LARGE_FACE_BOX: True
      LARGE_BOX_COEF: 1.5
      DETECTION:
        DO_DYNAMIC_DETECTION: False
        DYNAMIC_DETECTION_FREQUENCY : 30
        USE_MEDIAN_FACE_BOX: False    # This should be used ONLY if dynamic detection is used
    RESIZE:
      H: 72
      W: 72

VALID: DATA: FILTERING: USE_EXCLUSION_LIST: True EXCLUSION_LIST: [

Face detection failed - Defective videos - Defective PPG signal:

      'subject7_driving_small_motion_975',

      'subject2_driving_still_940', 'subject7_driving_still_975', 'subject16_driving_still_940',

      'subject2_garage_small_motion_940', 'subject6_garage_small_motion_975',

      'subject5_garage_still_975', 'subject6_garage_still_975',

      # 'subject12'
    ]
    SELECT_TASKS: True
    TASK_LIST: ['subject12', 'subject13', 'subject14', 'subject15']
  FS: 30
  DATASET: MR-NIRP
  DO_PREPROCESS: False                    # if first time, should be true
  DATA_FORMAT: NDCHW
  DATA_PATH: "/mimer/NOBACKUP/groups/naiss2024-23-123/MR-NIRP"                     # Raw dataset path, need to be updated
  CACHED_PATH: "/mimer/NOBACKUP/groups/naiss2024-23-123/Dimitrios_Salvija/data/PreprocessedData"    # Processed dataset save path, need to be updated
  EXP_DATA_NAME: "MR-NIRP_DiffNormalized-Standardized_DiffNormalized"
  BEGIN: 0.0
  END: 1.0
  PREPROCESS:
    DATA_TYPE: [ 'DiffNormalized', 'Standardized' ]
    LABEL_TYPE: DiffNormalized
    DO_CHUNK: True
    CHUNK_LENGTH: 900
    CROP_FACE:
      DO_CROP_FACE: True
      BACKEND: 'RF'    # HC for Haar Cascade, RF for RetinaFace
      USE_LARGE_FACE_BOX: True
      LARGE_BOX_COEF: 1.5
      DETECTION:
        DO_DYNAMIC_DETECTION: False
        DYNAMIC_DETECTION_FREQUENCY : 30
        USE_MEDIAN_FACE_BOX: False    # This should be used ONLY if dynamic detection is used
    RESIZE:
      H: 72
      W: 72

TEST: METRICS: ['MAE', 'RMSE', 'MAPE', 'Pearson', 'SNR', 'BA'] USE_LAST_EPOCH: False # to use provided validation dataset to find the best epoch, should be false DATA: FILTERING: USE_EXCLUSION_LIST: True EXCLUSION_LIST: [

Face detection failed - Defective videos - Defective PPG signal:

      'subject7_driving_small_motion_975',

      'subject2_driving_still_940', 'subject7_driving_still_975', 'subject16_driving_still_940',

      'subject2_garage_small_motion_940', 'subject6_garage_small_motion_975',

      'subject5_garage_still_975', 'subject6_garage_still_975',

      # 'subject12'
    ]
    SELECT_TASKS: True
    TASK_LIST: ['subject16', 'subject17', 'subject18', 'subject19']
  FS: 30
  DATASET: MR-NIRP
  DO_PREPROCESS: False                    # if first time, should be true
  DATA_FORMAT: NDCHW
  DATA_PATH: "/mimer/NOBACKUP/groups/naiss2024-23-123/MR-NIRP"                     # Raw dataset path, need to be updated
  CACHED_PATH: "/mimer/NOBACKUP/groups/naiss2024-23-123/Dimitrios_Salvija/data/PreprocessedData"    # Processed dataset save path, need to be updated
  EXP_DATA_NAME: "MR-NIRP_DiffNormalized-Standardized_DiffNormalized"
  BEGIN: 0.0
  END: 1.0
  PREPROCESS:
    DATA_TYPE: [ 'DiffNormalized', 'Standardized' ]
    LABEL_TYPE: DiffNormalized
    DO_CHUNK: True
    CHUNK_LENGTH: 900
    CROP_FACE:
      DO_CROP_FACE: True
      BACKEND: 'RF'    # HC for Haar Cascade, RF for RetinaFace
      USE_LARGE_FACE_BOX: True
      LARGE_BOX_COEF: 1.5
      DETECTION:
        DO_DYNAMIC_DETECTION: False
        DYNAMIC_DETECTION_FREQUENCY : 30
        USE_MEDIAN_FACE_BOX: False    # This should be used ONLY if dynamic detection is used
    RESIZE:
      H: 72
      W: 72

DEVICE: cuda:0 NUM_OF_GPU_TRAIN: 1 LOG: PATH: runs/exp MODEL: DROP_RATE: 0.2 NAME: Tscan TSCAN: FRAME_DEPTH: 10 INFERENCE: BATCH_SIZE: 4 EVALUATION_METHOD: "FFT" # "FFT" or "peak detection" EVALUATION_WINDOW: USE_SMALLER_WINDOW: True # Change this if you'd like an evaluation window smaller than the test video length WINDOW_SIZE: 10 # In seconds MODEL_PATH: "" `

[girishvn]

Hi @koutsd,

I don't think it is an issue that the predictions scale [-1, 1]. A lot of the NN methods struggle to retain the magnitude of the inputs, but retain the frequency content fine.

From what I see in this one plotted example, the ground truth waveform has a lot of additional high-frequency artifacts / features. I suggest applying either a smoothing filter or a low-pass filter, to your ground truth signals, and see if this helps promote better learning.

Thanks!

[koutsd]

Hi @girishvn,

I noticed that the bad results were correlated with the use of MSE loss as PHYSNET was performing fine. By changing to Negative Pearson loss the performance seems to be fixed. Do you have any explanation why this would be the case?

For reference here is the loss plot after changing the loss function:

[yahskapar]

Hi @koutsd,

My guess is that if MR-NIRP requires some additional considerations when normalizing and/or standardizing, training with the MSE loss will be much more difficult due to its sensitivity to scale. I think the need for additional care when normalizing and standardizing makes sense given the nature of MR-NIRP, especially parts of the dataset where, if I remember correctly, there are significant changes in lighting conditions and some subtle yet consistent motions.

Also, I'm going to go ahead and close this issue since it's been a few weeks and it seems like you were able to get better results as per the discussion on #244. Feel free to re-create a new issue or re-open this one if you feel a new discussion is needed.