Cross-dataset testing

[Dream999999]

I want to use the PURE dataset for training, testing, and validation with the PhysNet network, without conducting cross-dataset testing. How do I modify my code?

[yahskapar]

Hi @Dream999999,

If you're referencing a config that already uses PURE for training and validation, this should be as straightforward as just modifying the config and setting the TEST.DATA.DATASET to PURE in the config and adjusting the TRAIN/VALID/TEST.DATA.BEGIN and TRAIN/VALID/TEST.DATA.END config options appropriately (e.g., make sure the BEGIN and END for the different splits doesn't overlap).

Starting from PURE_PURE_UBFC-rPPG_PHYSNET_BASIC.yaml with 60/20/20 train/val/test splits of PURE for example:

BASE: ['']
TOOLBOX_MODE: "train_and_test"  # "train_and_test"  or "only_test"
TRAIN:
  BATCH_SIZE: 4
  EPOCHS: 30
  LR: 9e-3
  MODEL_FILE_NAME: PURE_PURE_UBFC_physnet_diffnormalized
  PLOT_LOSSES_AND_LR: True
  DATA:
    FS: 30
    DATASET: PURE
    DO_PREPROCESS: False            # if first time, should be true
    DATA_FORMAT: NCDHW
    DATA_PATH: "/gscratch/ubicomp/xliu0/data3/mnt/Datasets/PURE/RawData"                     # Raw dataset path, need to be updated
    CACHED_PATH: "/gscratch/ubicomp/xliu0/data3/mnt/Datasets/rppg_toolbox/PreprocessedData"    # Processed dataset save path, need to be updated
    EXP_DATA_NAME: ""
    BEGIN: 0.0
    END: 0.6
    PREPROCESS:
      DATA_TYPE: ['DiffNormalized']         #if use physnet, should be DiffNormalized
      DATA_AUG: ['None']    # 'None' or 'Motion' is supported, used if the data path points to an augmented dataset or requires augmentation
      LABEL_TYPE: DiffNormalized
      DO_CHUNK: True
      CHUNK_LENGTH: 128                 #only support for factor of 512
      CROP_FACE:
        DO_CROP_FACE: True
        BACKEND: 'HC'    # HC for Haar Cascade, RF for RetinaFace
        USE_LARGE_FACE_BOX: True
        LARGE_BOX_COEF: 1.5
        DETECTION:
          DO_DYNAMIC_DETECTION: False
          DYNAMIC_DETECTION_FREQUENCY : 32
          USE_MEDIAN_FACE_BOX: False    # This should be used ONLY if dynamic detection is used
      RESIZE:
        H: 72
        W: 72
VALID:
  DATA:
    FS: 30
    DATASET: PURE
    DO_PREPROCESS: False                # if first time, should be true
    DATA_FORMAT: NCDHW
    DATA_PATH: "/gscratch/ubicomp/xliu0/data3/mnt/Datasets/PURE/RawData"                     # Raw dataset path, need to be updated
    CACHED_PATH: "/gscratch/ubicomp/xliu0/data3/mnt/Datasets/rppg_toolbox/PreprocessedData"    # Processed dataset save path, need to be updated
    EXP_DATA_NAME: ""
    BEGIN: 0.6
    END: 0.8
    PREPROCESS:
      DATA_TYPE: ['DiffNormalized']         #if use physnet, should be DiffNormalized
      DATA_AUG: ['None']    # 'None' or 'Motion' is supported, used if the data path points to an augmented dataset or requires augmentation
      LABEL_TYPE: DiffNormalized
      DO_CHUNK: True
      CHUNK_LENGTH: 128                 #only support for factor of 512
      CROP_FACE:
        DO_CROP_FACE: True
        BACKEND: 'HC'    # HC for Haar Cascade, RF for RetinaFace
        USE_LARGE_FACE_BOX: True
        LARGE_BOX_COEF: 1.5
        DETECTION:
          DO_DYNAMIC_DETECTION: False
          DYNAMIC_DETECTION_FREQUENCY : 32
          USE_MEDIAN_FACE_BOX: False    # This should be used ONLY if dynamic detection is used
      RESIZE:
        H: 72
        W: 72
TEST:
  METRICS: ['MAE', 'RMSE', 'MAPE', 'Pearson', 'SNR', 'BA']
  USE_LAST_EPOCH: True                   # to use provided validation dataset to find the best epoch, should be false
  DATA:
    FS: 30
    DATASET: PURE
    DO_PREPROCESS: False                  # if first time, should be true
    DATA_FORMAT: NCDHW
    DATA_PATH: "/gscratch/ubicomp/xliu0/data3/mnt/Datasets/PURE/RawData"                     # Raw dataset path, need to be updated
    CACHED_PATH: "/gscratch/ubicomp/xliu0/data3/mnt/Datasets/rppg_toolbox/PreprocessedData"    # Processed dataset save path, need to be updated
    EXP_DATA_NAME: ""
    BEGIN: 0.8
    END: 1.0
    PREPROCESS:
      DATA_TYPE: ['DiffNormalized']         #if use physnet, should be DiffNormalized
      LABEL_TYPE: DiffNormalized
      DO_CHUNK: True
      CHUNK_LENGTH: 128                 #only support for factor of 512
      CROP_FACE:
        DO_CROP_FACE: True
        BACKEND: 'HC'    # HC for Haar Cascade, RF for RetinaFace
        USE_LARGE_FACE_BOX: True
        LARGE_BOX_COEF: 1.5
        DETECTION:
          DO_DYNAMIC_DETECTION: False
          DYNAMIC_DETECTION_FREQUENCY : 32
          USE_MEDIAN_FACE_BOX: False    # This should be used ONLY if dynamic detection is used
      RESIZE:
        H: 72
        W: 72
DEVICE: cuda:0
NUM_OF_GPU_TRAIN: 1
LOG:
  PATH: runs/exp
MODEL:
  DROP_RATE: 0.2
  NAME: Physnet
  PHYSNET:
    FRAME_NUM: 128
INFERENCE:
  BATCH_SIZE: 4
  EVALUATION_METHOD: "FFT"        # "FFT" or "peak detection"
  EVALUATION_WINDOW:
    USE_SMALLER_WINDOW: False        # Change this if you'd like an evaluation window smaller than the test video length
    WINDOW_SIZE: 10        # In seconds
  MODEL_PATH: ""

Let us know if that doesn't work for some reason.

[Dream999999]

I am using the PURE dataset with the modifications mentioned above. However, when I run the preprocessing, the progress bar is moving, but an error occurs, as shown in the figure. What is going on? I also encounter similar errors with other datasets and configuration files. Thank you for your help.

[yahskapar]

@Dream999999,

My guess is there's something wrong with how the OpenCV library was installed, maybe with respect to your operating system environment. What OS are you using, and did you follow the setup instructions here exactly or did you have to do anything different / possibly specific to your OS?

Alternatively, there could be an issue with how the below relative path is interpreted on your specific OS (I'm guessing Windows based on the screenshot):

https://github.com/ubicomplab/rPPG-Toolbox/blob/cf0c09419b9436a9395111dbd05703146b3ab471/dataset/data_loader/BaseLoader.py#L281C1-L282C61

Try changing the above code in BaseLoader.py to what appears below:

detector = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')

You can also try os.path.join(cv2.data.haarcascades, 'haarcascade_frontalface_default.xml') instead of using + in case anything weird happens with OS-specific path behavior.

Let me know if that works, that's also probably a better way to do this anyhow so I can update the repo if that does work out for you. If somehow you get an error that cv2.data is missing, you may also have to try installing opencv-contrib-python using pip.

[Dream999999]

Alright, thank you for your suggestions. I will go check my code.

[Dream999999]

@Dream999999，

我猜想 OpenCV 库的安装方式有问题，可能与你的操作系统环境有关。你使用的是什么操作系统？你是否严格按照此处的安装说明进行操作，还是必须执行与你的操作系统不同或可能特定于操作系统的操作？

或者，下面的相对路径在您的特定操作系统上的解释方式可能存在问题（根据屏幕截图，我猜测是 Windows）：

https://github.com/ubicomplab/rPPG-Toolbox/blob/cf0c0941​​9b9436a9395111dbd05703146b3ab471/dataset/data_loader/BaseLoader.py#L281C1-L282C61

尝试将上面的代码更改为BaseLoader.py如下所示的内容：

detector = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')

您也可以尝试os.path.join(cv2.data.haarcascades, 'haarcascade_frontalface_default.xml')使用，+以防操作系统特定的路径行为发生任何异常。

如果可行，请告诉我，无论如何，这也可能是一种更好的方法，这样我就可以更新 repo，如果这对你有用的话。如果你以某种方式收到 cv2.data 缺失的错误，你可能还必须尝试使用​​ 进行opencv-contrib-python安装pip。

Thank you for your suggestion. I have modified the code according to your provided method: detector = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml'), and it is now running successfully.

[yahskapar]

Great! I'll keep this issue open if you don't mind as a reminder to myself to make this small update to the repo and in case you run into any other trouble related to running with train, val, and test splits all from one dataset (e.g., PURE).

All the best.

[Dream999999]

太棒了！如果您不介意的话，我会保留这个问题，以提醒自己对 repo 进行这个小更新，以防您遇到与从一个数据集（例如 PURE）运行训练、验证和测试分割相关的任何其他问题。

一切顺利。

I don't mind. You can make the question public. I hope it can also help others. Thank you, and I wish you a happy life.

[Dream999999]

太棒了！如果您不介意的话，我会保留这个问题，以提醒自己对 repo 进行这个小更新，以防您遇到与从一个数据集（例如 PURE）运行训练、验证和测试分割相关的任何其他问题。

一切顺利。

I apologize for disturbing you again. I have modified the configuration using the PURE dataset with a 6:2:2 split and trained it with the PhysNet network. However, I am uncertain about the quality of the results I've obtained, and there was no validation curve at the end of the training. Is this normal? Thank you for your response. FFT MAE (FFT Label):4.1015625+/- 3.5480277587079985 FFT RMSE (FFT Label):12.957037727048426 +/- 160.2670241107494 FFT MAPE (FFT Label):4.6209336696608565 +/- 3.9135825616273108 FFT Pearson (FFT Label):0.7609866371115069 +/- 0.20515831402546658 FFT SNR (FFT Label):9.957052759942334 +/- 2.0770124277876505(dB)

[yahskapar]

Hi @Dream999999,

Sorry for the belated reply (got too busy with work on my end). That looks like a reasonable training loss curve and corresponding metric results on first glance.

Regarding the absence of a validation loss curve, can you verify you're actually preprocessing and using a validation set? That means you should have the VALID portion of your config properly set (e.g., DO_PREPROCESS set to True the first time) and you should also have USE_LAST_EPOCH set to False. Feel free to also share your full terminal output while training so that we can verify if the validation set is even being loaded and used.

[Dream999999]

你好@Dream999999，

抱歉回复迟了（我这边工作太忙了）。乍一看，这看起来像是合理的训练损失曲线和相应的指标结果。

关于没有验证损失曲线，您能否验证您确实在预处理并使用验证集？这意味着您应该VALID正确设置配置的一部分（例如，DO_PREPROCESS设置为True第一次），并且还应该设置USE_LAST_EPOCH为False。在训练期间，请随意分享您的完整终端输出，以便我们可以验证验证集是否正在加载和使用。

Thank you very much for replying to my question. As I am just starting out, I still have a lot to learn and I apologize for any inconvenience caused. I have another question: I have not conducted cross-dataset testing, so the RMSE value of my experimental results is very unsatisfactory and the value is very large. Is this experiment result related to my use of only the PURE dataset? Please help me translate the above content into English.

[yahskapar]

Hi @Dream999999,

No worries, I'm happy to try and help.

I have not conducted cross-dataset testing, so the RMSE value of my experimental results is very unsatisfactory and the value is very large. Is this experiment result related to my use of only the PURE dataset?

Sorry, can you elaborate on this? Are you saying the MAE value of ~4.1 you shared before is too large, or are you referring to other results you haven't shared here yet? Regarding intra-dataset testing, or testing only within a dataset, I would try a few different architectures (e.g., not just PhysNet, try TS-CAN and unsupervised methods such as POS across your PURE test set as well) before drawing further conclusions.

[Dream999999]

你好@Dream999999，

不用担心，我很乐意尝试提供帮助。

我没有进行跨数据集测试，所以我的实验结果的RMSE值很不理想，数值很大，这个实验结果和我只使用PURE数据集有关系吗？

抱歉，您能详细说明一下吗？您是说您之前分享的 ~4.1 的 MAE 值太大，还是您指的是您尚未在此处分享的其他结果？关于数据集内测试，或仅在数据集内进行测试，我会尝试几种不同的架构（例如，不仅仅是 PhysNet，还可以尝试 TS-CAN 和无监督方法，例如 PURE 测试集中的 POS），然后再得出进一步的结论。

Thank you for your answer.I am currently working with the PhysNet and EfficientPhys networks. I have modified the configuration file and have not conducted cross-dataset testing. I am using the PURE dataset, which is split according to a 6:2:2 ratio. Upon completing the training and conducting tests, I found that the standard error values of the RMSE are quite large. However, when I use the pre-trained models provided in your code, the standard error values of the RMSE are smaller. I have included my test results from the PhysNet and EfficientPhys training below, with the RMSE values marked in green. These values seem unusually high to me, and I am wondering if there might be an issue with my settings that is causing this problem. ![Uploading 屏幕截图 2024-06-23 145137.png…]()

[yahskapar]

Hi @Dream999999,

That result doesn't look that unusual, and actually seems plausible given some of the additional results in section G of the appendix of our toolbox paper here. Note those are cross-dataset results, but still, there are also many cases there where there is a fairly high RMSE present. This is most likely because of some particularly bad prediction errors, which can be caused by a variety of things including noisy test videos, poor model training, and so on.

Out of curiosity, what are you trying to use this within dataset or intra-dataset result using only PURE for? Do you have some other performance measurement with a lower RMSE, which you're trying to reasonably get closer to? If you just want a sanity check for these values, I think they're reasonable based on what I observed in the past. If you want to debug the performance a bit more, you can also try to print out which test video has the worst test performance by making a few modifications to this toolbox's evaluation code.

[Dream999999]

你好@Dream999999，

这个结果看起来并不奇怪，而且考虑到我们工具箱论文附录 G 部分中的一些附加结果，这个结果似乎很有道理。请注意，这些是跨数据集的结果，但仍然有很多情况下存在相当高的 RMSE。这很可能是因为一些特别严重的预测误差，这些误差可能是由多种因素造成的，包括嘈杂的测试视频、糟糕的模型训练等等。

出于好奇，您试图在数据集内或仅使用 PURE 的数据集内结果中使用它来做什么？您是否有其他具有较低 RMSE 的性能测量，并且您想合理地接近它？如果您只是想对这些值进行健全性检查，我认为根据我过去的观察，它们是合理的。如果您想进一步调试性能，您还可以尝试通过对此工具箱的评估代码进行一些修改来打印出哪个测试视频的测试性能最差。

Thank you for your answer. I have just learned about this and have not conducted any other experiments or have relevant indicators. I will try some other tests.

[yahskapar]

Sounds good, I'll go ahead and close this issue since the original concerns were addressed + it's running a bit long. Feel free to make a new issue if you run into any other specific problem with the toolbox.

All the best!