Some questions about the train and valid.

[sunmer1rain]

I used PURE as the training set to train the EfficientPyhs model, using PURE's 0 to 0.8 parts as the training set and 0.8 to 1 parts as the validation set. Afterwards, I tested the performance of the model on these two parts using the Infer configuration file. Is it normal to find that the metrics of the validation set is better than the train set? Is there any problem with this?

[yahskapar]

Hi @sunmer1rain,

Can you clarify what exactly your testing set is? Are you effectively testing on either the train split or the val split of PURE using a model trained with the same train split and validated with the same val split? I'm a bit confused by why you would want to do this, aside from maybe as a very basic sanity check to see if your model learned something reasonable from training (which based on the metrics you've shown, appears to be the case) and if that generalizes to the validation set (which it appears to do so). I would guess that should also be evident from the loss plots generated by the toolbox, however.

To avoid just overfitting to test data and any additional influence of the validation set used for training, it's more meaningful to test with a held-out test set, for example by using 0.0 to 0.6 of PURE for training, 0.6 to 0.8 for validation, and then 0.8 to 1.0 for testing. I recommend giving that a try for your experiments.

[sunmer1rain]

Initially, we used PURE as both the training and validation set, and UBFC as the test set. The metrics obtained on UBFC were better than those reported in many papers, which led us to suspect that we might have made some mistakes. Therefore, we wanted to see how it performs on PURE and check if there is any data leakage. Later, we found that the metrics on the validation set were even higher than those on the training set, which is not consistent with our usual understanding (the metrics on the training set should generally be higher than those on the validation set).

[yahskapar]

@sunmer1rain,

I'm still a bit confused, why would data leakage be a suspicion when doing cross-dataset evaluation? Is that with the idea that somehow the same subjects appear in both datasets? I don't think that's the case at all from my usage of those two datasets and based on the dataset descriptions.

It may be common that metrics on the validation set should be higher on the training set, but this is very dependent on the methods and datasets involved, and it's very possible what you're observing is a good enough signal is learned such that validation set performance is comparable to training set performance. This is definitely within the realm of possibility with a dataset like PURE, especially given there's not a particularly large amount of diversity between the individual subjects themselves. The splits in this toolbox are done subject-wise for PURE, so there's no possibility of leakage between the training set and the validation set with respect to subject identities at least.

Also, what kind of result did you get on UBFC? Is that result with EfficientPhys as implemented in this toolbox, or did you make some further modifications for your experiments? If you made various modifications and are testing this all on a forked, modified version of this repo, it's worth re-doing any sanity-check experiments (e.g., between the training set and validation set) with a fresh branch without any of your changes.

[sunmer1rain]

Thank you for your response. I have conducted several tests in the past few days and discovered that the results were influenced by the effectiveness of the model and preprocessing. There is no issue of data leakage in the code itself, and I have clarified everything. I sincerely appreciate your helpful assistance.

[yahskapar]

Happy to try to help, and good to hear that you were able to justify the results. Feel free to let us know if you have any other questions or concerns regarding the toolbox. I'll go ahead and close this issue now.