The lack of skin segmentation processing in unsupervised methods.

[myhomeboughtazoo]

Dear Developer,

I have recently been trying to fully understand the complete process of signal extraction and evaluation carried out by the classic rPPG method from your project. I admire the grand scale of your project. However, I found something confusing in the code. It seems that the unsupervised method does not use skin segmentation, which is mentioned in many literatures.

To be more specific, for the UBFC-rPPG dataset, main.py directly loads the dataset in unsupervised mode, and the dataset uses opencv or retinaface for face detection. The detected and cropped faces are directly saved and input into the unsupervised rPPG method. The rPPG method directly uses the mean of the video as the original RGB signal, which seems different from what their original paper mentioned. (Although skin segmentation often does not bring improvement)

I would like to know if this is my oversight for not noticing the part of skin segmentation, or if this function is still being implemented. Thank you!

[yahskapar]

Hi @myhomeboughtazoo,

@xliucs may be able to add some perspective as to why skin segmentation was not included.

My understanding is that this was intentionally not implemented. It should be noted that a reliable face detection and cropping step itself is both more efficient and reliable in most cases. Such an approach avoids having to deal with error from more traditional segmentation methods without fine-tuning, though maybe this does not matter as much anymore with some really amazing, more recent segmentation models (e.g., SAM). Also, in some approaches (e.g., DeepPhys, TS-CAN), the attention mechanism itself can act as a form of skin segmentation.

If you happen to use skin segmentation in your experiments and find it to make a positive difference with even the simpler unsupervised baselines, let us know and we can try to take a deeper look into adding that as a feature selectable via a config parameter. Maybe you can spearhead the skin segmentation feature itself and contribute back to the toolbox, given it seems useful to have from your experiments.

[myhomeboughtazoo]

Thank you very much for your reply. @yahskapar

In fact, I found from experiments with multiple datasets that as long as stable face detection is ensured, skin segmentation does not seem to bring effective performance improvement. Considering that many literatures have mentioned the skin segmentation model, I suspected that there was a problem with my skin segmentation code at that time.

I may conduct more experiments in related directions in the future. If there is an accurate result, I will try to contribute as much as I can to this project.

[yahskapar]

Sounds good, thank you!