Evaluate RR

[koutsd]

Hi, I'm wondering if it is possible to evaluate respiration rate using the pretrained models. Can I use a method like FFT similar to the one used for heart rate to get respiration rate out of the predicted signal?

[yahskapar]

Regarding the method to calculate video-level respiration rate, it's nearly identical to how it would be done for heart rate aside from the bandpass filter's cutoff frequencies being different (which corresponds to a normal range of respiration, I assume in adults). You can view the video-level respiration rate calculation code in bigsmall_multitask_metrics.py below:

https://github.com/ubicomplab/rPPG-Toolbox/blob/7e33a44b390414e0fc8a12f34b03896ed53d1b19/evaluation/bigsmall_multitask_metrics.py#L42-L65

As for doing this with pretrained models, which pretrained models are you referring to? Multi-task models, such as BigSmall, definitely can be used toward that end since respiration is a targeted task. I would guess other pre-trained models in this repo would not be able to get reasonable respiration rate results since they weren't trained in a way that explicitly targets predicting respiration rate (and may as a result rely on different cues). @girishvn, the lead author of BigSmall, likely will have some insight here from prior experience.

[koutsd]

Thank you for the quick reply. I was referring to non multi-task models. As for BigSmall is there an inference configuration file that I could use to test it out or is it similar to the existing ones for other models?

[yahskapar]

There doesn't seem to be an inference config from what I can see, but it should be similar to other existing inference configs. I don't have time to test this out right now unfortunately, but maybe try leveraging information (e.g., model info, batch size) from one of the BigSmall model training configs (e.g., here).

EDIT: I dug into this a little bit more and I think you'd have to change a fair bit more than I expected. Specifically, you'd have to write some code to leverage the call to calculate the respiration rate metrics (e.g., here) when using your custom inference config. Currently, it wouldn't be as simple as just trying to use another inference config with the BigSmall model specified and the corresponding pre-trained model loaded, partly because the inference-related elements there seem to be only contained in the BigSmall trainer.

[girishvn]

Hi @koutsd,

Like @yahskapar mentioned this toolkit doesn't explicitly enable respiration rate evaluation. Models like BigSmall will do this but for the multitask scenario.

If you are looking for a RR only model, I would suggest looking at the DeepPhys. MTTS-CAN, and BigSmall papers. All evaluate models, traditionally used for rPPG on the respiration task.

It should be do-able (but probably not trivial) to adapt this toolbox to the respiration task. I would take the following steps:

1. process video inputs in the same way as for rPPG
2. switch the signal used for labels from a pulse signal to a continuous respiration signal (maybe from a respiration band).
3. Adapt evaluation metric to use a more appropriate bandpass filter for the RR frequencies.

I would first try DeepPhys as it publishes resp evaluations. Unfortunately, this toolbox does not support any respiration datasets (other than BP4D+ which does process this signal for any model other than BigSmall which uses a slightly different training / data pipeline).

[koutsd]

Thanks a lot for the help. I was hoping to find some pre-trained models for respiration to start but i guess that is not the case. I will try training the models you mentioned following the steps.

[girishvn]

Best of luck! I'm closing this issue, but feel free to open it if something relevant pops up.

[koutsd]

Hi again,

Excuse my potential ignorance since I'm not very familiar with signal processing and PPG in general.

I have seen papers about extracting respiration rate from PPG signals. Do you think that those methods would be valid for the rPPG signal produced by the neural network models?

From what I understand from the trainer implementations, the NN models are trained using the PPG ground truth data as labels without applying any filter to remove the respiratory signal. So, shouldn't the predicted rPPG signal contain respiratory information?