search

ubicomplab / rPPG-Toolbox

rPPG-Toolbox: Deep Remote PPG Toolbox (NeurIPS 2023)
https://arxiv.org/abs/2210.00716
Other
442 stars 106 forks source link

Issue with output signals #240

Closed Vivaldi4F closed 7 months ago

Vivaldi4F commented 7 months ago

Hello

I tried to use the following config file on the PURE Dataset with the exact environment created by setup.sh When I tried to visualize the output of my model, it seems not to give the expected amplitude, even if the model was trained on the same Dataset I try to test it on. I also tried with several networks but in every case it gives me the same problem. Do you have any idea how to visualize the output signal properly?

Also how is your "sample_output.pickle" generated? What network and training configuration were used to generate it?

output_signal config.txt

girishvn commented 7 months ago

Hi @Vivaldi4F ,

The pickle generation function can be found in the BaseTrainer.py, and is used in the test function of the rPPG trainer files near the bottom (eg. TscanTrainer.py).

I don't remember exactly, but I'm pretty sure that the sample pickle was generate from either TSCAN or DeepPhys model trained on PURE and tested on UBFC-rPPG.

There are a couple things you can do to validate if things are working:

  1. Are the results / metrics you receive at test believable and similar to what is published?

2a. If so: Try applying manual magnitude normalization to the predicted and target signals (we may implement this as well). Though rPPG has been validated to provide robust HR metrics, variance in coloration / lighting / camera equipment make amplitude mappings often erroneous when the train and test sets are not of the same data distribution.

2b. If not: there is likely something wrong with either the pretrained model or the data you are using.

Vivaldi4F commented 7 months ago

Hi

Thank you for your response. I tried calculating the various metrics proposed on the toolbox and I had really low performances:

With PURE_DeepPhys.pth, I had: FFT MAE (FFT Label): 19.45511122881356 +/- 2.711587622827439 FFT RMSE (FFT Label): 28.501071763452934 +/- 235.98955351661155 FFT MAPE (FFT Label): 23.844100827017737 +/- 2.036397116447253 FFT Pearson (FFT Label): 0.06749050732314234 +/- 0.13215123124538533 FFT SNR (FFT Label): -12.267450627410605 +/- 1.1607222126067818 (dB)

PURE_DeepPhys_PURE_FFT_BlandAltman_DifferencePlot.pdf PURE_DeepPhys_PURE_FFT_BlandAltman_ScatterPlot.pdf

And with PURE_TSCAN.pth, I had: FFT MAE (FFT Label): 23.507018008474578 +/- 2.0163015579792094 FFT RMSE (FFT Label): 28.150359536613415 +/- 154.24583900262272 FFT MAPE (FFT Label): 34.81952293602054 +/- 2.806331060121648 FFT Pearson (FFT Label): 0.01683632463106777 +/- 0.13243446165788036 FFT SNR (FFT Label): -14.581422889969968 +/- 1.0891138319811409 (dB)

PURE_TSCAN_PURE_FFT_BlandAltman_DifferencePlot.pdf PURE_TSCAN_PURE_FFT_BlandAltman_ScatterPlot.pdf

But I succeeded to make it work running the following command: pip install torch==1.8.2 torchvision==0.9.2 torchaudio==0.8.2 --extra-index-url https://download.pytorch.org/whl/lts/1.8/cu111

My issue seemed to be linked with the pytorch version given by the setup.sh file