search

CoVital-Project / Spo2_evaluation

Python script to evaluation the correctness of SpO2 estimation algorithms
18 stars 6 forks source link

spo2 from video #33

Closed MAFEMV closed 3 years ago

MAFEMV commented 3 years ago

Hello, I'm trying to use this repo to extract spo2 from a short video, I'm not quite sure if I'm using it in the proper manner... I'm running the Lemonaca.py script in the src folder, where I changed the file-path to my video. Could you clarify the steps to run this repo, please? Thank you.

gianlucatruda commented 3 years ago

Hi @MAFEMV. I'll be honest, this project has been dormant for a long while now. I'm working off memory from a year ago and a bit of experimentation here.

Depending on what you're looking to do, this fork and associated paper may be more relevant to you.

I believe you are running the code correctly, but perhaps the path to your video file is wrong. I'm running /modelling/lamonaca_and_nemcova/main.py (the equivalent to Lemonaca.py, I think) from this refactored fork of the repo and am able to get an MP4 file to work.

Note: As far as I remember, the first step is to convert the video into a timeseries of pixel value averages (mean and std) over all 3 colour channels. From there, various algorithmic techniques are attempting to convert that into SpO2 estimates. You may need to pick through the code in spo2evaluation/modelling/lamonaca_and_nemcova to figure out when these steps actually happen.

This repo wasn't set up as a CLI for getting SpO2 estimates from single video files. It was a research project for doing so at scale by pulling batches of videos from an API. As a result, it's not very user friendly for the former use case.

This is the kind of output you can expect when running python3 main.py from within /Spo2_evaluation/modelling/lamonaca_and_nemcova in the refactored fork:

> py main.py
FPS 29.0
Frame: 0
Frame: 50
Frame: 100
Frame: 150
Frame: 200
Frame: 250
Frame: 300
Video length 12.0
timestamps [0.0, 0.034482758620689655, 0.06896551724137931, 0.10344827586206896, 0.13793103448275862, 0.1724137931034483, 0.20689655172413793, 0.2413793103448276, 0.27586206896551724, 0.3103448275862069, 0.3448275862068966, 0.3793103448275862, 0.41379310344827586, 0.4482758620689655, 0.4827586206896552, 0.5172413793103449, 0.5517241379310345, 0.5862068965517241, 0.6206896551724138, 0.6551724137931034, 0.6896551724137931, 0.7241379310344828, 0.7586206896551724, 0.7931034482758621, 0.8275862068965517, 0.8620689655172413, 0.896551724137931, 0.9310344827586207, 0.9655172413793104, 1.0, 1.0344827586206897, 1.0689655172413792, 1.103448275862069, 1.1379310344827587, 1.1724137931034482, 1.206896551724138, 1.2413793103448276, 1.2758620689655173, 1.3103448275862069, 1.3448275862068966, 1.3793103448275863, 1.4137931034482758, 1.4482758620689655, 1.4827586206896552, 1.5172413793103448, 1.5517241379310345, 1.5862068965517242, 1.6206896551724137, 1.6551724137931034, 1.6896551724137931, 1.7241379310344827, 1.7586206896551724, 1.793103448275862, 1.8275862068965518, 1.8620689655172413, 1.896551724137931, 1.9310344827586208, 1.9655172413793103, 2.0, 2.0344827586206895, 2.0689655172413794, 2.103448275862069, 2.1379310344827585, 2.1724137931034484, 2.206896551724138, 2.2413793103448274, 2.2758620689655173, 2.310344827586207, 2.3448275862068964, 2.3793103448275863, 2.413793103448276, 2.4482758620689653, 2.4827586206896552, 2.5172413793103448, 2.5517241379310347, 2.586206896551724, 2.6206896551724137, 2.6551724137931036, 2.689655172413793, 2.7241379310344827, 2.7586206896551726, 2.793103448275862, 2.8275862068965516, 2.8620689655172415, 2.896551724137931, 2.9310344827586206, 2.9655172413793105, 3.0, 3.0344827586206895, 3.0689655172413794, 3.103448275862069, 3.1379310344827585, 3.1724137931034484, 3.206896551724138, 3.2413793103448274, 3.2758620689655173, 3.310344827586207, 3.3448275862068964, 3.3793103448275863, 3.413793103448276, 3.4482758620689653, 3.4827586206896552, 3.5172413793103448, 3.5517241379310347, 3.586206896551724, 3.6206896551724137, 3.6551724137931036, 3.689655172413793, 3.7241379310344827, 3.7586206896551726, 3.793103448275862, 3.8275862068965516, 3.8620689655172415, 3.896551724137931, 3.9310344827586206, 3.9655172413793105, 4.0, 4.0344827586206895, 4.068965517241379, 4.103448275862069, 4.137931034482759, 4.172413793103448, 4.206896551724138, 4.241379310344827, 4.275862068965517, 4.310344827586207, 4.344827586206897, 4.379310344827586, 4.413793103448276, 4.448275862068965, 4.482758620689655, 4.517241379310345, 4.551724137931035, 4.586206896551724, 4.620689655172414, 4.655172413793103, 4.689655172413793, 4.724137931034483, 4.758620689655173, 4.793103448275862, 4.827586206896552, 4.862068965517241, 4.896551724137931, 4.931034482758621, 4.9655172413793105, 5.0, 5.0344827586206895, 5.068965517241379, 5.103448275862069, 5.137931034482759, 5.172413793103448, 5.206896551724138, 5.241379310344827, 5.275862068965517, 5.310344827586207, 5.344827586206897, 5.379310344827586, 5.413793103448276, 5.448275862068965, 5.482758620689655, 5.517241379310345, 5.551724137931035, 5.586206896551724, 5.620689655172414, 5.655172413793103, 5.689655172413793, 5.724137931034483, 5.758620689655173, 5.793103448275862, 5.827586206896552, 5.862068965517241, 5.896551724137931, 5.931034482758621, 5.9655172413793105, 6.0, 6.0344827586206895, 6.068965517241379, 6.103448275862069, 6.137931034482759, 6.172413793103448, 6.206896551724138, 6.241379310344827, 6.275862068965517, 6.310344827586207, 6.344827586206897, 6.379310344827586, 6.413793103448276, 6.448275862068965, 6.482758620689655, 6.517241379310345, 6.551724137931035, 6.586206896551724, 6.620689655172414, 6.655172413793103, 6.689655172413793, 6.724137931034483, 6.758620689655173, 6.793103448275862, 6.827586206896552, 6.862068965517241, 6.896551724137931, 6.931034482758621, 6.9655172413793105, 7.0, 7.0344827586206895, 7.068965517241379, 7.103448275862069, 7.137931034482759, 7.172413793103448, 7.206896551724138, 7.241379310344827, 7.275862068965517, 7.310344827586207, 7.344827586206897, 7.379310344827586, 7.413793103448276, 7.448275862068965, 7.482758620689655, 7.517241379310345, 7.551724137931035, 7.586206896551724, 7.620689655172414, 7.655172413793103, 7.689655172413793, 7.724137931034483, 7.758620689655173, 7.793103448275862, 7.827586206896552, 7.862068965517241, 7.896551724137931, 7.931034482758621, 7.9655172413793105, 8.0, 8.03448275862069, 8.068965517241379, 8.10344827586207, 8.137931034482758, 8.172413793103448, 8.206896551724139, 8.241379310344827, 8.275862068965518, 8.310344827586206, 8.344827586206897, 8.379310344827585, 8.413793103448276, 8.448275862068966, 8.482758620689655, 8.517241379310345, 8.551724137931034, 8.586206896551724, 8.620689655172415, 8.655172413793103, 8.689655172413794, 8.724137931034482, 8.758620689655173, 8.793103448275861, 8.827586206896552, 8.862068965517242, 8.89655172413793, 8.931034482758621, 8.96551724137931, 9.0, 9.03448275862069, 9.068965517241379, 9.10344827586207, 9.137931034482758, 9.172413793103448, 9.206896551724139, 9.241379310344827, 9.275862068965518, 9.310344827586206, 9.344827586206897, 9.379310344827585, 9.413793103448276, 9.448275862068966, 9.482758620689655, 9.517241379310345, 9.551724137931034, 9.586206896551724, 9.620689655172415, 9.655172413793103, 9.689655172413794, 9.724137931034482, 9.758620689655173, 9.793103448275861, 9.827586206896552, 9.862068965517242, 9.89655172413793, 9.931034482758621, 9.96551724137931, 10.0, 10.03448275862069, 10.068965517241379, 10.10344827586207, 10.137931034482758, 10.172413793103448, 10.206896551724139, 10.241379310344827, 10.275862068965518, 10.310344827586206, 10.344827586206897, 10.379310344827585, 10.413793103448276, 10.448275862068966, 10.482758620689655, 10.517241379310345, 10.551724137931034, 10.586206896551724, 10.620689655172415, 10.655172413793103, 10.689655172413794, 10.724137931034482, 10.758620689655173, 10.793103448275861, 10.827586206896552, 10.862068965517242, 10.89655172413793, 10.931034482758621, 10.96551724137931, 11.0, 11.03448275862069, 11.068965517241379, 11.10344827586207, 11.137931034482758, 11.172413793103448, 11.206896551724139, 11.241379310344827, 11.275862068965518, 11.310344827586206, 11.344827586206897, 11.379310344827585, 11.413793103448276, 11.448275862068966, 11.482758620689655, 11.517241379310345, 11.551724137931034, 11.586206896551724, 11.620689655172415, 11.655172413793103, 11.689655172413794, 11.724137931034482, 11.758620689655173, 11.793103448275861, 11.827586206896552, 11.862068965517242, 11.89655172413793, 11.931034482758621, 11.96551724137931]
five sec 145
Midddle 174  from  29  to  319
290
289
289
289
Estimation
Getting the peaks
finding the peaks 43.5
[ 2.10344828  3.17241379  4.20689655  5.27586207  6.27586207  7.37931034
  8.44827586  9.5862069  10.62068966]
[1.06896552 1.03448276 1.06896552 1.         1.10344828 1.06896552
 1.13793103 1.03448276]
heart beat in s  1.0646551724137931  and in minutes  56.35627530364372
62.00000000000001
59.99999999999999
61.999999999999986
58.0
64.00000000000003
62.000000000000036
65.99999999999997
60.00000000000004
peaks [0.9962045  0.9977819  0.98068846 0.96122764 0.93649623 0.94848953
 0.93710383 0.94910316 0.95644423]
bottom [0.95721753 0.94203057 0.93967944 0.91206527 0.89731509 0.88693582
 0.87734978 0.88978765 0.90160571]
timestamps [ 2.10344828  3.17241379  4.20689655  5.27586207  6.27586207  7.37931034
  8.44827586  9.5862069  10.62068966]
Found 9 in the signal
m  [0.03898696 0.05575133 0.04100903 0.04916237 0.03918114 0.06155372
 0.05975405 0.05931551 0.05483852]
duration [0.82758621 0.89655172 0.48275862 0.55172414 0.72413793 0.65517241
 0.72413793 0.72413793 0.65517241]
height [0.03898696 0.05575133 0.04100903 0.04916237 0.03918114 0.06155372
 0.05975405 0.05931551 0.05483852]
slope in rad [0.04707444 0.06210421 0.08474383 0.08887208 0.05405458 0.09367544
 0.08233097 0.08172943 0.08350626]
slope in def [2.69716693 3.5583091  4.85546358 5.09199483 3.09709918 5.36720764
 4.71721708 4.68275163 4.78455605]
finding the peaks 43.5
[ 2.10344828  3.17241379  4.24137931  5.31034483  6.31034483  7.37931034
  8.44827586  9.5862069  10.62068966]
[1.06896552 1.06896552 1.06896552 1.         1.06896552 1.06896552
 1.13793103 1.03448276]
heart beat in s  1.0646551724137931  and in minutes  56.35627530364372
62.00000000000001
61.999999999999986
62.000000000000036
58.0
61.999999999999986
62.000000000000036
65.99999999999997
60.00000000000004
peaks [0.99580522 0.99699325 0.99361667 0.99126918 0.98577317 0.9899837
 0.98701982 0.98502857 0.9867609 ]
bottom [0.98513953 0.98259566 0.98319006 0.97773679 0.9759608  0.97661536
 0.97494546 0.9715725  0.9742019 ]
timestamps [ 2.10344828  3.17241379  4.24137931  5.31034483  6.31034483  7.37931034
  8.44827586  9.5862069  10.62068966]
Found 9 in the signal
m  [0.01066569 0.01439758 0.01042661 0.01353238 0.00981237 0.01336834
 0.01207435 0.01345606 0.012559  ]
duration [0.82758621 0.89655172 0.5862069  0.82758621 0.75862069 0.55172414
 0.55172414 0.79310345 0.65517241]
height [0.01066569 0.01439758 0.01042661 0.01353238 0.00981237 0.01336834
 0.01207435 0.01345606 0.012559  ]
slope in rad [0.01288699 0.01605746 0.0177847  0.01635017 0.01293376 0.02422538
 0.02188127 0.01696471 0.01916665]
slope in def [0.73837013 0.92002494 1.01898807 0.93679569 0.74104998 1.38801199
 1.25370463 0.97200645 1.09816824]
Formulation
mult: [2.69716693+0.j 3.5583091 +0.j 4.85546358+0.j 5.09199483+0.j
 3.09709918+0.j 5.36720764+0.j 4.71721708+0.j 4.68275163+0.j
 4.78455605+0.j]  *  [-3.24452798+0.j -2.88685399+0.j -3.19396303+0.j -3.0126268 +0.j
 -3.2395598 +0.j -2.787845  +0.j -2.81751829+0.j -2.82488442+0.j
 -2.90336235+0.j]
mres: [ -8.75103356+0.j -10.27231883+0.j -15.50817114+0.j -15.34028007+0.j
 -10.033238  +0.j -14.96294299+0.j -13.2908454 +0.j -13.22823214+0.j
 -13.89129992+0.j]
sqrt: [0.+2.95821459j 0.+3.20504584j 0.+3.93804154j 0.+3.91666696j
 0.+3.16752869j 0.+3.86819635j 0.+3.64566118j 0.+3.63706367j
 0.+3.72710342j]
[0.+42.14858564j 0.+45.6713933j  0.+56.30409194j 0.+56.0935084j
 0.+45.20680157j 0.+55.07726289j 0.+51.87634656j 0.+51.96286083j
 0.+53.18307908j]  /  [0.+34.90519054j 0.+37.81317935j 0.+46.32022386j 0.+45.99748062j
 0.+37.31798672j 0.+45.67000924j 0.+43.06692567j 0.+42.80857696j
 0.+43.91803624j]
Before optimization 1.2109621383527966 56.35627530364372
6.66854166666667   8.457663845486117   3.5270735677083334
Done: 94.48537559743244  and hr  56.35627530364372

That very last line is, I believe, and estimate of the SpO2 and heart rate values.

I hope that helps you somewhat!

MAFEMV commented 3 years ago

Hello @gianlucatruda. Thank you for your response. I will try what you're are sugesting. I been trying to figure out from your code what should be the next steps if I want to extract a spo2 measurement, if I already have the mean and std form the RGB channels,but as you sayd, there is a lot of scripts and fuctions there to follow the track... I been studying the papers related, including the one from Lamonaca, but is not clear for me what are the math operations once I have the mean and std form the RGB channels and their respective spectrum... Do you know where can I found this steps explained better? Thank you so much for your help.

MalcolmMielle commented 3 years ago

@MAFEMV do you have any problem when running Lemonaca.py and changing the video file?

AFAIRemember, which was a long time ago, just running 

nemcova = nemcova_2020.Nemcova2020()
o2, hr = nemcova.spo2_estimation("Video/S87T78.mp4", optimize=True)

with your video file should return the spo2 and heart rate measurements from the video file. To do Lemonaca maybe do somethinf along the lines of 

lamonaca = lamonaca_2015.Lamonaca2015()
o2, hr = lamonaca.lacomana("Video/S98T89.avi")

The algorithms are implemented in lamonaca and nemcova. The method to calculate the spo2 from the mean signal can be found in this file in def spo2_estimation(ppg_green_940, ppg_red_600, timestamps, fps): where it's quite literally the steps described in the paper :).

Hope that helps

MAFEMV commented 3 years ago

Hello @gianlucatruda. Yes, I had a problem running /modelling/lamonaca_and_nemcova/main.py (the equivalent to Lemonaca.py) from the fork that you suggest. I'm getting this:

FPS 30 Video length 0.0 timestamps [] five sec 150 Midddle 0 from 0 to -1 Traceback (most recent call last): File "main.py", line 18, in main() File "main.py", line 13, in main o2, hr = nemcova.spo2_estimation("home/mafe/Desacargas/id1/VID_20210526_184811133.mp4", optimize=True) File "/home/mafe/Repos/Spo2-2/Spo2_evaluation/modelling/lamonaca_and_nemcova/nemcova_2020.py", line 222, in spo2_estimation ppg_green_filtered_normalized, ppg_full_green, ppg_red_filtered_normalized, ppg_full_red, ppg_full_blue, duration, timestamps, fps = self.ppg_filtered_from_video_file( File "/home/mafe/Repos/Spo2-2/Spo2_evaluation/modelling/lamonaca_and_nemcova/nemcova_2020.py", line 153, in ppg_filtered_from_video_file green_absolute_max = max([abs(i) for i in ppg_full_green]) ValueError: max() arg is an empty sequence

I've tried with 3 different short videos (20s length at 30fpm .mp4) but still getting the same result. I don't understand why is showing : Video length 0.0... When I tried with a video .avi I obtained the same result and also the following error:

[ERROR:0] global /tmp/pip-req-build-ms668fyv/opencv/modules/videoio/src/cap.cpp (140) open VIDEOIO(CV_IMAGES): raised OpenCV exception:

OpenCV(4.5.1) /tmp/pip-req-build-ms668fyv/opencv/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): /home/mafe/Desacargas/id1/alex/alex_resting/cv_camera_sensor_stream_handler.avi in function 'icvExtractPattern'

Any way I'm going to try to follow your scripts and study the algorithm Thank you very much for all your help.

MalcolmMielle commented 3 years ago

As a ten-second wild guess, did you compile opencv with the correct flag to handle those video formats?

MAFEMV commented 3 years ago

Hi @MalcolmMielle yes, I've been using opencv for both formats in other script and it runs fine...

gianlucatruda commented 3 years ago

@MAFEMV I'm not sure what that OpenCV error means. I only ever tested the code on my macOS environment and working with MP4 files.

You mention getting this output:

FPS 30
Video length 0.0
timestamps []
five sec 150
Midddle 0 from 0 to -1
Traceback (most recent call last):
File "main.py", line 18, in
main()
File "main.py", line 13, in main
o2, hr = nemcova.spo2_estimation("home/mafe/Desacargas/id1/VID_20210526_184811133.mp4", optimize=True)
File "/home/mafe/Repos/Spo2-2/Spo2_evaluation/modelling/lamonaca_and_nemcova/nemcova_2020.py", line 222, in spo2_estimation
ppg_green_filtered_normalized, ppg_full_green, ppg_red_filtered_normalized, ppg_full_red, ppg_full_blue, duration, timestamps, fps = self.ppg_filtered_from_video_file(
File "/home/mafe/Repos/Spo2-2/Spo2_evaluation/modelling/lamonaca_and_nemcova/nemcova_2020.py", line 153, in ppg_filtered_from_video_file
green_absolute_max = max([abs(i) for i in ppg_full_green])
ValueError: max() arg is an empty sequence

I got exactly the same error when I was pointing to an invalid path. Check that the file is reachable from the current directory with the path you have supplied to the main.py file. E.g. just try ls <path>. Also try a relative path instead of an absolute one.

MAFEMV commented 3 years ago

Hi @gianlucatruda , you were right, thank you

Do you know if it is normal to obtain different lengths for the vectors of red and green signal?, because I'm receiving the assertion len(vp_940)==len(vp_600)

I forced it to work shortening the biggest vector, but I'm not sure if this has much influence in the final result of spo2

Thank you again for your help

gianlucatruda commented 3 years ago

@MAFEMV I'm glad that helped!

Do you know if it is normal to obtain different lengths for the vectors of red and green signal?

I don't recall having anything like that. It seems unlikely, given that they both come from the same video file.