Issue when having missing data (?)

[fedeAR24]

Hi,

we adapted the DemRawDataICU.m to analyze one of our ECG files, which has some segments of noisy data and also some missing values (NaNs). The Toolbox ran well, just as when used with the TestData. However, the .csv file with the Output data had no avgsqi values below the first row whose value was below the HRVparams.sqi.LowQualityThreshold. Also, the ECG and SQI plots obtained with Plot_SignalDetection_SQI(time, signal(:,1), jqrs_ann, sqijw'./100,'ECG')) did not correspond well in time. The SQI values were all 'compressed' to the beginning of the signal, and segments that looked clean had zero SQI. We tried to track the error, and we managed to 'solve it' by changing the outputs of the function ConvertRawDataToRRIntervals() as follows:

Original: function [t,rr,jqrs_ann,SQIjw, StartIdxSQIwindows_jw] = ConvertRawDataToRRIntervals(ECG_RawData ,HRVparams, subjectID)

New: function [t,rr,jqrs_ann,SQIjs, StartIdxSQIwindows_js] = ConvertRawDataToRRIntervals(ECG_RawData ,HRVparams, subjectID)

We basically matched the annotations with the SQI and the indexes of the windows. This solved both problems: the .csv file has now all the avgsgi values above the threshold, and the two plots correspond well in time (noisy or empty data segments have low SQI).

We are not submitting it as a pull request because we are not sure whether this is right for other types of analyses. Maybe the change has to be done somewhere else?

Best, Federico

[GiuliaDAP]

Hi Federico, thanks for bringing up this issue. Although, the solution you proposed is not recommended.

You replaced the output of a very sensitive signal quality algorithm with one that is less affected by noise. Therefore, you are having an output when the other one would say that is too noisy. It is recommended to use sqijw when excluding segments for the analysis. Furthermore, your fix might not work with other signals.

I will look into the plotting problem, while for the missing sqi on the results spreadsheets it was designed to be like that. The entire sequence of sqi is saved in a separate file with extension .sqijw

Is it possible for you to share the data that is giving the problem?

Giulia

[fedeAR24]

Hi Giulia,

thank you for your clear answer. I will undo the changes and I will use the function as it was. I will look for the entire sqi sequence in that file then. Regarding sharing the data, I need to ask for permission first. But I will let you know ASAP.

Federico

[GiuliaDAP]

You’re welcome. If you think your signal is clean, a better way to included more analyzed windows is to lower the rejection threshold based on SQI in the Initialization file: HRVparams.sqi.LowQualityThreshold

On Mar 1, 2019, at 3:31 PM, fedeAR24 notifications@github.com wrote:

Hi Giulia,

thank you for your clear answer. I will undo the changes and I will use the function as it was. I will look for the entire sqi sequence in that file then. Regarding sharing the data, I need to ask for permission first. But I will let you know ASAP.

Federico

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cliffordlab/PhysioNet-Cardiovascular-Signal-Toolbox/issues/10#issuecomment-468801832, or mute the thread https://github.com/notifications/unsubscribe-auth/ANeJrFnV38JX7jUG-KGd-fwm4Kt466QVks5vSY4ogaJpZM4bYnFH.

[fedeAR24]

Hi Giulia,

here you can find the data that gave me problems. My guess is that both the .csv file and the plot start having empty SQI values when the analyzed segment of the signal includes NaN values. sample.zip

Best, Federico

[GiuliaDAP]

Hi Federico,

it is really easy to solve your problem in two simple stapes:

1. make sure that your input ECG is in mV (as explained in the toolbox readme). Your data are in digital units, to convert them to mV you can divide by 500. Roughly speaking, the R-waves should have an amplitude of 0.5-1 mV, so you can guess the conversion factor if you don't know the exact one.
2. the SQI algorithm does not support missing data (i.e., NaNs), replace all the NaNs in your data with constant values, for example zeros.

That worked on my old data, and with the data you shared with me.

signal_new = signal/500; signal_new(isnan(signal_new)) = 0; [t,rr,jqrs_ann,SQIjw, StartIdxSQIwindows_jw] = ConvertRawDataToRRIntervals(signal_new,HRVparams, 'test_new'); Plot_SignalDetection_SQI(time, signal_new, jqrs_ann, SQIjw,'ECG')

[fedeAR24]

Hi Giulia,

thanks again for your helpful response!

-Federico