ecg_intervalrelated() output formatting

[Mitchellb16]

Just wondering if anyone could explain the rationale for the current formatting output by ecg_intervalrelated, being the HRV features (columns) are series of arrays of arrays of single floats.

This prevents some dataframe functionality, e.g. quick plotting with df.HRV_MeanNN.plot(), but I wanted to know if there are other benefits to it that I'm not seeing. Thanks!

[Mitchellb16]

This code will fix this problem in the meantime

# Define a function to apply to each cell of the DataFrame
def extract_value(cell):
    try:
        # Use np.squeeze to convert array of arrays to single value
        return np.squeeze(cell)
    except (ValueError, TypeError):
        # If unable to extract, return the original value
        return cell

# Apply the function to all cells in the DataFrame and drop columns that have NaN
analyze_epochs_clean = analyze_epochs.map(extract_value).dropna(axis=1)

[DominiqueMakowski]

I am not sure I understood what's the issue is, could you paste an example of what the output is currently and how it could be improved?

[Mitchellb16]

The dataframe that is returned by ecg_intervalrelated() has columns that are arrays of arrays, i.e. each cell is formatted like [[value]].

So when we call

type(analyze_epochs.HRV_MeanNN[0])

we get numpy.ndarray

but in the ECG_Rate_Mean column

type(analyze_epochs.ECG_Rate_Mean[0])

we get numpy.float64

Many functions want the data of a column to be formatted like the latter. For example, calling pandas plotting functions:

analyze_epochs.ECG_Rate_Mean.plot()

runs just fine, whereas

analyze_epochs.HRV_MeanNN.plot()

returns this error:

TypeError                                 Traceback (most recent call last)
Cell In[12], line 1
----> 1 analyze_epochs.HRV_MeanNN.plot()

File ~/anaconda3/envs/phys-env/lib/python3.12/site-packages/pandas/plotting/_core.py:1030, in PlotAccessor.__call__(self, *args, **kwargs)
   1027             label_name = label_kw or data.columns
   1028             data.columns = label_name
-> 1030 return plot_backend.plot(data, kind=kind, **kwargs)

File ~/anaconda3/envs/phys-env/lib/python3.12/site-packages/pandas/plotting/_matplotlib/__init__.py:71, in plot(data, kind, **kwargs)
     69         kwargs["ax"] = getattr(ax, "left_ax", ax)
     70 plot_obj = PLOT_CLASSES[kind](data, **kwargs)
---> 71 plot_obj.generate()
     72 plot_obj.draw()
     73 return plot_obj.result

File ~/anaconda3/envs/phys-env/lib/python3.12/site-packages/pandas/plotting/_matplotlib/core.py:499, in MPLPlot.generate(self)
    497 @final
    498 def generate(self) -> None:
--> 499     self._compute_plot_data()
    500     fig = self.fig
    501     self._make_plot(fig)

File ~/anaconda3/envs/phys-env/lib/python3.12/site-packages/pandas/plotting/_matplotlib/core.py:698, in MPLPlot._compute_plot_data(self)
    696 # no non-numeric frames or series allowed
    697 if is_empty:
--> 698     raise TypeError("no numeric data to plot")
    700 self.data = numeric_data.apply(type(self)._convert_to_ndarray)

TypeError: no numeric data to plot

[Mitchellb16]

So the goal would be to make all the columns look like the ECG_Rate_Mean column

Label   ECG_Rate_Mean   HRV_MeanNN  HRV_SDNN    HRV_SDANN1  HRV_SDNNI1  HRV_SDANN2  HRV_SDNNI2  HRV_SDANN5  HRV_SDNNI5  ... HRV_ShanEn  HRV_FuzzyEn HRV_MSEn    HRV_CMSEn   HRV_RCMSEn  HRV_CD  HRV_HFD HRV_KFD HRV_LZC Result
1   1   309.270961  [[194.01973684210526]]  [[11.692288281733303]]  [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] ... [[4.969829777788752]]   [[0.553501197143664]]   [[0.3023398153051495]]  [[0.5160015858082174]]  [[0.47375541520904296]] [[0.9551228546531801]]  [[1.4211529594205168]]  [[1.57705002598466]]    [[0.6576765803624827]]  1
2   2   331.799217  [[180.8048780487805]]   [[3.889488920027716]]   [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] ... [[4.465374986373309]]   [[0.6737756837362977]]  [[0.5565117746613248]]  [[0.6740184947096288]]  [[0.7788313298597798]]  [[1.3277555897307036]]  [[1.4958067765285579]]  [[1.9160438008808878]]  [[0.6977800980678385]]  1
3   3   338.470362  [[177.23493975903614]]  [[1.094156734540991]]   [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] ... [[2.9651779060178822]]  [[1.164101079947016]]   [[1.1193441620306803]]  [[1.093515223546829]]   [[1.3785313363717893]]  [[1.316759824479642]]   [[1.668715093844506]]   [[2.521786615224251]]   [[0.9216924479055795]]  1
4   4   338.595333  [[177.21428571428572]]  [[1.933044471664326]]   [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] ... [[3.6583790275546058]]  [[0.8101207469655183]]  [[1.180222840763136]]   [[0.898173376947552]]   [[0.8624968864578968]]  [[1.409783438421504]]   [[1.6399397761432095]]  [[1.9890208511328105]]  [[0.6848911524405815]]  1

looking at the source code, I believe this function in ecg_intervalrelated() is our culprit

def _ecg_intervalrelated_hrv(data, sampling_rate, output={}):

    # Sanitize input
    colnames = data.columns.values
    if len([i for i in colnames if "ECG_R_Peaks" in i]) == 0:
        raise ValueError(
            "NeuroKit error: ecg_intervalrelated(): Wrong input,"
            "we couldn't extract R-peaks. Please make sure"
            "your DataFrame contains an `ECG_R_Peaks` column."
        )

    # Transform rpeaks from "signal" format to "info" format.
    rpeaks = np.where(data["ECG_R_Peaks"].values)[0]
    rpeaks = {"ECG_R_Peaks": rpeaks}

    results = hrv(rpeaks, sampling_rate=sampling_rate)
    for column in results.columns:
        # Add and convert to float
        output[column] = results[[column]].values

    return output

I think the double brackets around results[[column]].values may be the problem. I can fork and test it out this afternoon.

[Mitchellb16]

fixed this! If you approve, I can open a PR.

Old:

df, info = nk2.ecg_process(data["ECG"], sampling_rate=100)
nk2.ecg_intervalrelated(df, sampling_rate=100)
epochs = nk2.epochs_create(df, events=[0, 15000], sampling_rate=100, epochs_end=150)
nk2.ecg_intervalrelated(epochs)

    Label   ECG_Rate_Mean   HRV_MeanNN  HRV_SDNN    HRV_SDANN1  HRV_SDNNI1  HRV_SDANN2  HRV_SDNNI2  HRV_SDANN5  HRV_SDNNI5  ... HRV_SampEn  HRV_ShanEn  HRV_FuzzyEn HRV_MSEn    HRV_CMSEn   HRV_RCMSEn  HRV_CD  HRV_HFD HRV_KFD HRV_LZC
1   1   86.389814   [[69.49767441860465]]   [[5.167181138835641]]   [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] ... [[1.2527629684953678]]  [[4.290505175709077]]   [[1.1667625041387144]]  [[1.3234302834078784]]  [[1.3340298186240032]]  [[1.5730504365078102]]  [[1.6535777692661076]]  [[1.8035044265520663]]  [[2.336829556302458]]   [[0.8288764443746864]]
2   2   86.394396   [[69.46046511627907]]   [[4.648089725942579]]   [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] [[nan]] ... [[1.8817856208857746]]  [[4.080221616903997]]   [[1.3473463161429127]]  [[1.7371494891482802]]  [[1.4976794053994398]]  [[1.9578060920316949]]  [[1.5369379755191352]]  [[1.8952794259141006]]  [[3.094110204759245]]   [[0.9730288694833277]]

New:

df, info = nk2_new.ecg_process(data["ECG"], sampling_rate=100)
nk2_new.ecg_intervalrelated(df, sampling_rate=100)
epochs = nk2_new.epochs_create(df, events=[0, 15000], sampling_rate=100, epochs_end=150)
nk2_new.ecg_intervalrelated(epochs)

Label   ECG_Rate_Mean   HRV_MeanNN  HRV_SDNN    HRV_SDANN1  HRV_SDNNI1  HRV_SDANN2  HRV_SDNNI2  HRV_SDANN5  HRV_SDNNI5  ... HRV_SampEn  HRV_ShanEn  HRV_FuzzyEn HRV_MSEn    HRV_CMSEn   HRV_RCMSEn  HRV_CD  HRV_HFD HRV_KFD HRV_LZC
1   1   86.389814   69.497674   5.167181    NaN NaN NaN NaN NaN NaN ... 1.252763    4.290505    1.166763    1.323430    1.334030    1.573050    1.653578    1.803504    2.33683 0.828876
2   2   86.394396   69.460465   4.648090    NaN NaN NaN NaN NaN NaN ... 1.881786    4.080222    1.347346    1.737149    1.497679    1.957806    1.536938    1.895279    3.09411 0.973029

I just changed the last line of _ecg_intervalrelated_hrv():

def _ecg_intervalrelated_hrv(data, sampling_rate, output={}):

    # Sanitize input
    colnames = data.columns.values
    if len([i for i in colnames if "ECG_R_Peaks" in i]) == 0:
        raise ValueError(
            "NeuroKit error: ecg_intervalrelated(): Wrong input,"
            "we couldn't extract R-peaks. Please make sure"
            "your DataFrame contains an `ECG_R_Peaks` column."
        )

    # Transform rpeaks from "signal" format to "info" format.
    rpeaks = np.where(data["ECG_R_Peaks"].values)[0]
    rpeaks = {"ECG_R_Peaks": rpeaks}

    results = hrv(rpeaks, sampling_rate=sampling_rate)
    for column in results.columns:
        # Add and convert to float
        output[column] = results[column].to_numpy()[0]

    return output