InvalidIndexError for overlapping indices in xforms.average_away_stim_occurrences

[jacobpennington]

Getting this error for about 90% of batch 289 cells. The final call in our repository (i.e. before it starts going into pandas files) is line 123 of nems.preprocessing:

    # build a series with an interval index, to lookup where epochs fall
    s_name = pd.Series(epoch_stims['name'].values, pd.IntervalIndex.from_arrays(epoch_stims['start'], epoch_stims['end'], closed='both'))
    s_cat = pd.Series(np.arange(len(epoch_stims['start']), dtype='int'), pd.IntervalIndex.from_arrays(epoch_stims['start'], epoch_stims['end'], closed='both'))

    # add helper columns using the interval index lookups
    temp_epochs['cat'] = temp_epochs['start'].map(s_cat)   # line 123

I've tried updating pandas/numpy but it didn't help. Is anyone else running into this issue, or did anyone change that preprocessing function recently that might have an idea how to fix it?

[nems.xforms INFO] Evaluating: nems.xforms.split_by_occurrence_counts
[nems.xforms INFO] Evaluating: nems.xforms.average_away_stim_occurrences
Traceback (most recent call last):

  File "<ipython-input-6-2bedf608a226>", line 1, in <module>
    save_resp_stats(289, cells, spont_path=spont_289)

  File "/auto/users/jacob/env_nems/nems_db/nems_lbhb/gcmodel/figures/respstats.py", line 117, in save_resp_stats
    df = rate_by_batch(batch, cells=cells, stat='spont', fs=fs)

  File "/auto/users/jacob/env_nems/nems_db/nems_lbhb/gcmodel/figures/respstats.py", line 48, in rate_by_batch
    xfspec, ctx = xhelp.load_model_xform(cellid, batch, ln_dexp3)

  File "/auto/users/jacob/bin/anaconda3/envs/jacob_nems/nems/nems/xform_helper.py", line 279, in load_model_xform
    only=only)

  File "/auto/users/jacob/bin/anaconda3/envs/jacob_nems/nems/nems/xforms.py", line 1306, in load_analysis
    ctx, log_xf = evaluate(xfspec, ctx)

  File "/auto/users/jacob/bin/anaconda3/envs/jacob_nems/nems/nems/xforms.py", line 157, in evaluate
    context = evaluate_step(xfa, context)

  File "/auto/users/jacob/bin/anaconda3/envs/jacob_nems/nems/nems/xforms.py", line 114, in evaluate_step
    new_context = fn(**args)

  File "/auto/users/jacob/bin/anaconda3/envs/jacob_nems/nems/nems/xforms.py", line 610, in average_away_stim_occurrences
    val = preproc.average_away_epoch_occurrences(val, epoch_regex=epoch_regex)

  File "/auto/users/jacob/bin/anaconda3/envs/jacob_nems/nems/nems/preprocessing.py", line 123, in average_away_epoch_occurrences
    temp_epochs['cat'] = temp_epochs['start'].map(s_cat)

  File "/auto/users/jacob/bin/anaconda3/envs/jacob_nems/lib/python3.7/site-packages/pandas/core/series.py", line 3631, in map
    new_values = super()._map_values(arg, na_action=na_action)

  File "/auto/users/jacob/bin/anaconda3/envs/jacob_nems/lib/python3.7/site-packages/pandas/core/base.py", line 1130, in _map_values
    indexer = mapper.index.get_indexer(values)

  File "/auto/users/jacob/bin/anaconda3/envs/jacob_nems/lib/python3.7/site-packages/pandas/core/indexes/interval.py", line 794, in get_indexer
    "cannot handle overlapping indices; "

InvalidIndexError: cannot handle overlapping indices; use IntervalIndex.get_indexer_non_unique

np.__version__
Out[7]: '1.18.1'

pd.__version__
Out[8]: '1.0.0'

[arrrobase]

This is some code I updated to dramatically speed up the preprocessing. It's likely I introduced some bugs that weren't covered under the existing tests.

The error seems to indicate that the epochs that are being extracted to average (the epoch_regex epochs) overlap. It's coded to be able to handle identical start/end times among epochs. Do your epochs your averaging overlap more than that?

[jacobpennington]

Trying to figure that out, I don't see any overlapping but I might be missing something. Here is a minimal code example that recreates the bug for me:

import nems.xform_helper as xhelp

batch = 289
cellid = 'gus018b-a1'
modelname = 'ozgf.fs100.ch18-ld-sev_dlog.f-wc.18x3.g-fir.3x15-lvl.1-dexp.1_init-basic'

xfspec, ctx = xhelp.load_model_xform(cellid, batch, modelname)

I run that with import pdb; pdb.set_trace() added before line 123 of nems.preprocessing. The first pass (for the estimation data) doesn't return an error, so i enter continue to proceed to the second pass for the validation data. On that one I get the error. This is s_cat at that point:

ipdb> s_cat
[5.51, 11.01]          0
[44.08, 49.58]         1
[104.69, 110.19]       2
[121.22, 126.72]       3
[154.28, 159.78]       4
[187.34, 192.84]       5
[198.36, 203.86]       6
[203.87, 209.37]       7
[225.91, 231.41]       8
[264.48, 269.98]       9
[275.5, 281.0]        10
[303.05, 308.55]      11
[341.62, 347.12]      12
[352.64, 358.14]      13
[391.21, 396.71]      14
[418.76, 424.26]      15
[451.82, 457.32]      16
[468.35, 473.85]      17
[473.86, 479.36]      18
[517.94, 523.44]      19
[534.47, 539.97]      20
[545.49, 550.99]      21
[562.02, 567.52]      22
[622.63, 628.13]      23
[628.13, 633.63]      24
[672.21, 677.71]      25
[688.74, 694.24]      26
[710.78, 716.28]      27
[716.29, 721.79]      28
[721.8, 727.3]        29
[738.33, 743.83]      30
[743.84, 749.34]      31
[760.37, 765.87]      32
[776.9, 782.4]        33
[798.94, 804.44]      34
[804.45, 809.95]      35
[859.55, 865.05]      36
[881.59, 887.09]      37
[1008.32, 1013.82]    38
[1063.42, 1068.92]    39
[1151.58, 1157.08]    40
[1179.13, 1184.63]    41
[1184.64, 1190.14]    42
[1223.21, 1228.71]    43
[1234.23, 1239.73]    44
[1245.25, 1250.75]    45
dtype: int64

[arrrobase]

Can you test the latest version awctomlinson/epoch-speedup and see if that solves it? There's a bit of extra handling for overlapping epochs there that hasn't been merged into master yet.

[jacobpennington]

Yep that fixes it. So whatever you did in there did the trick.

[arrrobase]

Okay cool, I'll merge this into master.

[jacobpennington]

Great, thanks