new version of neo (?) breaks read_abf

[DGalt]

I think the issue is with neo, as that's what makes the most sense. Just updated everything (conda, anaconda, neo), and am getting this exception:

Exception                                 Traceback (most recent call last)
<ipython-input-3-7e6a005709a2> in <module>()
      1 filepath = r'E:\Users\Dan\Desktop\17111016.abf'
----> 2 df = read_abf(filepath)
      3 plot_all(df)

C:\Anaconda3\lib\site-packages\neurphys-0.0-py3.5.egg\neurphys\read_abf.py in read_abf(filepath)
     45         time = seg.analogsignals[0].times - seg.analogsignals[0].times[0]
     46         data_dict['time'] = time
---> 47         df = pd.DataFrame(data_dict)
     48         df_list.append(df)
     49         sweep_list.append('sweep' + str(seg_num + 1).zfill(3))

C:\Anaconda3\lib\site-packages\pandas\core\frame.py in __init__(self, data, index, columns, dtype, copy)
    225                                  dtype=dtype, copy=copy)
    226         elif isinstance(data, dict):
--> 227             mgr = self._init_dict(data, index, columns, dtype=dtype)
    228         elif isinstance(data, ma.MaskedArray):
    229             import numpy.ma.mrecords as mrecords

C:\Anaconda3\lib\site-packages\pandas\core\frame.py in _init_dict(self, data, index, columns, dtype)
    360 
    361         return _arrays_to_mgr(arrays, data_names, index, columns,
--> 362                               dtype=dtype)
    363 
    364     def _init_ndarray(self, values, index, columns, dtype=None,

C:\Anaconda3\lib\site-packages\pandas\core\frame.py in _arrays_to_mgr(arrays, arr_names, index, columns, dtype)
   5095 
   5096     # don't force copy because getting jammed in an ndarray anyway
-> 5097     arrays = _homogenize(arrays, index, dtype)
   5098 
   5099     # from BlockManager perspective

C:\Anaconda3\lib\site-packages\pandas\core\frame.py in _homogenize(data, index, dtype)
   5409                 v = lib.fast_multiget(v, oindex.values, default=NA)
   5410             v = _sanitize_array(v, index, dtype=dtype, copy=False,
-> 5411                                 raise_cast_failure=False)
   5412 
   5413         homogenized.append(v)

C:\Anaconda3\lib\site-packages\pandas\core\series.py in _sanitize_array(data, index, dtype, copy, raise_cast_failure)
   2838     elif subarr.ndim > 1:
   2839         if isinstance(data, np.ndarray):
-> 2840             raise Exception('Data must be 1-dimensional')
   2841         else:
   2842             subarr = _asarray_tuplesafe(data, dtype=dtype)

Exception: Data must be 1-dimensional 

Started digging in to read_abf and what I'm thinking is that there is now a lot more information in something one of the things neo returns when it reads the abf file. Coming to this conclusion based on the data_dict that is passed to pd.DataFrame(data_dict) looks something like this:

{'channel_1': AnalogSignal with 1 channels of length 2000; units V; datatype float32 name: "b'IN3'" annotations: {'channel_index': 3} sampling rate: 20000.0 Hz time: 0.0 s to 0.1 s, 'primary': AnalogSignal with 1 channels of length 2000; units V; datatype float32 name: "b'IN2'" annotations: {'channel_index': 2} sampling rate: 20000.0 Hz time: 0.0 s to 0.1 s, 'time': array([ 0.00000000e+00, 5.00000000e-05, 1.00000000e-04, ..., 9.98500000e-02, 9.99000000e-02, 9.99500000e-02]) * s}

which obviously is a bit of a mess and pd.DataFrame can't handle.

[chadestep]

Send me the file and I'll get a fix up and running.

[DGalt]

This will happen with any abf file. Do you not have any on you?

[chadestep]

Really? I'm currently opening 'abfs' on a daily basis and playing around with them, no issues.

What version of neo are you using?

[DGalt]

The newest version from their master branch. Did they finally update whatever is available through pypy to make abf reading in Python 3 functional?

[chadestep]

So looks like it was neo 0.5.1alpha that was causing the issue.

Wanna close this or keep it open and eventually fix the bug?

[chadestep]

Closing this for now since it was solved by using the current pypi available version of neo - 0.4.1.

[DGalt]

So I look in to this today, since for whatever reason I'm having trouble with 0.4.1 on my machine. Seems that to get to the actual underlying data you need to refer to analagsignals.data (which doesn't actually give back anything useful, but you can convert it to a list or a numpy array).

This fixes the problem, if/when we decide to update to 0.5.1:

data = np.array(bl.segments[seg_num].analogsignals[i].data)
signals.append(data.T[0])

[chadestep]

It actually looks like they updated PyPi to 0.5.1 earlier this month, so feel free to make the change and push the changes. Or I can, but I won't be able to make the changes until after my interview.