search

pennmem / ptsa

PTSA - Python Time Series Analysis
https://pennmem.github.io/ptsa/
GNU General Public License v3.0
20 stars 9 forks source link

timeseries.to_hdf method not working #228

Closed nherweg closed 6 years ago

nherweg commented 6 years ago

When trying to save a timeseries object (generated with the cmlreader .to_ptsa method) to disc, I get the following error:

TypeError Traceback (most recent call last) ~/anaconda2/envs/pythree/lib/python3.6/site-packages/ptsa/data/timeseries.py in to_hdf(self, filename, mode, compression, compression_opts, encode_string_arrays, encoding) 169 name, data=data, chunks=chunks, --> 170 **compression_kwargs) 171 except TypeError as e:

~/anaconda2/envs/pythree/lib/python3.6/site-packages/h5py/_hl/group.py in create_dataset(self, name, shape, dtype, data, kwds) 105 with phil: --> 106 dsid = dataset.make_new_dset(self, shape, dtype, data, kwds) 107 dset = dataset.Dataset(dsid)

~/anaconda2/envs/pythree/lib/python3.6/site-packages/h5py/_hl/dataset.py in make_new_dset(parent, shape, dtype, data, chunks, compression, shuffle, fletcher32, maxshape, compression_opts, fillvalue, scaleoffset, track_times) 99 dtype = numpy.dtype(dtype) --> 100 tid = h5t.py_create(dtype, logical=1) 101

h5py/h5t.pyx in h5py.h5t.py_create()

h5py/h5t.pyx in h5py.h5t.py_create()

h5py/h5t.pyx in h5py.h5t.py_create()

h5py/h5t.pyx in h5py.h5t._c_compound()

h5py/h5t.pyx in h5py.h5t.py_create()

h5py/h5t.pyx in h5py.h5t.py_create()

TypeError: Object dtype dtype('O') has no native HDF5 equivalent

During handling of the above exception, another exception occurred:

TypeError Traceback (most recent call last)

in () 100 print ('done'+sessi.path) 101 --> 102 mwf([sessionList[0],'RET']) 103 # Check if analyses is already complete 104 #if np.any([not path.isfile('/scratch/nherweg/FR1_clustering/POW/'+session(ind,True,sessionList).subj+'_sess'+str(session(ind,True,sessionList).get_events().snumber)+'_POW_RET') for ind in range(len(sessionList))]): in mwf(varin, subjectList, sessionList, exclude) 74 75 if 'RET' in outp: ---> 76 EEG.to_hdf('/scratch/nherweg/FR1_clustering/PHS/'+subi+'_sess'+str(sessi)+'_RAW'+outp) 77 f2use = np.logspace(np.log10(3), np.log10(100), 25) 78 ~/anaconda2/envs/pythree/lib/python3.6/site-packages/ptsa/data/timeseries.py in to_hdf(self, filename, mode, compression, compression_opts, encode_string_arrays, encoding) 172 if chunks is not False: 173 dset = coords_group.create_dataset( --> 174 name, data=data, chunks=False) 175 else: 176 raise e ~/anaconda2/envs/pythree/lib/python3.6/site-packages/h5py/_hl/group.py in create_dataset(self, name, shape, dtype, data, **kwds) 104 """ 105 with phil: --> 106 dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds) 107 dset = dataset.Dataset(dsid) 108 if name is not None: ~/anaconda2/envs/pythree/lib/python3.6/site-packages/h5py/_hl/dataset.py in make_new_dset(parent, shape, dtype, data, chunks, compression, shuffle, fletcher32, maxshape, compression_opts, fillvalue, scaleoffset, track_times) 98 else: 99 dtype = numpy.dtype(dtype) --> 100 tid = h5t.py_create(dtype, logical=1) 101 102 # Legacy h5py/h5t.pyx in h5py.h5t.py_create() h5py/h5t.pyx in h5py.h5t.py_create() h5py/h5t.pyx in h5py.h5t.py_create() h5py/h5t.pyx in h5py.h5t._c_compound() h5py/h5t.pyx in h5py.h5t.py_create() h5py/h5t.pyx in h5py.h5t.py_create() TypeError: Object dtype dtype('O') has no native HDF5 equivalent
mivade commented 6 years ago

See #216.

mivade commented 6 years ago

And curse @ctw.

nherweg commented 6 years ago

I talked to Christoph and it seems this is actually a CML reader - ptsa compatibility problem. When constructing the time series object with .to_ptsa the dtype of the events looks like this:

dtype=(numpy.record, [('index', '<i8'), ('eegfile', 'O'), ('eegoffset', '<i8'), ('exp_version', '<f8'), ('experiment', 'O'), ('intrusion', '<i8'), ('is_stim', '?'), ('item_name', 'O'), ('item_num', '<i8'), ('list', '<i8'), ('montage', '<i8'), ('msoffset', '<i8'), ('mstime', '<i8'), ('protocol', 'O'), ('recalled', '?'), ('rectime', '<i8'), ('serialpos', '<i8'), ('session', '<i8'), ('stim_list', '?'), ('stim_params', 'O'), ('subject', 'O'), ('type', 'O')]))

ptsa's to_hdf method can not deal with the 'O' dtype.

mivade commented 6 years ago

That's a @ctw problem. He implemented the current version. I'm reverting to the previous way of doing things which require a lot fewer conditionals to make sure it works.