Exporting pickle files to mat files

[QNeuron]

Yep, it's not directly related to the toolbok, but I know you love the issue system better than emails :-)

So I am trying to convert the piclke files to mat files using the following instructions :

data = pickle.load(open(os.path.join(root,f), "rb" )) scipy.io.savemat(os.path.join(root,fnew), mdict={'data': data})

The problem is that for some of the ROI files the savemat fails, and I don't see any difference between .p files that works and .p files that don't. I have tried to change the .p files by adding a ROI and re-doing the trace extraction, but no luck.

Any help or idea would be appreciated, I am reaching the limits of my googling abilities here...

Content of the pickle file (same for files that can be converted or not): patches <type 'list'> masks <type 'list'> idxs <type 'list'> centres <type 'list'> corr_traces <type 'numpy.ndarray'> raw_traces <type 'numpy.ndarray'> dfF <type 'numpy.ndarray'> traces <type 'numpy.ndarray'> neuropil_traces <type 'numpy.ndarray'>

error message:

ValueError Traceback (most recent call last)

in () 8 data = pickle.load(open(os.path.join(root,f), "rb" )) 9 print(type(data)) ---> 10 scipy.io.savemat(os.path.join(root,fnew), mdict={'data': data}) 11 print("Done.") 12 C:\Users\dpag0680\AppData\Local\Continuum\anaconda2\lib\site-packages\scipy\io\matlab\mio.pyc in savemat(file_name, mdict, appendmat, format, long_field_names, do_compression, oned_as) 217 else: 218 raise ValueError("Format should be '4' or '5'") --> 219 MW.put_variables(mdict) 220 if file_opened: 221 file_stream.close() C:\Users\dpag0680\AppData\Local\Continuum\anaconda2\lib\site-packages\scipy\io\matlab\mio5.pyc in put_variables(self, mdict, write_header) 847 self.file_stream.write(out_str) 848 else: # not compressing --> 849 self._matrix_writer.write_top(var, asbytes(name), is_global) C:\Users\dpag0680\AppData\Local\Continuum\anaconda2\lib\site-packages\scipy\io\matlab\mio5.pyc in write_top(self, arr, name, is_global) 588 self._var_name = name 589 # write the header and data --> 590 self.write(arr) 591 592 def write(self, arr): C:\Users\dpag0680\AppData\Local\Continuum\anaconda2\lib\site-packages\scipy\io\matlab\mio5.pyc in write(self, arr) 617 self.write_empty_struct() 618 elif narr.dtype.fields: # struct array --> 619 self.write_struct(narr) 620 elif narr.dtype.hasobject: # cell array 621 self.write_cells(narr) C:\Users\dpag0680\AppData\Local\Continuum\anaconda2\lib\site-packages\scipy\io\matlab\mio5.pyc in write_struct(self, arr) 736 self.write_header(matdims(arr, self.oned_as), 737 mxSTRUCT_CLASS) --> 738 self._write_items(arr) 739 740 def _write_items(self, arr): C:\Users\dpag0680\AppData\Local\Continuum\anaconda2\lib\site-packages\scipy\io\matlab\mio5.pyc in _write_items(self, arr) 753 for el in A: 754 for f in fieldnames: --> 755 self.write(el[f]) 756 757 def write_object(self, arr): C:\Users\dpag0680\AppData\Local\Continuum\anaconda2\lib\site-packages\scipy\io\matlab\mio5.pyc in write(self, arr) 606 return 607 # Try to convert things that aren't arrays --> 608 narr = to_writeable(arr) 609 if narr is None: 610 raise TypeError('Could not convert %s (type %s) to array' C:\Users\dpag0680\AppData\Local\Continuum\anaconda2\lib\site-packages\scipy\io\matlab\mio5.pyc in to_writeable(source) 448 return EmptyStructMarker 449 # Next try and convert to an array --> 450 narr = np.asanyarray(source) 451 if narr.dtype.type in (object, np.object_) and \ 452 narr.shape == () and narr == source: C:\Users\dpag0680\AppData\Local\Continuum\anaconda2\lib\site-packages\numpy\core\numeric.pyc in asanyarray(a, dtype, order) 542 543 """ --> 544 return array(a, dtype, copy=False, order=order, subok=True) 545 546 ValueError: setting an array element with a sequence.

[QNeuron]

OK actually I explored a little more, and found out that for the files that could not be converted, there are some NaN values in the 'patches' entry. So when trying to save, the function finds a NaN instead of an array, and fails.

The problem is that these nan values are replacing patches missing in the pickle file. On that particular file I expected 74 ROIs, and I have 17 nans and 57 actual patches. So, how can I correct that ? I already tried to run extract_traces.py again, but it did not help...

When are the patches generated ? And actually, what are they ? A chunk of the mean stack ?

[samupicard]

Just saw this @QNeuron, and I had the same issue yesterday. It turns out that for the ROIs that were automatically selected (using 'run_classifier') the 'patches' entry is just a single NaN tuple (instead of the 30-by-30-ish 2D array that it would be if you'd drawn the ROI yourself.

As a work-around I've written this piece of code wherever I save to mat-file so any nan-values are always converted to a 2-by-2 array of nans. (You might have solved this issue already but I thought I'd post my workaround just in case it's still a problem)

for idx, val in enumerate(roiattrs['patches']): if np.any(np.isnan(val)): roiattrs['patches'][idx] = np.array([[np.nan,np.nan],[np.nan,np.nan]])

[QNeuron]

Thanks @samupicard . My workaround was to ignore the patches, I don't use them in my analysis anyway ^^.

But I am still wondering what should be the proper behaviour for the code ? Why would you expect NaNs when the ROIs are automatically sorted ?