Invalid matlab file generated for stacked spectra

[Famlam]

1. Have a processed Nx6 spectrum (say, the spectra belonging to a T1 curve)
2. Export as Matlab data
3. Try to load said data (either in ssNake or Matlab)

The data in the .mat file is invalid and cannot be loaded. Reason: xaxArray is of a bad format. What it tries to export to .mat format is:

[array([0., 1., 2., 3., 4., 5.]), array([-2.86115706e+04, -2.86100448e+04, -2.86085189e+04, -2.86069930e+04,
       -2.86054671e+04, -2.86039412e+04, -2.86024154e+04, -2.86008895e+04,
       -2.85993636e+04, -2.85978377e+04, -2.85963118e+04, -2.85947860e+04,
       -2.85932601e+04, -2.85917342e+04, -2.85902083e+04, -2.85886824e+04,
       -2.85871566e+04, -2.85856307e+04, -2.85841048e+04, -2.85825789e+04,
       -2.85810531e+04, -2.85795272e+04, -2.85780013e+04, -2.85764754e+04,
       -2.85749495e+04, -2.85734237e+04, -2.85718978e+04, -2.85703719e+04,
       -2.85688460e+04, -2.85673201e+04, -2.85657943e+04, -2.85642684e+04,
       -2.85627425e+04, -2.85612166e+04, -2.85596907e+04, -2.85581649e+04,
...
       3.108e-02, 3.110e-02, 3.112e-02, 3.114e-02, 3.116e-02, 3.118e-02])]

What matlab says if you try to load it:

Number of columns on line 2 of ASCII file C:\...\filename.mat must be the same as previous lines.

What ssNake says if you try to load it:

Traceback (most recent call last): 
File "C:\...\ssNake.py", line 1476, in loadFromMenu self.loadData(fileList) 
File "C:\...\ssNake.py", line 1484, in loadData masterData = io.autoLoad(filePath) ^^^^^^^^^^^^^^^^^^^^^ 
File "C:\...\specIO.py", line 62, in autoLoad masterData = autoLoadSingle(filePathList[0], asciiInfoList[0]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "C:\...\specIO.py", line 119, in autoLoadSingle tmpSpec = loadFile(filePath, asciiInfo=asciiInfo) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "C:\...\specIO.py", line 172, in loadFile masterData = loadMatlabFile(filePath) ^^^^^^^^^^^^^^^^^^^^^^^^ 
File "C:\...\specIO.py", line 874, in loadMatlabFile matlabStruct = scipy.io.loadmat(filePath) ^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "C:\...\scipy\io\matlab\_mio.py", line 227, in loadmat matfile_dict = MR.get_variables(variable_names) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "C:\...\scipy\io\matlab\_mio5.py", line 312, in get_variables hdr, next_position = self.read_var_header() ^^^^^^^^^^^^^^^^^^^^^^ 
File "C:\...\scipy\io\matlab\_mio5.py", line 259, in read_var_header raise ValueError("Did not read any bytes") 
ValueError: Did not read any bytes 

---

p.s. in some cases saving as .mat also fails, I haven't found a clear pattern in this.

Traceback (most recent call last): 
File "C:\...\ssNake.py", line 1569, in saveMatlabFile self.mainWindow.get_mainWindow().saveMatlabFile() 
File "C:\...\ssNake.py", line 1795, in saveMatlabFile io.saveMatlabFile(name, self.masterData, self.father.workspaceNames[self.father.workspaceNum]) 
File "C:\...\specIO.py", line 852, in saveMatlabFile scipy.io.savemat(filePath, matlabStruct) 
File "C:\...\scipy\io\matlab\_mio.py", line 300, in savemat MW.put_variables(mdict) 
File "C:\...\scipy\io\matlab\_mio5.py", line 892, in put_variables self._matrix_writer.write_top(var, name.encode('latin1'), is_global) 
File "C:\...\scipy\io\matlab\_mio5.py", line 633, in write_top self.write(arr) 
File "C:\...\scipy\io\matlab\_mio5.py", line 662, in write self.write_struct(narr) 
File "C:\...\scipy\io\matlab\_mio5.py", line 781, in write_struct self._write_items(arr) 
File "C:\...\scipy\io\matlab\_mio5.py", line 798, in _write_items self.write(el[f]) 
File "C:\...\scipy\io\matlab\_mio5.py", line 651, in write narr = to_writeable(arr) ^^^^^^^^^^^^^^^^^ 
File "C:\...\scipy\io\matlab\_mio5.py", line 493, in to_writeable narr = np.asanyarray(source) ^^^^^^^^^^^^^^^^^^^^^ 
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part. 

[wfranssen]

For me it is even worse, I cannot save any data to .mat. Something must have changed in the library versions that cause this. I tested ssNake v1.4 with new libraries, and there is the same issue. Currently running Scipy 1.10.1. I for now cannot find any info on changes on the scipy side though...

Edit: 1D data does seem to work.

[wfranssen]

Problem seems to occur when the arrays in 'xaxArray' are of unequal length. (This is a list of 1D array's.) If I make 2D data with equal lengths in dimension, I can again save and load the data.

If I remove the 'xaxArray' part from the save command, it works again for any shape data.

The 'data' part of hypercomplex data does not have the same issues. Probably because these data matrices are of equal shape by definition.

I can still load old 2D data with unequal shape (tested on MQMAS data from the advanced tutorial), so I guess it is a problem only for the save command of recent scipy versions.

[MirjamSchr]

Hi there,

I would like to add, that I cannot save any (1D, processed FID) data correctly neither as .json nor as .mat file. With the developer version v1.5b it did save a file which was subsequently not possible to be loaded again.

The attempts did not raise messages in the console except for one Deprecation Warning in v1.5b which I'll leave here for completeness: [...]ssNake.py:3303: DeprecationWarning: string or file could not be read to its end due to unmatched data; this will raise a ValueError in the future. data = np.fromstring(line, sep=sepList[self.delimiters.index(sep)])

The version of my packages are the following:

Kind Regards, Mirjam

[wfranssen]

Hi Mirjam,

Thanks for reporting. I cannot reproduce this with my system, which has slightly older library versions. I need to play around a bit to find out the problem.

Does the issue occur with any data set, both 1D and 2D?

Regards,

Wouter

[groenoord]

Hi Wouter,

I have similar problems related to saving matlab files that occur with the latest developer version together with an updated anaconda3-2023-03. Although I tried to go back to older versions and that didn't seem to help.

When I save a spectrum in matlab format and load it again, I can't do any lineshape fitting. The error I get is:

File "/usr/local/ssnake-develop/src/fitting.py", line 1915, in disp outCurve += y ValueError: non-broadcastable output operand with shape (65536,) doesn't match the broadcast shape (1,65536)

There seems to be some inconsistency when saving and re-loading data with the shape of the array/

All the best,

Andreas

[groenoord]

Some additions: ssNake 1.4 can save and re-open matlab files and I don't get the error above. I can also re-open the matlab file saved in 1.4 in version 1.5b. However, when I save a matlab file in 1.5b and re-opened it in any version I get the above error. My conclusion would be that in 1.5b the matlab saving was somehow broken.

[wfranssen]

Hi Andreas,

I can reproduce this error. I lately pushed a fix for saving to .mat data for newer scipy versions. It seems that 1D data was now saved as pseudo2D when it comes to the x-axis.

I now pushed a fix for this. Let me know if the issue is now fixed.

Note that the 1D data that you saved on the version of yesterday is faulty, so please create new data to test.

Regards,

Wouter

[groenoord]

Thanks, Wouter, Your fix works! All the best, Andreas