IOError on dsres.mld

[tbeu]

I converted dsres.mat to mld using script dsres2meld. Then I got:

Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
>>> from recon.meld import MeldWriter, MeldReader
>>> fp = open(r'c:\Download\recon-master\tests\dsres.mld', 'rb')
>>> meld = MeldReader(fp, verbose=True)
Compression: True
Header = {'tabs': {'T2': {'vmeta': {'der(i_L)': {'desc': ''}, 'i_L': {'desc': ''}, 'time': {'desc': 'Simulation time [s]'}, 'i_C': {'desc': ''}, 'der(V)': {'desc': ''}, 'V': {'desc': ''}, 'i_R': {'desc': ''}}, 'vars': ['time', 'der(i_L)', 'i_L', 'i_C', 'der(V)', 'V', 'i_R'], 'tmeta': {}, 'toff': {'der(i_L)': {'i': 2056, 'l': 2444}, 'i_L': {'i': 4510, 'l': 2233}, 'time': {'i': 568, 'l': 1483}, 'i_C': {'i': 6748, 'l': 2419}, 'der(V)': {'i': 9178, 'l': 2431}, 'V': {'i': 11616, 'l': 2113}, 'i_R': {'i': 13737, 'l': 2117}}}, 'T1': {'vmeta': {'time': {'desc': 'Simulation time [s]'}, 'Vb': {'desc': 'Battery voltage'}, 'R': {'desc': ''}, 'C': {'desc': ''}, 'L': {'desc': ''}}, 'vars': ['time', 'C', 'L', 'Vb', 'R'], 'tmeta': {}, 'toff': {'Vb': {'i': 16015, 'l': 49}, 'R': {'i': 16065, 'l': 49}, 'C': {'i': 15911, 'l': 52}, 'L': {'i': 15963, 'l': 51}, 'time': {'i': 15861, 'l': 50}}}}, 'fmeta': {}, 'objs': {}, 'comp': True}
>>> meld.tables()
['T2', 'T1']
>>> table = meld.read_table('T2')
>>> table.signals()
['time', 'der(i_L)', 'i_L', 'i_C', 'der(V)', 'V', 'i_R']
>>> table.data('time')
Traceback (most recent call last):
  File "<pyshell#11>", line 1, in <module>
    table.data('time')
  File "C:\Programme\Python27\lib\site-packages\pyrecon-0.3.0-py2.7.egg\recon\meld.py", line 568, in data
    data = self.reader.ser.decode_vec(self.reader.fp, blen)
  File "C:\Programme\Python27\lib\site-packages\pyrecon-0.3.0-py2.7.egg\recon\serial.py", line 114, in decode_vec
    verbose=verbose, uncomp=uncomp)
  File "C:\Programme\Python27\lib\site-packages\pyrecon-0.3.0-py2.7.egg\recon\serial.py", line 106, in decode_obj
    data = decompress(data)
  File "C:\Programme\Python27\lib\site-packages\pyrecon-0.3.0-py2.7.egg\recon\serial.py", line 17, in decompress
    return c.decompress(data)
IOError: invalid data stream

Environment is WinXP with Python 2.7.6, setuptools 2.2 and msgpack-python 0.4.1.

[xogeny]

Thomas,

One thing I notice. In your case you are opening files in "tests". But I'm not sure how you are testing this. Are you using the targets in the Makefile? Because if you were running the tests through the Makefile, the tests should be in a directory called test_output. So I'm wondering if perhaps some of the issues you are seeing might be because of some potential interaction with the material in the tests directory.

[tbeu]

Test is using Python console. Dir "tests" should not matter.

[xogeny]

Just as a note (mostly to myself), I've confirmed that I cannot reproduce this bug on the Mac. I'll check Linux and Windows when I get a chance.

[tbeu]

For sake of completeness: http://www.FastShare.org/download/dsres.zip

[tbeu]

Some more debugging

>>> ind = table.indices['time']['i']
>>> ind
568
>>> blen = table.indices['time']['l']
>>> blen
1483

Not sure if the error is already in the file. I guess that blen is wrong here.

[xogeny]

OK, another data point. I tried this under Linux. Again, unable to reproduce the issue (both with and without compression). I'm going to look at your sample files to see if I can spot any differences. I get the same values for i and l that you show above. Why do you think blen is wrong? It is a little strange that the l value is different for different signals, but msgpack tries to be clever about storage, so it is possible for vectors of the same size to have different lengths when packed.

One thing I noticed...you ran this on a 32-bit version of Windows. I'm wondering if that has something to do with it... ?

[xogeny]

OK, a bit more information. Under Linux, I cannot read the dsres.mld file you sent me. In fact, it fails in exactly the same place. So it appears the issue is in the writing, not the reading. As a test, here is the dsres.mld file I wrote under Linux. It would be interesting to see if it could be read under Windows (this would further point the finger at writing as the issue under Windows):

https://dl.dropboxusercontent.com/u/126298/dsres.mld

[xogeny]

OK, even more details. When I print the header of a file generated under Linux, I get the following index and length for time: {'i': 568, 'l': 1483}. I get the same data for a file generated under Windows. However, the next signal (according to the vars array) in both cases is der(i_L). Under Linux, the data for this signal is: {'i': 2051, 'l': 2444}. But, the data for this signal under Windows is {'i': 2056, 'l': 2444}. The Windows value is wrong since 568+1483=2051. So somehow the Windows version is writing the wrong index (off by 5?). The Windows signals consistently start at the wrong index!

[tbeu]

Fix confirmed. Thanks.

[xogeny]

For reference, the fix was committed in 50c48a0f8d264dd2619f42c214db1eedd03e5c76