Corrupt RocketLogger data files when logging no binary data channels

[gemarcano]

Per https://github.com/ETHZ-TEC/RocketLogger/wiki/data-format,

But, using data collected by an updated RocketLogger to version 2, even when the number of binary channels is zero, the data file contains a 0 4-byte integer for the binary channels, running counter to the specification. Is the specification wrong? Or is the implementation when recording with the RocketLogger incorrect in adding that extra byte there?

I found this because I made my own parser just to understand what was going on under the hood, which worked fine for v3 RLD files, but stopped working for v4.

Just to be thorough, I'm attaching the beginning of one of my RLD files. The extra byte in the first block is at offset 0x138.

00000000: 25 52 4c 44 04 00 18 01 01 00 00 00 60 17 00 00  %RLD........`...
00000010: 60 17 00 00 00 00 00 00 0a 00 98 5d ad 37 62 f3  `..........].7b.
00000020: b4 4a 9f 61 00 00 00 00 9a cd 05 0f 00 00 00 00  .J.a............
00000030: 70 00 00 00 00 00 04 00 34 20 6d 75 64 77 61 74  p.......4 mudwat
00000040: 74 73 2c 20 61 6c 6c 20 73 74 61 72 74 20 77 69  ts, all start wi
00000050: 74 68 20 67 72 6f 75 6e 64 20 73 68 61 72 65 64  th ground shared
00000060: 2e 20 45 76 65 72 79 20 6d 69 6e 75 74 65 2c 20  . Every minute, 
00000070: 49 20 77 69 6c 6c 20 64 69 73 63 6f 6e 6e 65 63  I will disconnec
00000080: 74 20 6f 6e 65 20 67 72 6f 75 6e 64 20 61 6e 64  t one ground and
00000090: 20 63 6f 6e 6e 65 63 74 20 69 74 20 69 6e 20 73   connect it in s
000000a0: 65 72 69 65 73 2e 00 00 01 00 00 00 f8 ff ff ff  eries...........
000000b0: 04 00 ff ff 56 31 00 00 00 00 00 00 00 00 00 00  ....V1..........
000000c0: 00 00 00 00 01 00 00 00 f8 ff ff ff 04 00 ff ff  ................
000000d0: 56 32 00 00 00 00 00 00 00 00 00 00 00 00 00 00  V2..............
000000e0: 01 00 00 00 f8 ff ff ff 04 00 ff ff 56 33 00 00  ............V3..
000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
00000100: f8 ff ff ff 04 00 ff ff 56 34 00 00 00 00 00 00  ........V4......
00000110: 00 00 00 00 00 00 00 00 b4 4a 9f 61 00 00 00 00  .........J.a....
00000120: 2f 46 4e 0f 00 00 00 00 05 ca 1e 00 00 00 00 00  /FN.............
00000130: 44 e3 d8 31 00 00 00 00 00 00 00 00 20 59 91 04  D..1........ Y..
00000140: 9b f5 2c 04 af 38 6a 04 6e 84 bc 03 b4 4a 9f 61  ..,..8j.n....J.a
00000150: 00 00 00 00 74 cc 49 15 00 00 00 00 05 ca 1e 00  ....t.I.........
00000160: 00 00 00 00 c4 61 d4 37 00 00 00 00 00 00 00 00  .....a.7........
00000170: 90 5a 91 04 2c f4 2c 04 43 42 6a 04 6e 84 bc 03  .Z..,.,.CBj.n...
00000180: b4 4a 9f 61 00 00 00 00 e0 f1 44 1b 00 00 00 00  .J.a......D.....

[lsigrist]

Thanks a lot @gemarcano for reporting this issues with detailed example! You're absolutely right: there should no binary channel data included in the file if there is no binary data being logged (i.e. no digital channels, no low current channels that come together with a binary range valid channel).

I can reproduce this problem as described and have to confirm that this unfortunately is a regression that was introduced in version 2.0.0.

[lsigrist]

Inspection of the relevant code for different RocketLogger version suggests that an incorrectly formatted RLD file is generated when using the measurement configuration and RocketLogger versions listed below:

• Measurement configurations with no binary data logged, i.e. with all of the following channels disabled: I1L, I2L, digital inputs DI1-DI6
• RocketLogger versions 2.0.0 and 2.0.1

If one of the above listed channels is enabled, a correctly formatted RLD file is generated. RocketLogger versions 1.1.6 and earlier are not affected. CSV file outputs are not affected.

[lsigrist]

Obviously, the provided Python support library also cannot deal with the incorrectly structured RLD files. At least it correctly detects and reports corrupt data due to file size mismatch.

A candidate fix is available as part of #60. I'll have to follow up on this later this week to verify this fix and to provide means to recover corrupt RLD files. As I consider this a quite critical problem, the idea is performing bugfix release ASAP.

[lsigrist]

@gemarcano Apart from confirming the effectiveness of the earlier mentioned fix, I also added a standalone Python script to recover measurements files suffering from this bug. That script creates a copy with patched header info to let the file have a specification compliant data structure. Could you double check whether this also works for your measurements files (its also part of #60)?

[lsigrist]

fix verified using 868d732