XDATA decoding & new option for JSON output for RS41

[pavolgaj]

Hi, I wrote a code for decoding XDATA/AUX data for different instruments. It's based on JS of @LukePrior in the SondeHub project: https://github.com/projecthorus/sondehub-tracker. I added the new decoding option only to rs41mod code - it can be turned on using --aux option. Decoded data are shown with PTU values. For standard output, only the most important values are shown. All decoded data are displayed in JSON output as a dictionary with individual dictionaries for all presented auxiliary instruments.

I also add a new option for JSON output (--json2). It's focused mainly on sending output to JSON file. The output file is now the correct JSON file and can be correctly opened as a JSON file. Data are written as a list of frames.

If you find some of my changes useful, include them in your code.

Pavol

[LukePrior]

Hi @pavolgaj,

I just had a few notes about my code and documentation that you might wish to consider.

• The Vaisala OIF411 code works great and is suitable to be added.
• The ETH Zurich COBALD code has been tested with live data and everything appears correct so it should be good to go.
• The ETH Zurich PCFH code has been tested with live data and everything appears correct so it should be good to go.
• The Central Aerological Observatory FLASH-B code has not been tested with live data and is missing approximate calibration values. I would suggest not implementing this here until it has been tested on a live flight or more example/reference data is acquired.
• The Meisei Electric SKYDEW code has also not been tested with any live data and transmits it's calibration values over multiple frames which need to be stored so that the mirror temperature can be calculated. I would recommend acquiring some test data and confirming it works as expected before implementing here.

Thanks, Luke

[rs1729]

@pavolgaj As I said before, I didn't want the aux data decoding in the rs41-decoder https://github.com/rs1729/RS/issues/20. And it is done in Sondehub now: https://github.com/projecthorus/sondehub-infra/wiki/XDATA-Decoding

You put some work into the xdata parsing. There are many diffs, though the only file needed would be xdata.h, right? And then minor additions to rs41mod.c, maybe optional as #define. But then there is this additional asprintf.h - I understand there is a lot of string processing involved, but I'm not sure if I would like to have stdarg.h variable arguments functions (except for the string.h function, but asprintf() is not included by default.) Anyway, some remarks/questions:

You allocate strings via

asprintf(&output->serial,"%s",tmp);

but there is no null pointer check before

fprintf(stdout, "\"serial\": \"%s\"",output.serial);

? (same for diagnostics) It's unlikely that it would not allocate the string, but still.

For strtol() I see you want the 0-termination, but when you compare

memcpy(tmp,&xdata[0],2);
tmp[2]='\0';
//printf("-%s-\n",tmp);
if (strcmp(tmp,"08") != 0){
    // Not an CFH (shouldn't get here)
    return;
}

wouldn't it be better just strncmp(xdata, "08", 2) ? (Also e.g. &xdata[12] would be xdata+12 if you mean the pointer, but that's maybe just my preference.)

I testet xdata.h in rs41mod.c, and for OIF411 (which is the most common aux data I guess) it showed only "O3_P: ...", not very much for so much code. So I changed prn_aux() to prn_full(), and it showed (maybe you put a xdata_-prefix to the external functions in xdata.h?)

OIF411: pump_temp=5.37C current=1.1059uA battery_volt=12.1V pump_current=68.0mA external_volt=0.0V O3_pressure=4.086mPa

That looks good. Then I tested a flight from Switzerland, but only the instrument IDs showed up:

xdata=0101033ffdce4196#080245A9F180005607D60169
 V7: CFH:

The ID=0x01 was somehow familiar, because I implemented this for the iMet-1-RS, I modified it a bit, so I get

ID=0x01  Icell:0.831uA  Tpump:-5.62°C  Ipump:65mA  Vbat:15.0V 

Is this reasonable? I think it is the same format as described in the iMet-O3-Documentation. Here what I adopted from the iMet-1-RS:

// N=8, ID=0x01: Ozonesonde (MSB)
// 0     1     Instrument_type = 0x01 (ID)
// 1     1     Instrument_number
// 2     2     Icell, uA (I = n/1000)
// 4     2     Tpump, °C (T = n/100)
// 6     1     Ipump, mA
// 7     1     Vbat, (V = n/10)
static int print_aux01(char *xdata) {
    int P, U;
    ui8_t InstrumentNum;
    short Tpump;
    unsigned short Icell, Ipump, Vbat;

    char *px = xdata;

    if (*px) {

        if (strncmp(px, "01", 2) != 0) {
            px = strstr(xdata, "#01");
            if (px == NULL) return -1;
            else px += 1;
        }
        // N == 8
        fprintf(stdout, " ID=0x01 ");
        InstrumentNum = hex2uint(px+2, 2);
        Icell = hex2uint(px+ 4, 4);
        Tpump = hex2uint(px+ 8, 4);
        Ipump = hex2uint(px+12, 2);
        Vbat  = hex2uint(px+14, 2);
        fprintf(stdout, " Icell:%.3fuA ", Icell/1000.0);
        fprintf(stdout, " Tpump:%.2f°C ", Tpump/100.0);
        fprintf(stdout, " Ipump:%dmA ", Ipump);
        fprintf(stdout, " Vbat:%.1fV ", Vbat/10.0);
    }
    else {
        return -2;
    }

    return 0;
}

The json2 xdata output in rs41mod.c is not for Sondehub I guess? I don't need the JSON output, but for post processing it could be useful. I you decode a live stream and abort with ctrl-c, wouldn't the closing ] be missing? Then you would need to catch the signal (though I would not want add this in the code).

In https://github.com/rs1729/RS/blob/3ccc0b3d07d253db7acfc193956cab4d8c238d99/demod/mod/rs41mod.c#L1829-L1832 you exclude RH2 == 0, but that's a valid value! I have seen this in Sondehub where 0% rel. humidity was not shown. Invalid value would be -1.0. When calculating RH it may happen that the calculated value is slightly below 0 or above 100, then it is set to the mininum or maximum value, resp. That's why the test is RH > -0.5. For dew point it is another story, there you cannot put RH=0 into the log, the spread would be infinite. Also in the original code it is either RH or RH2. For RH2 more calibration data has to be gathered, but if you take the simpler RH-values in the meantime, it depends on when you start decoding, if you get the same data.

So I don't know if I want to include xdata processing. But thanks to you and @LukePrior. It is nonetheless interesting to see all the information about the different instruments. Since I have the ID=0x01 implemented somehow, maybe I add OIF411 some day. Or an optional slim xdata.h.

[LukePrior]

Once we get some iMet-1/4 in SondeHub with XDATA packets I will implement the Ozonesonde (MSB) decoder you have described above into the SondeHub XDATA processing.

[rs1729]

I believe I have data from imet1rsb and rs41aux with 0101..., I can put some json output together.

[LukePrior]

That would be perfect if you could, thanks.

[rs1729]

xdata examples ID=0x01 imet1rsb_aux_20150616.json.gz rs41aux_20160202.json.gz

[pavolgaj]

@rs1729 Thanks for the comments. I am not an expert in C. So, I used functions I knew ;) I just rewrite code a bit using strncmp etc.

To JSON output: You're right it's mainly for postprocessing. I use this option if I load RAW data from a file using --rawhex. In this case, I don't need to care about aborting. I tried to catch the signals for a similar situation in a different code - but nothing works:( Don't you have some working code?

[rs1729]

Added xdata decoding for ID=0x01 and ID=0x05. But no O3-Pressure calculation. I think there is a typo in the Vaisala OIF411 manual, 5.2 ID Data, the software version is probably 4 bytes (ascii-nibbles). "bytes" meaning ascii character nibbles is a bit confusing. (cf. https://github.com/projecthorus/sondehub-infra/wiki/XDATA-Decoding ?)

@pavolgaj I will look at your changes later, however for now I'm good with the basic decoding I added. Maybe ID=0x08 CFH would be interesting. Who want's more, can look at your fork.

[rs1729]

@pavolgaj

To JSON output: You're right it's mainly for postprocessing. I use this option if I load RAW data from a file using --rawhex. In this case, I don't need to care about aborting. I tried to catch the signals for a similar situation in a different code - but nothing works:( Don't you have some working code?

Well, I believe I used signal handler before, but I don't remember... Do you redirect the output to a file? Maybe try this as test exampe:

#include <stdio.h>
#include <signal.h>
#include <stdlib.h>
#include <unistd.h>

void sig_handler(int sig) {
    if (sig == SIGINT) fprintf(stderr, "SIGINT\n"); // if more signals are registered
    printf("(signal exit) }\n");
    exit(1);
}

int main(int argc, char *argv[]) {

    // ctrl-c: SIGINT
    if (signal(SIGINT, sig_handler) == SIG_ERR) {
        fprintf(stderr, "error register SIGINT\n");
    }

    int t = 0;
    printf("{\n");
    while (t < 10) {
        printf("%d\n", t);
        sleep(1);
        t++;
    }
    printf("(normal exit) }\n");

    return 0;
}

^C should go to stderr, if you ./a.out > txt. If you have only one signal registered, probably you won't need to check the signal in the handler.

However in your program, it might not be clear where you aborted and what else would have to be put out before exiting. Maybe fflush(stdout) or so. And/or maybe close the input stream. Hm... Now that I think about it, maybe it is not worth the effort. And if you use only static files as input, you don't need it anyway.

[LukePrior]

xdata examples ID=0x01 imet1rsb_aux_20150616.json.gz rs41aux_20160202.json.gz

Thanks for these.

I have written up an initial version of the protocol here, I think the ozonesonde is made by EnSci but please correct me if wrong: https://github.com/projecthorus/sondehub-infra/wiki/XDATA-Decoding#ensci-ozonesonde-notes

I have also updated the javascript code here: https://github.com/projecthorus/sondehub-tracker/blob/4bc225fb22756dadad585ea7de73bd3cf1707831/js/xdata.js#L42

Please let me know if any of this seems incorrect.

[rs1729]

There is this older manual still available here http://harbor.weber.edu/Hardware/Ozonesonde.html http://harbor.weber.edu/Hardware/Ozonesonde/ECC_Ozonesonde-1.pdf (Somewhere I found rev B-3 also from 2014.) Don't know if there exists a newer version. There is the iMet-1-RSB protocol description. I think it has a typo in the "Ozonesonde Data (Using Extra Data Packet)" description, Ipump offset should be 9 (not 8). (In the iMet data it is really bytes, not ascii-nibbles)

The typo in the Vaisala OIF411 manual (software version length is 4 bytes as in the example, not 2), I think you have it also in the Wiki, but correct in the js.

[rs1729]

@pavolgaj (SIGINT) Perhaps it is better to set a (global) flag and check when to break out of the main loop (and/or close the input FILE).

#include <stdio.h>
#include <signal.h>
#include <unistd.h>

int flag_sigint = 0;

void sig_handler(int sig) {
    if (sig == SIGINT) fprintf(stderr, "SIGINT\n");

    flag_sigint = 1;
}

int main(int argc, char *argv[]) {

    // ctrl-c: SIGINT
    if (signal(SIGINT, sig_handler) == SIG_ERR) {
        fprintf(stderr, "error register SIGINT\n");
    }

    int t = 0;
    printf("{\n");
    while (t < 10) {
        if (flag_sigint) break;
        printf("%d\n", t);
        sleep(1);
        t++;
    }
    printf("}\n");

    return 0;
}

However in the more complex decoder you would probably have to check the flag at different locations and decide how much should be processed before aborting/exiting. E.g. the decoder spends some time detecting the next frame header. And the different inputs wav/samples or raw frames are in different loops. You don't want to wait for too long before aborting after SIGINT. On the other hand you want to exit with a clean output. There is also the setjmp()/longjmp(), but I'm not a fan of that. The more libraries you add, the less portable it looks.