aodn / imos-toolbox

Graphical tool for QC'ing and NetCDF'ing oceanographic datasets
GNU General Public License v3.0
46 stars 31 forks source link

error proccessing CTD data from multiple casts #603

Closed anaredd closed 5 years ago

anaredd commented 5 years ago

The toolbox gives an error when trying to process data from profiler SBE37 when only downcast data from a single cast is present but such data is from a cast number >1. The SBE19Parser (i.e. genTimestamps function, line 735) generates timestamps for the data assuming that all data from all casts are present in the file. (Can work around it by commenting out lines 735 to 744 fixes the problem, see example file attached) vb1_4cafld.zip

ocehugo commented 5 years ago

Thanks for the report @anaredd.

may you describe a bit more? After a quick try, I couldn't reproduce the exact description. The SBE19Parse can successfully parse your file. However, the SBE37Parser couldn't but showed a different error (is this the error you faced first?):

Error using datenum (line 190)
DATENUM failed.

Error in SBE3x (line 417)
        time_block = datenum(time_block, 'dd mmm yyyy, HH:MM:SS');

Error in SBE37Parse (line 207)
    sample_data = SBE3x(filename, mode);

Caused by:
    Error using dtstr2dtnummx
    Failed to convert from text to date number.

Moreover, the SBE19Parse didn't even touch the lines you mentioned since the data loaded by readSBE19cnv contains TIME and returns earlier than those 735+ lines.

anaredd commented 5 years ago

Sorry Hugo, the problem does need more info. First of all, I meant SBE19Parse (not SBE37). The parse itself doesn't give any errors, the problem is that it parse the wrong info. The readSBE19cnv contains ETIME (not TIME) so the genTimestamps function doesn't return after the first few lines. Instead it returns an oversized variable 'time' (oversized because it considers the total number of samples (i.e. the 3 casts, in this case 5429) to be present. And because the variable 'time' is the wrong size, then the descendingTime = time(iD) is wrong and the error carries forward to return the wrong 'First value over profile measurement.' (i.e. a NaN) that triggers the overall error.

Hope it makes sense,

Ana

ocehugo commented 5 years ago

Sorry Hugo, the problem does need more info. First of all, I meant SBE19Parse (not SBE37). The parse itself doesn't give any errors, the problem is that it parse the wrong info.

Thanks, I thought was an actual matlab error!

The readSBE19cnv contains ETIME (not TIME) so the genTimestamps function doesn't return after the first few lines.

Yes, it contains ETIME but also TIME. The call after line 106 in SBE19Parse return a data variable with TIME and ETIME with 1610 records. Hence, when it reaches Line 137 for the genTimestamps call, it will return at line 713.

Instead it returns an oversized variable 'time' (oversized because it considers the total number of samples (i.e. the 3 casts, in this case 5429) to be present. And because the variable 'time' is the wrong size, then the descendingTime = time(iD) is wrong and the error carries forward to return the wrong 'First value over profile measurement.' (i.e. a NaN) that triggers the overall error.

I can't reproduce the 5429 record length. Did you clip the cnv file or sent the wrong one!? As said, the cnv file attached above only got 1610 records

anaredd commented 5 years ago

My apologies Hugo, on trying to fix a problem I created another one. I had modified the function 'convertSBEcnvVar.m' a few weeks ago to solve a problem with the new moored SBE37s and forgot about it. Please disregard the above discussion, it all works fine.

ocehugo commented 5 years ago

No worries @anaredd!

Mind creating an issue for your problem/required modifications on convertSBEcnvVar? If we don't know the errors, we can't fix them :)