PollyNET / Pollynet_Processing_Chain

NRT lidar data processing program for multiwavelength polarization Raman lidar network (PollyNET)
https://polly.tropos.de/
GNU General Public License v3.0
20 stars 8 forks source link

corrupted time in netcdf --> chain stops #202

Closed HolgerPollyNet closed 1 year ago

HolgerPollyNet commented 1 year ago

In case of power failures etc. it can happen that the netcdf is kind of corrupted in the last entry: grafik In the example here, the time values are completely nonsense. But currently the processing chain is aborting while plotting the long-term cali. Is there a way to avoid the stopping of the chain but to give a warning?

HolgerPollyNet commented 1 year ago

2023-02-28 00:14:56] --> start diplaying lidar housekeeping data. {Error using timelabellayout (line 166) The span is too large.

Error in pollyxt_displayHousekeeping (line 21) [xtick, xtickstr] = timelabellayout(data.mTime, 'HH:MM');

Error in pollyDisplayHousekeeping (line 19) pollyxt_displayHousekeeping(data);

Error in picassoProcV3 (line 4844) pollyDisplayHousekeeping(data);

Error in picassoProcTodolist (line 62) reportTmp = picassoProcV3(pollyDataFile, pollyDataTasks.pollyType{iTask},

ZPYin commented 1 year ago

To correct it a little bit... The error happens in the plotting routine of housekeeping data, while it detects an unreasonably large time interval between the first and the last profile.

To keep processing chain running without collapse, we can can a try..catch... in pollyxt_displayHousekeeping.m. which I will implement in the next commit.

But this seems not perfect for this situation. Because the processing for the current netcdf file will mostly stop and only produce several profile plots of optical products. So what about inserting another condition check in the preprocessing module, to rule out profiles with strange timestamp (let's say negative values...)? Then all the plots will still be produced.

HolgerPollyNet commented 1 year ago

I completely agree. Try and catch can be used as workaround, but more sustainable is the preproc solution!

ZPYin commented 1 year ago

see below vol. depol. for PollyXT_CPV when wrong timestamp occurs in the netcdf data

2023_02_26_Sun_CPV_18_00_31_VDR_532

HolgerPollyNet commented 1 year ago

@ulysses78 can you pull the new version to the server?

ulysses78 commented 1 year ago

job done!

ulysses78 commented 1 year ago

this one can be closed, right?