search

PollyNET / Pollynet_Processing_Chain

NRT lidar data processing program for multiwavelength polarization Raman lidar network (PollyNET)
https://polly.tropos.de/
GNU General Public License v3.0
20 stars 8 forks source link

store configuration file in netcdf #235

Open HolgerPollyNet opened 9 months ago

HolgerPollyNet commented 9 months ago

To allow reproducibility, I would propose to store the whole config file which has been used, i.e. all parameters, in the netcdf as well. Maybe as simple character variable first. But this would help to identify issues in case the processing was made elsewhere....

HolgerPollyNet commented 2 months ago

I tried to do so.

  1. I just wanted to dump the whole PollyConfig global variable in one string and store it as one global attribute in the netcdf file. Similar to the history global attribute in each netcdf: :history = "Last processing time at 2024-07-31 13:49:40 by pollySaveProfiles, git branch: cleaning_up, git commit: 6...";

However I failed to convert the whole PollyConfig struct to a string. grafik grafik

because matlab cannot easily do this for type 'struct'

  1. I tried to use a for loop reading the field names of struct PollyConfig, and put this as attribute together with the respective value: names=fieldnames(PollyConfig) values = struct2cell(PollyConfig) s = strings for i=1:size(names) s = strcat(s,names,string(values)) end however I failed because the respective data type of the fields is always different:

grafik

Any idea/advise? @ZPYin @cristoferjimenez @ulysses78 @Moritz-TROPOS @AthenaAugousta

Best Holger

cristoferjimenez commented 2 months ago

Hi Holger,

I think in that case, one needs to adapt the types of each parameter. I tried this change and it works.

Best regards, Cristofer

s = strings
for i=1:size(names)

    type=   class(values{i});
    if strcmp(type,'cell')
        values2=num2str(str2double(values{i})) ;
    elseif strcmp(type,'logical')
        values2=num2str(double(values{i}));
    else
        values2=(values{i});

        if strcmp(type,'double')

            if size(values2,1)>1
                values2=num2str(reshape(values2,1,size(values2,1)*size(values2,2)));
            else
                values2=num2str(values2);
            end
        end
    end

    s = strcat(s,names{i},',',string(values2),',')
end
ZPYin commented 2 months ago

Hi Holger,

If you want to convert a Matlab struct to a char array or a string, you need to specify the output format for different data types (as suggested by @cristoferjimenez ).

I modified the code (from @cristoferjimenez ) a little bit to make it support the conversion of PollyConfig struct. Please see the two little functions in the link below.

matlab-funcs-struct-2-char.zip

Below is an output test.

PollyConfig

{
    "flagCorrectFalseMShots": true, 
    "flagFilterFalseMShots": false,
    "flagDTCor": true,

    "flagWVCalibration": true,
    "flagUsePreviousDepolCali": true,
    "flagUsePreviousWVconst": true,
    "flagUsePreviousLC": true,

    "dataFileFormat": "(?<year>\\d{4})_(?<month>\\d{2})_(?<day>\\d{2})_\\w*_(?<hour>\\d{2})_(?<minute>\\d{2})_(?<second>\\d{2})\\w*.nc",
    "gdas1Site": "cabauw",
    "AERONETSite": "Cabauw",

    "max_height_bin": 3000,
    "first_range_gate_indx": [261, 261, 261, 261, 261, 261, 261, 261, 262, 262],
    "first_range_gate_height": 78.75,
    "dtCorMode": 2,
    "dt": [[0.0, 0.972992, 0.00353332, -7.90981e-006, 1.06451e-007, 1.42895e-009], 
        [0, 1.0117, -0.0014, 0.0002, -0.0000, 0.0000],
        [0, 0.9674, 0.0023, 0.0000, 0.0000, 0.0000],
        [0, 0.9929, 0.0000, 0.0001, -0.0000, 0.0000],
        [0, 0.9843, 0.0022, 0.0001, -0.0000, 0.0000],
        [0, 0.9391, 0.0063, -0.0001, 0.0000, -0.0000],
        [0, 1.0035, 0.0003, 0.0001, -0.0000, 0.0000],
        [0, 1.0000, 0, 0, 0, 0],
        [0, 1.0000, 0.0029, 0.0000, 0.0000, 0.0000],
        [0, 1.0000, 0.0028, 0.0000, 0.0000, 0.0000]],
    "bgCorRangeIndx": [10, 240],
    "mask_SNRmin": [1, 1, 1, 0.01, 1, 1, 1, 1, 1, 1], 
    "init_depAng": 19.99,
    "maskDepCalAng": ["none", "none", "p", "p", "p", "p", "p", "p", "p", "p", "none", "none", "n", "n", "n", "n", "n", "n", "n", "n", "none"],
    "depol_cal_minbin_532": 40,
    "depol_cal_maxbin_532": 300,
    "depol_cal_SNRmin_532": [1, 1, 1, 1],
    "depol_cal_sigMax_532": [1500, 1500, 1500, 1500],
    "rel_std_dplus_532": 0.2,
    "rel_std_dminus_532": 0.2,
    "depol_cal_segmentLen_532": 40,
    "depol_cal_smoothWin_532": 8,
    "depol_cal_minbin_355": 40,
    "depol_cal_maxbin_355": 300,
    "depol_cal_SNRmin_355": [2, 2, 2, 2],
    "depol_cal_sigMax_355": [1000, 1000, 1000, 1000],
    "rel_std_dplus_355": 0.2,
    "rel_std_dminus_355": 0.2,
    "depol_cal_segmentLen_355": 40,
    "depol_cal_smoothWin_355": 8,
    "isFR": [1, 1, 1, 1, 1, 1, 1, 1, 0, 0], 
    "isNR": [0, 0, 0, 0, 0, 0, 0, 0, 1, 1], 
    "is532nm": [0, 0, 0, 0, 1, 1, 0, 0, 1, 0], 
    "is355nm": [1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
    "is1064nm": [0, 0, 0, 0, 0, 0, 0, 1, 0, 0],
    "isTot": [1, 0, 0, 0, 1, 0, 0, 1, 1, 0],
    "isCross": [0, 1, 0, 0, 0, 1, 0, 0, 0, 0],
    "is387nm": [0, 0, 1, 0, 0, 0, 0, 0, 0, 0],
    "is407nm": [0, 0, 0, 1, 0, 0, 0, 0, 0, 0],
    "is607nm": [0, 0, 0, 0, 0, 0, 1, 0, 0, 1],
    "channelTag": ["FR-total-355 nm", "FR-cross-355 nm", "FR-387 nm", "FR-407 nm", "FR-total-532 nm", "FR-cross-532 nm", "FR-607 nm", "FR-total-1064 nm", "NR-total-532 nm", "NR-607 nm"],
    "minPC_fog": 60,
    "TR": [0.69, 500, 1, 1, 1.09, 500, 1, 1, 1, 1], 
    "overlapCalMode": 1,
    "maxSigSlope4FilterCloud": 3e6,
    "saturate_thresh": 100,
    "heightFullOverlap": [500, 500, 500, 500, 500, 500, 500, 500, 150, 150],
    "minSNR_4_sigNorm": [10],
    "intNProfiles": 120,
    "minIntNProfiles": 90,
    "meteorDataSource": "gdas1",
    "radiosondeSitenum": 0,
    "IWV_instrument": "AERONET",
    "maxIWVTLag": 0.0833,
    "minDecomLogDist355": 0.2,
    "minDecomLogDist532": 0.2,
    "minDecomLogDist1064": 0.2,
    "maxDecomHeight355": 8000,
    "maxDecomHeight532": 8000,
    "maxDecomHeight1064": 8000,
    "maxDecomThickness355": 700,
    "maxDecomThickness532": 700,
    "maxDecomThickness1064": 700,
    "decomSmoothWin355": 60,
    "decomSmoothWin532": 60,
    "decomSmoothWin1064": 120,
    "minRefThickness355": 300,
    "minRefThickness532": 300,
    "minRefThickness1064": 300,
    "minRefDeltaExt355": 1,
    "minRefDeltaExt532": 1,
    "minRefDeltaExt1064": 1,
    "minRefSNR355": 5,
    "minRefSNR532": 5,
    "minRefSNR1064": 4,
    "LR355": 50,
    "LR532": 50,
    "LR1064": 50,
    "refBeta355": 2e-8,
    "refBeta532": 1e-8,
    "refBeta1064": 2e-9,
    "smoothWin_klett_355": 21,
    "smoothWin_klett_532": 21,
    "smoothWin_klett_1064": 21,
    "maxIterConstrainFernald": 20,
    "minLRConstrainFernald": 1,
    "maxLRConstrainFernald": 150,
    "minDeltaAOD": 0.01,
    "minRamanRefSNR387": 20,
    "minRamanRefSNR607": 10,
    "angstrexp": 0.9,
    "smoothWin_raman_355": 61,
    "smoothWin_raman_532": 61,
    "smoothWin_raman_1064": 61,
    "LCMeanWindow": 50,
    "LCMeanMinIndx": 70,
    "LCMeanMaxIndx": 1000,
    "LCCalibrationStatus": ["none", "Klett", "Raman", "Defaults", "History"],
    "quasi_smooth_h": [8, 8, 8, 8, 8, 8, 8, 8, 8, 8],
    "quasi_smooth_t": [10, 10, 10, 10, 10, 10, 10, 10, 10, 10],
    "hWVCaliBase": 120,
    "hWVCaliTop": 8000,
    "minSNRWVCali": 10,

    "clear_thres_par_beta_1064": 1e-8,
    "turbid_thres_par_beta_1064": 2e-7,
    "turbid_thres_par_beta_532": 2e-7,
    "droplet_thres_par_depol": 0.05,
    "spheroid_thres_par_depol": 0.07,
    "unspheroid_thres_par_depol": 0.2,
    "ice_thres_par_depol": 0.35,
    "ice_thres_vol_depol": 0.3,
    "large_thres_ang": 0.75,
    "small_thres_ang": 0.5,
    "cloud_thres_par_beta_1064": 2e-5,
    "min_atten_par_beta_1064": 10,
    "search_cloud_above": 300,
    "search_cloud_below": 100,

    "overlap532Color": [0, 255, 64],
    "overlap355Color": [0, 0, 255],
    "xLim_Profi_RCS": [0.3, 10],
    "xLim_Profi_Bsc": [-0.05, 3],
    "xLim_Profi_Ext": [-1, 150],
    "xLim_Profi_LR": [0, 120],
    "xLim_Profi_WV_RH": [0, 10],
    "xLim_Profi_WV_RH": [0, 8],
    "yLim_FR_RCS": [0, 20000],
    "yLim_NR_RCS": [0, 3000],
    "yLim_LC_355": [0, 1e14],
    "yLim_LC_532": [0, 2e14],
    "yLim_LC_1064": [0, 6e14],
    "yLim_LC_387": [0, 1e14],
    "yLim_LC_607": [0, 3e14],
    "yLim_WVConst": [0, 20],
    "yLim_depolConst_355": [0, 0.2],
    "yLim_depolConst_532": [0, 0.2],
    "zLim_att_beta_355": [0, 15],
    "zLim_att_beta_532": [0, 5],
    "zLim_att_beta_1064": [0, 2],
    "zLim_quasi_beta_532": [0, 2],
    "zLim_quasi_beta_1064": [0, 0.8],
    "zLim_quasi_Par_DR_532": [0, 0.4],
    "zLim_FR_RCS_355": [1e-2, 15],
    "zLim_FR_RCS_532": [1e-2, 15],
    "zLim_FR_RCS_1064": [1e-2, 15],
    "zLim_NR_RCS_355": [1e-2, 2],
    "zLim_NR_RCS_532": [1e-2, 2],

    "calibrationDB": "arielle_calibration.db",
    "logbookFile": "/data/level1a/polly/logbook_polly_Arielle_til_2018.csv"
}

Output

flagCorrectFalseMShots,1,flagFilterFalseMShots,0,flagForceMeasTime,0,flagDTCor,1,flagSigTempCor,0,flagLCCalibration,1,flagDepolCali,1,flagWVCalibration,1,flagUsePreviousDepolCali,1,flagUsePreviousWVconst,1,flagUsePreviousLC,1,flagUseManualRefH,0,flagUseSameRefH,0,flagAutoscaleRCS,1,flagMolDepolCali,0,flagTransCor,1,MWRFolder,,dataFileFormat,(?<year>\d{4})_(?<month>\d{2})_(?<day>\d{2})_\w*_(?<hour>\d{2})_(?<minute>\d{2})_(?<second>\d{2})\w*.nc,gdas1Site,cabauw,AERONETSite,Cabauw,max_height_bin,3000,first_range_gate_indx,261  261  261  261  261  261  261  261  262  262,first_range_gate_height,78.75,deltaT,30,dtCorModeLabel,[1],Polynomial coefficients stores in netcdf file,[2],nonparalyzable correction,[3],User defined polynomial cofficients,,dtCorMode,2,dt,0           0           0           0           0           0           0           0           0           0     0.97299      1.0117      0.9674      0.9929      0.9843      0.9391      1.0035           1           1           1   0.0035333     -0.0014      0.0023           0      0.0022      0.0063      0.0003           0      0.0029      0.0028 -7.9098e-06      0.0002           0      0.0001      0.0001     -0.0001      0.0001           0           0           0  1.0645e-07           0           0           0           0           0           0           0           0           0  1.4289e-09           0           0           0           0           0           0           0           0           0,bgCorRangeIndx,10  240,mask_SNRmin,1           1           1        0.01           1           1           1           1           1           1,tempCorFunc,[1],1,[2],1,[3],1,[4],1,[5],1,[6],1,[7],1,[8],1,[9],1,[10],1,[11],1,[12],1,[13],1,,init_depAng,19.99,depol_cali_mode,2,depol_cal_time_fixed_p_start,,depol_cal_time_fixed_p_end,,depol_cal_time_fixed_m_start,,depol_cal_time_fixed_m_end,,maskDepCalAng,[1],none,[2],none,[3],p,[4],p,[5],p,[6],p,[7],p,[8],p,[9],p,[10],p,[11],none,[12],none,[13],n,[14],n,[15],n,[16],n,[17],n,[18],n,[19],n,[20],n,[21],none,,depol_cal_minbin_532,40,depol_cal_maxbin_532,300,depol_cal_SNRmin_532,1  1  1  1,depol_cal_sigMax_532,1500  1500  1500  1500,rel_std_dplus_532,0.2,rel_std_dminus_532,0.2,depol_cal_segmentLen_532,40,depol_cal_smoothWin_532,8,depol_cal_minbin_355,40,depol_cal_maxbin_355,300,depol_cal_SNRmin_355,2  2  2  2,depol_cal_sigMax_355,1000  1000  1000  1000,rel_std_dplus_355,0.2,rel_std_dminus_355,0.2,depol_cal_segmentLen_355,40,depol_cal_smoothWin_355,8,depol_cal_minbin_1064,100,depol_cal_maxbin_1064,300,depol_cal_SNRmin_1064,1  1  1  1,depol_cal_sigMax_1064,2000  2000  2000  2000,rel_std_dplus_1064,0.2,rel_std_dminus_1064,0.2,depol_cal_segmentLen_1064,40,depol_cal_smoothWin_1064,8,isFR,1  1  1  1  1  1  1  1  0  0,isNR,0  0  0  0  0  0  0  0  1  1,is532nm,0  0  0  0  1  1  0  0  1  0,isRR,0  0  0  0  0  0  0  0  0  0,is355nm,1  1  0  0  0  0  0  0  0  0,is1064nm,0  0  0  0  0  0  0  1  0  0,isTot,1  0  0  0  1  0  0  1  1  0,isCross,0  1  0  0  0  1  0  0  0  0,isParallel,0  0  0  0  0  0  0  0  0  0,is387nm,0  0  1  0  0  0  0  0  0  0,is407nm,0  0  0  1  0  0  0  0  0  0,is607nm,0  0  0  0  0  0  1  0  0  1,is1058nm,0  0  0  0  0  0  0  0  0  0  0  0  0,channelTags,,channelTag,[1],FR-total-355 nm,[2],FR-cross-355 nm,[3],FR-387 nm,[4],FR-407 nm,[5],FR-total-532 nm,[6],FR-cross-532 nm,[7],FR-607 nm,[8],FR-total-1064 nm,[9],NR-total-532 nm,[10],NR-607 nm,,minPC_fog,60,TR,0.69           500             1             1          1.09           500             1             1             1             1,overlapCalMode,1,overlapCorMode,3,overlapSmoothBins,8,saturate_thresh,100,heightFullOverlap,500  500  500  500  500  500  500  500  150  150,cloudScreenMode,1,maxSigSlope4FilterCloud,3000000,maxSigSlope4FilterCloud_NR,400000,intNProfiles,120,minIntNProfiles,90,meteorDataSource,gdas1,flagUseLatestGDAS,1,radiosondeType,1,radiosondeSitenum,0,minDecomLogDist355,0.2,minDecomLogDist532,0.2,minDecomLogDist1064,0.2,maxDecomHeight355,8000,maxDecomHeight532,8000,maxDecomHeight1064,8000,maxDecomThickness355,700,maxDecomThickness532,700,maxDecomThickness1064,700,decomSmoothWin355,60,decomSmoothWin532,60,decomSmoothWin1064,120,minRefThickness355,300,minRefThickness532,300,minRefThickness1064,300,minRefDeltaExt355,1,minRefDeltaExt532,1,minRefDeltaExt1064,1,refH_FR_355,,refH_FR_532,,refH_FR_1064,,refH_NR_355,2500  3000,refH_NR_532,2500  3000,minRefSNR355,5,minRefSNR532,5,minRefSNR1064,4,minRefSNR_NR_355,10,minRefSNR_NR_532,5,LR355,50,LR532,50,LR1064,50,LR_NR_355,50,LR_NR_532,50,refBeta355,2e-08,refBeta532,1e-08,refBeta1064,2e-09,smoothWin_klett_355,21,smoothWin_klett_532,21,smoothWin_klett_1064,21,smoothWin_klett_NR_355,13,smoothWin_klett_NR_532,13,smoothWin_raman_355,61,smoothWin_raman_532,61,smoothWin_raman_1064,61,smoothWin_raman_NR_355,25,smoothWin_raman_NR_532,25,maxIterConstrainFernald,20,minLRConstrainFernald,1,maxLRConstrainFernald,150,minDeltaAOD,0.01,minRamanRefSNR355,50,minRamanRefSNR532,20,minRamanRefSNR1064,10,minRamanRefSNR387,20,minRamanRefSNR607,10,minRamanRefSNR_NR_355,50,minRamanRefSNR_NR_532,20,minRamanRefSNR_NR_387,40,minRamanRefSNR_NR_607,20,min_RR_RefSNR1058,0,angstrexp,0.9,angstrexp_NR,0.9,LCMeanWindow,50,LCMeanMinIndx,70,LCMeanMaxIndx,1000,LCCalibrationStatus,[1],none,[2],Klett,[3],Raman,[4],Defaults,[5],History,,quasi_smooth_h,8  8  8  8  8  8  8  8  8  8,quasi_smooth_t,10  10  10  10  10  10  10  10  10  10,IWV_instrument,AERONET,maxIWVTLag,0.0833,tTwilight,0.0347,hWVCaliBase,120,hWVCaliTop,8000,minSNRWVCali,10,clear_thres_par_beta_1064,1e-08,turbid_thres_par_beta_1064,2e-07,turbid_thres_par_beta_532,2e-07,droplet_thres_par_depol,0.05,spheroid_thres_par_depol,0.07,unspheroid_thres_par_depol,0.2,ice_thres_par_depol,0.35,ice_thres_vol_depol,0.3,large_thres_ang,0.75,small_thres_ang,0.5,cloud_thres_par_beta_1064,2e-05,min_atten_par_beta_1064,10,search_cloud_above,300,search_cloud_below,100,overlap532Color,0  255   64,overlap355Color,0    0  255,xLim_Profi_Bsc,-0.05           3,xLim_Profi_NR_Bsc,-0.1           20,xLim_Profi_Ext,-1  150,xLim_Profi_NR_Ext,-1  500,xLim_Profi_WV_RH,0  8,xLim_Profi_WVMR,0  50,xLim_Profi_RCS,0.3           10,xLim_Profi_LR,0  120,yLim_LC_355,0  100000000000000,yLim_LC_532,0  200000000000000,yLim_LC_1064,0  600000000000000,yLim_LC_387,0  100000000000000,yLim_LC_607,0  300000000000000,yLim_LC_355_NR,0  10000000000000,yLim_LC_532_NR,0  20000000000000,yLim_WVConst,0  20,yLim_FR_RCS,0  20000,yLim_NR_RCS,0  3000,yLim_FR_DR,0  15000,yLim_att_beta,0  15000,yLim_att_beta_NR,0  3000,yLim_OC_att_beta,0  15000,yLim_Quasi_Params,0  15000,yLim_WV_RH,0  7000,yLim_Profi_Ext,0  5000,yLim_Profi_LR,0  5000,yLim_Profi_DR,0  20000,yLim_Profi_Bsc,0  20000,yLim_Profi_WV_RH,0  7000,yLim_LC_ratio_355_387,0  2,yLim_LC_ratio_532_607,0  2,yLim_depolConst_355,0         0.2,yLim_depolConst_532,0         0.2,yLim_depolConst_1064,0  50,zLim_att_beta_355,0  15,zLim_att_beta_532,0  5,zLim_att_beta_1064,0  2,zLim_quasi_beta_355,0  8,zLim_quasi_beta_532,0  2,zLim_quasi_beta_1064,0         0.8,zLim_quasi_Par_DR_532,0         0.4,zLim_FR_RCS_355,0.01           15,zLim_FR_RCS_532,0.01           15,zLim_FR_RCS_1064,0.01           15,zLim_NR_RCS_355,0.01           2,zLim_NR_RCS_532,0.01           2,zLim_VolDepol_355,0         0.2,zLim_VolDepol_532,0         0.3,zLim_VolDepol_1064,0         0.3,colormap_basic,labview,PI,,PI_affiliation,,PI_affiliation_acronym,,PI_address,,PI_phone,,PI_email,,Data_Originator,,Data_Originator_affiliation,,Data_Originator_affiliation_acronym,,Data_Originator_address,,Data_Originator_phone,,Data_Originator_email,,comment,,logbookFile,/data/level1a/polly/logbook_polly_Arielle_til_2018.csv,radiosondeFolder,,calibrationDB,arielle_calibration.db,imgFormat,png,partnerLabel,,prodSaveList,[1],overlap,[2],aerProfFR,[3],aerProfNR,[4],aerProfOC,[5],aerAttBetaFR,[6],aerAttBetaOC,[7],aerAttBetaNR,[8],WVMR_RH,[9],volDepol,[10],quasiV1,[11],quasiV2,[12],TC,[13],TCV2,[14],cloudinfo,,
HolgerPollyNet commented 2 months ago

@cristoferjimenez @ZPYin Thanks a lot! 🥇 I will implement it the next days. Then, together with the PPC commit, we have a full reproducible processing chain!

ulysses78 commented 2 months ago

Looks good. One question concerning the deadtime-correction polynomial matrix (keyword: "dt").... at the moment the matrix is written channel-wise and polynomial-wise all together. This is very hard to read and distinguish in the end. Is it possible to output it with brackets, like: [polynom 1.channel][polynom 2.channel][polynom 3.channel] ....?

HolgerPollyNet commented 2 months ago

Hi all,

i implemented Zhenpings solution to the PPC, see https://github.com/PollyNET/Pollynet_Processing_Chain/tree/Polly_Config_Storing or the repsective pull request: Polly_Config_Storing

@ulysses78 @ZPYin @cristoferjimenez if you have a solution for @ulysses78 idea, please feel free to change the code and update it to the branch before we merge it with dev.