ncsa / ameriflux-pipeline

AmeriFlux Pipeline Repository
5 stars 0 forks source link

Add QA/QC in the code #80

Closed ywkim312 closed 1 year ago

ywkim312 commented 2 years ago

Add QA/QC in the code. The user should be able to turn this option on/off probably by using the config file like,

QC_PERFORM = os.getenv(QC_PERFORM , 'True') and also put the variable in .env file as well

This will do the date exclusion, for example, for the precipitation, the data with the precipitation value is lower and upper value as 0.0 and 0.2, then those will be exclude for output

Making for other dataset, we can make a separate text file that contains the information of the variable name and high/low value for each month that is extracted out from L2.txt. Or, we can directly use L2.txt and parse this. If the user doesn't need to change the values in the L2.txt, we can directly use L2.txt by parsing the values in there and apply those to the code. L2.txt contains the variable names and the high/low values for each month, so this should work.

minump commented 2 years ago

This QA/QC check reflects the RangeCheck done by L2. The mapping from met tower variables (Met_data_30) and eddypro labels(full_output) to pyfluxpro variable names is found in L1 under the [[[xl]]] tag. Use -9999 as fill value if value is outside the range. The automation stages after this would be master met > eddypro > QAQC check > L2_proposed.txt The current automation would remain the same. master met > eddypro > pyfluxpro

ywkim312 commented 1 year ago

This is implemented partially.