Update AirNow converter

weiwilliam commented 3 months ago

Description

This PR updates the converter for AirNow observations with,

Add land use to variable airqualityClassification (0=UNKNOWN, 1=RURAL, 2=SUBURBAN, 3=URBAN AND CENTER CITY) based on a full site list provided by EPA
Users can define whether the site list contains the land use information by giving --epa_list argument
multi-files capability

Issue(s) addressed

Resolves #1297

Dependencies

No dependencies

Impact

Expected impact on downstream repositories:

Checklist

[ ] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation
[ ] I have run the unit tests before creating the PR

weiwilliam commented 3 months ago

@jeromebarre and @mer-a-o Is surfaceQualifier good to store land use information?

mer-a-o commented 2 months ago

@weiwilliam I rebuilt the code and the error went away. But I'm getting a different error:

66: Processing infile= testinput/airnow_2020081306.dat
66: DIFFER : NUMBER OF GLOBAL ATTRIBUTES : 7 <> 6
66: DIFFER : VALUES OF GLOBAL ATTRIBUTE : source : "https://files.airnowtech.org/?prefix=airnow/" <> "Unknown (ftp)"
66: DIFFER : NAME OF GLOBAL ATTRIBUTE : sourceFiles : GLOBAL ATTRIBUTE DOESN'T EXIST IN "testoutput/airnow_2020081306.nc"
66: DIFFER : VARIABLE : carbonmonoxideSurface : DOES NOT EXIST IN "testoutput/airnow_2020081306.nc"
66: DIFFER : VARIABLE : nitrogendioxideSurface : DOES NOT EXIST IN "testoutput/airnow_2020081306.nc"
66: DIFFER : VARIABLE : sulfurdioxideSurface : DOES NOT EXIST IN "testoutput/airnow_2020081306.nc"
66: DIFFER : VARIABLE : carbonmonoxideSurface : DOES NOT EXIST IN "testoutput/airnow_2020081306.nc"
66: DIFFER : VARIABLE : nitrogendioxideSurface : DOES NOT EXIST IN "testoutput/airnow_2020081306.nc"
66: DIFFER : VARIABLE : sulfurdioxideSurface : DOES NOT EXIST IN "testoutput/airnow_2020081306.nc"
66: DIFFER : VARIABLE : carbonmonoxideSurface : DOES NOT EXIST IN "testoutput/airnow_2020081306.nc"
66: DIFFER : VARIABLE : nitrogendioxideSurface : DOES NOT EXIST IN "testoutput/airnow_2020081306.nc"
66: DIFFER : VARIABLE : sulfurdioxideSurface : DOES NOT EXIST IN "testoutput/airnow_2020081306.nc"
1/1 Test #66: test_iodaconv_airnow .............***Failed    1.51 sec

weiwilliam commented 2 months ago

@mer-a-o Yes, the testoutput is not updated, because I would like to know whether surfaceQualifier is good to use. The current usages are 0=land, 1=sea, and 2=seaice.

mer-a-o commented 2 months ago

How about stationType: https://github.com/JCSDA-internal/ioda/blob/develop/share/ioda/yaml/validation/ObsSpace.yaml#L1075?

weiwilliam commented 2 months ago

I also considered that, but the description said "type of station such as ASOS, AWOS". Then it would be "AirNow" in this case. I use "surfaceQualifier" because there is another variable "earthSurfaceType" noted with "better for future".

@BenjaminRuston Can you suggest a variable that fits "land use" information better, such as "rural" and "urban"? Thanks.

jeromebarre commented 2 months ago

replace surfaceQualifier by airqualityClassification i.e. https://www.eea.europa.eu/en/topics/in-depth/air-pollution/monitoring-station-classifications-and-criteria

weiwilliam commented 2 months ago

@jeromebarre Thank you! I will update it and address Maryam's comments.

weiwilliam commented 2 months ago

We will need to add the following variables into ObsSpace.yaml for validation:

MetaData/airqualityClassification
[ObsValue, ObsError, PreQC]/particulatematter2p5Surface
[ObsValue, ObsError, PreQC]/ozoneSurface
[ObsValue, ObsError, PreQC]/carbonmonoxideSurface
[ObsValue, ObsError, PreQC]/nitrogendioxideSurface
[ObsValue, ObsError, PreQC]/sulfurdioxideSurface

mer-a-o commented 2 months ago

@weiwilliam Do the airnow tests pass for you? Do you need to update the reference files?

weiwilliam commented 2 months ago

@mer-a-o I put the csv file in the wrong place, and I will update the variables in original testoutput.

mer-a-o commented 2 months ago

@weiwilliam I'm looking at testrun/airnow_2020081306.nc why all the values are 0.1 for all the measured species?

weiwilliam commented 2 months ago

@mer-a-o let me check.

weiwilliam commented 2 months ago

@mer-a-o were you checking ObsError? ObsError is filled with 0.1 at all data points.

mer-a-o commented 2 months ago

@weiwilliam Yes!! Sorry for the false alarm.

jeromebarre commented 2 months ago

@mer-a-o were you checking ObsError? ObsError is filled with 0.1 at all data points.

This is because Airnow doesn't provide the obs error. Should we set the error to a fraction of the measured quantity? The fraction should change for each species and class I think? In the future, if we want to assimilate air now it would be good to do a little bit of research to see if in the literature there is a way to estimate the error for DA and implement it in the converter. If this doesn't exist this is a little project that we should carry forward. This is important for regional AQ DA.

BenjaminRuston commented 2 months ago

not particularly satisfied with the state of surfaceQualifier as in the ioda validation yaml, and this PR is perfectly valid but the values are ambiguous as to what they refer to:

https://github.com/JCSDA-internal/ioda/blob/d49ed17e115f5d7ff3310cd1dfe4798c10aa03db/share/ioda/yaml/validation/ObsSpace.yaml#L1274C1-L1274C80

  - Variable: [ "surfaceQualifier", "surface_type" ]  # 0=land, 1=sea, 2=seaice

weiwilliam commented 2 months ago

@BenjaminRuston Sorry, I forgot to update the PR description at the top. We are using airqualityClassification now.

BenjaminRuston commented 2 months ago

@BenjaminRuston Sorry, I forgot to update the PR description at the top. We are using airqualityClassification now.

yes seeing that... the airqualityClassification is not in the IODA standards we need to add this and am changing it to airQualityClassification

will update this PR, please note this change and lets use this moving forward

BenjaminRuston commented 2 months ago

@weiwilliam updated the name please review and let me know if this work

we will need to update: share/ioda/yaml/validation/ObsSpace.yaml

please go ahead and put a PR into IODA to add this new Qualifier and we should have folks review across agencies

BenjaminRuston commented 2 months ago

@weiwilliam we should talk there's likely a JEDI integration test expecting this psuedo kind-of like a camel case name airqualityClassification ... but this is another thing would like to fix as:

and though I completely understand that this is an "air-quality" Classification.... camelCase is agnostic to the words and would like to just follow and not redefine camel case with new entries

JCSDA-internal / ioda-converters