search

JCSDA-internal / ioda-converters

Various converters for getting obs data in and out of IODA
9 stars 4 forks source link

Subset profile data into _m and _s files #153

Closed CoryMartin-NOAA closed 5 years ago

CoryMartin-NOAA commented 5 years ago

@louiskouvaris needs a full profile for his ozone UFO operator. For a test, we'll need a small subset of data, which can be done manually, but this may be needed for other applications. How to best do this for when we may not know the size of the profile in an automated script. Is this when we will need to use record_number rather than forcing it to be 1 as @svahl991 pointed out earlier? If so, how do we get record_number from the GSI diag files?

shlyaeva commented 5 years ago

For radiosondes and aircrafts I was using station_id to put all observations with the same station_id into the subset files. I'll look into GSI ozone files to see if there are any ids there that we can use.

CoryMartin-NOAA commented 5 years ago

Ah yeah, I don't think I'm doing that properly for radiosondes and aircraft in subset_files.py, so that will need to be changed for all of these types. I guess this brings up the question though of record_number and if that should be used somehow too? Perhaps we'll wait until @srherbener is back to clarify the intentions of record_number.

shlyaeva commented 5 years ago

@louiskouvaris: From reading GSI code (setupozlay routine), it looks like each ozone observation is processed separately. However, to compute H(x) for each of those observations GSI is using at least two variables, "apriori" and "efficiency" (metadata, I guess?), which are arrays size nloz_omi. I am not familiar with ozone observations, and I'd like to understand this better:

LouisKouvaris-NOAA commented 5 years ago

I haven't looked at OMI yet but for sbuv we need the pressure at the top of the layer and at the bottom. The bottom pressure level is the top of the layer below it. Therefore we need two pieces of metadata per observation for ozone layer instruments. This is not conveyed presently but is an option for computing hofx.

shlyaeva commented 5 years ago

@louiskouvaris: do you know what those pieces of metadata are named in GSI in setupozlay? we can add code to GSI to output them in the netcdf diagnostics files (that we use to generate ioda obs files).

LouisKouvaris-NOAA commented 5 years ago

Reference pressure for that observation gives the top layer pressure. And reference pressure for the next observation is the bottom pressure level of the previous layer. We need both pieces of information for each observation layer.

⁣Sent from BlueMail ​

On May 2, 2019, 3:15 PM, at 3:15 PM, Anna Shlyaeva notifications@github.com wrote:

@louiskouvaris: do you know what those pieces of metadata are named in GSI in setupozlay? we can add code to GSI to output them in the netcdf diagnostics files (that we use to generate ioda obs files).

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/JCSDA/ioda-converters/issues/153#issuecomment-488796567

srherbener commented 5 years ago

The purpose of the record number is to group observations into "atomic" units of which don't get broken apart when distributing across multiple process elements. Typically, a record corresponds to a unique sounding, or profile so it seems that the record number in this case could be based on unique profiles and then used to extract a single profile for testing.

See issues #151 and #152 for more details about grouping the records (and locations) together and using the NcWriter class.

CoryMartin-NOAA commented 5 years ago

@louiskouvaris can you determine a way to identify separate ozone profiles in setupoz.f90 in GSI? It would be easier if it's written out to the netCDF files directly (like even something simple like a loop index would be fine), rather than searching for where reference pressure equals bottom pressure. I'll use station_ID for sondes and aircraft but will need something to use for the ozone profiles.

CoryMartin-NOAA commented 5 years ago

addressed by PR #187