Direct conversion of NCEP BUFR data to IODA (skipping GSI processing)

[CoryMartin-NOAA]

Sooner rather than later, we need to have the abillity to take NCEP BUFR data files that are available operationally (and archived) as part of the GFS/GDAS dump, and convert them directly to IODA netCDF files (or ODC, whatever will be used) and bypass processing using the GSI observer.

[CoryMartin-NOAA]

@srherbener are there any plans to make the NcWriter class available for C++ (or is it theoretically possible to just use the NetcdfIO.cc file from IODA?) to develop a compiled executable to do this task. Thinking ahead (in the not too distant future) of operations, I think that Python scripts will have too much overhead to do the BUFR to IODA conversion in real time.

[srherbener]

@CoryMartin-NOAA, I have it on my list to create a Python and Fortran (the BUFR parser is written in Fortran) interface for the IodaIO class (of which the NetcdfIO class is a subclass). I agree that we will likely need a compiled executable to process the operational BUFR files, and this work will allow us to continue using Python and add the capability to write a converter all in Fortran. (We wouldn't need the Fortran interface if there is a good C/C++ BUFR parser.)

I'm thinking of keeping the NcWriter class and just replacing a portion of it with the IodaIO interface.

I have a couple other things on my to do list that are higher priority for now. When do you think this will be needed?

[CoryMartin-NOAA]

@srherbener I believe the plan is for the fall to have the UFO working as part of the GSI EnKF as the first piece to test for operations. I don't know exactly how soon we will need to have this piece ready (can be something to discuss in greater detail during the UFO code sprint in a couple of weeks).

If you're referring to the NCEP BUFR library written in F77, there is an alternative keeping with the theme of borrowing from the European Centre: https://confluence.ecmwf.int/display/ECC/What+is+ecCodes. It has a C, F90, and Python API to read/write BUFR and GRIB, and uses ecbuild, etc. so it might be a good option to pursue. Then you could probably skip the Fortran interface if this works. I'm planning on looking into eccodes a bit more in the coming week or so to hopefully understand better if it will work for this (and potentially other) needs.

[shlyaeva]

For the UFO as part of GSI EnKF, my first iteration plan was:

• run GSI observer on the ensemble mean
• convert GSI diags to ioda files
• use these files to run UFO for the ensemble members either:
• EnKF runs with ioda file[s] output from ufo or
• converter is run to convert ioda files to GSI diags format (one probably doesn't even need to convert the whole file, just copy H(x) result)
• EnKF runs with GSI diags.

[shlyaeva]

then, when QC and bias correction are fully ready in UFO, one could skip GSI observer completely and then we'd definitely need converters from bufr to ioda.

but I agree it's best to get started on the converters soon.

[CoryMartin-NOAA]

@shlyaeva with your first iteration plan I still fear that the current Python-based ioda-converters for GSI diags to IODA are a non-starter operationally (but I'm not an expert on implementations), both from the dependencies of Python modules that are not available on WCOSS currently, and the computational cost. This plan should work but I think there would then have to be a compiled converter from GSI to IODA and then another (or same with a different runtime option) converter to go from UFO back to GSI.

[ytremolet]

Would it be difficult to write a IODA file from GSI? And then run the EnKF from IODA files? Would that be more work than developing the converters? By how much?

[CoryMartin-NOAA]

I suspect it will be a similar amount of work to do the modification to GSI as it would be to write a separate converter. However, I'm not sure that "forking" GSI and rewriting the ncdiag routines is the best way to go. I'm not sure if there are any users of the current formatted files, for example.

[shlyaeva]

I suspect this will be a lot of work because of conventional obs. GSI would be able to write out all temperature obs in one file, and all wind obs in one file. UFO needs all radiosonde obs together, all aircraft obs together, etc.

[CoryMartin-NOAA]

@shlyaeva good point. This is probably a good argument for doing a converter. Even the radiances would have to go from brightness_temperature_NNN to "Observation" with a new channel_index field.

[srherbener]

I think no matter what we do, the interface to the IodaIO class is a key piece. We would want that for Python scripts, NCEP's Fortran library, eccodes C++ library, and a GSI to IODA file converter.

Should the interface to IodaIO be a higher priority than it is now?

I think we should be working on this sooner rather than later. At least have some discussion and get an initial schedule put together. I agree with @CoryMartin-NOAA that the current python converters have performance issues - fine for test data, not so good for operational data. And I agree with @shlyaeva's point about collecting up the obs (by location) for the conventional data. That's a big part of what the Python converters are doing.

[shlyaeva]

EnKF only needs to read in obs and metadata, and I don't foresee problems with adding read of ioda files in EnKF (it should be very similar to reading ncdiag, with some caveats of brightness_temperature_NNN instead of Observation). I might be missing something here though.

[CoryMartin-NOAA]

@shlyaeva no I agree I think the EnKF read part will be much easier than the high-performance write to IODA part. @srherbener I don't know if that collection of obs will be a big issue or not, as that is really just a design choice for combining all T,P,Q,U,V into one file. Is there any reason why IODA/UFO can't just read in sondes_T, sondes_uv, etc. for this real-time application?

[CoryMartin-NOAA]

Also, @srherbener IMO I think the IodaIO should be a high priority because I don't think there's a possibility of getting this running with GSI on WCOSS without compiled converters. There's something like 20 mins allowed total for GSI to run (not sure about the GDAS EnKF), but every second will count, particularly for GFSv16 with the 127 layer model.

[shlyaeva]

@CoryMartin-NOAA: agree on a possibility of using sondes_T, sondes_uv, etc.

[srherbener]

IODA is in a prototype stage now and we, of course, want to move it toward production quality this year. Discussions like this are very important and helpful with this goal, so I appreciate this one.

The memory store in IODA needs to be efficient for radiance data (at the potential expense of conventional data storage) since the amount of radiance data far exceeds that of conventional data. This lead to the organization of obs data looking like a table that has channels on one axis and unique locations on the other axis. We don't want to repeat locations since that would be a large amount of redundant metadata in the radiance cases.

When I started at UCAR, the system instantiated a specific (subclassed) ObsSpace object for each ObsType and each of these objects had its own memory store and reader. We recognized that many, if not all, ObsTypes could be cast into the design we have now which is one generic ObsSpace with one reader. We want to avoid having to create separate ObsSpaces specific to each ObsType.

In the current system after an ObsSpace is constructed, it is queried by other object constructors on the number of unique locations which becomes "nlocs" in the system. "nlocs" is then used for these other constructors.

The above are the design decisions that lead to requiring the alignment of the T,Q,U,V by locations. This isn't as efficient as it could be (introduces missing data), but that's okay. Python was chosen to do the location alignment simply because it was quick and easy for prototyping. I think this function could be pulled into IodaIO. This seems like it would be a good thing for streamlining the whole observation handling system (both inside and outside IODA).

I'm not sure I understand the need for compiled converters when running GSI on WCOSS (does running on WCOSS mean GSI operational runs, for example?). GSI already has it's compiled converters for ingesting observations, and won't we be running JEDI off-line from the GSI run. Is the concern that the ncdiag writer as it stands now is going to be too slow. Or are you saying that having GSI write IODA files directly won't have enough time to execute unless that conversion is compiled code.

[CoryMartin-NOAA]

Hi Steve,

The concern here is two fold.

The main problem is the converters as they are now are too slow. In operations, on time products are key. So for us to even test this with the operational workflow, we cannot take more than a couple minutes longer than it is now to do everything extra that needs to be done for processing. If we are taking several minutes to process in each direction, it’ll be a non starter from NWS’s perspective (IMO)

Second, NCO is the institution who we (EMC) hand code off to and they are who make sure the trains run on time. They have strict rules and guidelines and are very choosy about what goes on the machines for fear of compatibility/stability issues because their concern is not that the output is correct but that it exists and on time. So I also fear about the use of high level python libraries, etc. that might not be easily added to the operational environment because we might not be able to rely on them being available.

Regarding the data structure I agree that radiance data is king and we build around that. I like the idea of building locations into IODAIO because it could help for cases (can’t think of any off the top of my head) where obs come in from different products/sources but are physically colocated (perhaps met obs with air quality observations?)

Sorry about the short replies, on my phone. Have a great weekend!

On Friday, August 2, 2019, Stephen Herbener notifications@github.com wrote:

IODA is in a prototype stage now and we, of course, want to move it toward production quality this year. Discussions like this are very important and helpful with this goal, so I appreciate this one.

The memory store in IODA needs to be efficient for radiance data (at the potential expense of conventional data storage) since the amount of radiance data far exceeds that of conventional data. This lead to the organization of obs data looking like a table that has channels on one axis and unique locations on the other axis. We don't want to repeat locations since that would be a large amount of redundant metadata in the radiance cases.

When I started at UCAR, the system instantiated a specific (subclassed) ObsSpace object for each ObsType and each of these objects had its own memory store and reader. We recognized that many, if not all, ObsTypes could be cast into the design we have now which is one generic ObsSpace with one reader. We want to avoid having to create separate ObsSpaces specific to each ObsType.

In the current system after an ObsSpace is constructed, it is queried by other object constructors on the number of unique locations which becomes "nlocs" in the system. "nlocs" is then used for these other constructors.

The above are the design decisions that lead to requiring the alignment of the T,Q,U,V by locations. This isn't as efficient as it could be (introduces missing data), but that's okay. Python was chosen to do the location alignment simply because it was quick and easy for prototyping. I think this function could be pulled into IodaIO. This seems like it would be a good thing for streamlining the whole observation handling system (both inside and outside IODA).

I'm not sure I understand the need for compiled converters when running GSI on WCOSS (does running on WCOSS mean GSI operational runs, for example?). GSI already has it's compiled converters for ingesting observations, and won't we be running JEDI off-line from the GSI run. Is the concern that the ncdiag writer as it stands now is going to be too slow. Or are you saying that having GSI write IODA files directly won't have enough time to execute unless that conversion is compiled code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JCSDA/ioda-converters/issues/202?email_source=notifications&email_token=ABQPN3GBDNDQEGGGW7H7CCDQCSRT3A5CNFSM4IIIHCD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3O4JUQ#issuecomment-517850322, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQPN3FJ6KDMHVBHZIPXM6LQCSRT3ANCNFSM4IIIHCDQ .

--

Cory R. Martin, PhD Support Scientist, RedLine Performance Solutions, LLC Modeling and Data Assimilation Branch, Environmental Modeling Center (EMC) National Centers for Environmental Prediction (NCEP) NOAA National Weather Service 5830 University Research Court, Room 2767 College Park, MD 20740 cory.r.martin@noaa.gov 301-683-1524

[CoryMartin-NOAA]

https://www.nco.ncep.noaa.gov/idsb/implementation_standards/Implementation%20Standards%20v10.2.pdf Just adding the NCO implementation standards document here for reference since I mentioned they have rules/guidelines on operational systems.

[srherbener]

@CoryMartin-NOAA, thanks for the information!

I think my confusion is coming from my thinking that the UFO in GSI EnKF mode would be running outside of production runs, and I'm guessing that you are thinking of this being run inside of production runs. I'm coming up to speed on this topic, and my view of this being run outside production may not be in alignment with the way anyone else is viewing this. I apologize if this is causing confusion.

If we are indeed running the UFO in GSI EnKF flow in production mode, I agree that there is not much "spare" time for obs data conversion, the current python based solution is way too slow, and something needs to be done about this relatively soon.

I've got a couple high priority items that I have to focus on for now. The UFO/QC code sprint starts in two weeks, and Anna is on vacation until just before then (through Aug 20th). It might make sense to defer this discussion until the code sprint where we can focus on getting the solution in place.

If you feel that there is something urgent that can't wait for the code sprint, please feel free to raise that concern so we can help get it addressed.

Thanks!

[CoryMartin-NOAA]

@srherbener No worries at all. Even if it is running outside of production (a parallel run), it will have to be set up to mimic what will happen in operations and tested for several weeks unattended before it can be cleared for transitioning to production.

I don't think we need to have anything done before Aug 20, so we can just have a more serious discussion about this during the code sprint. I just mainly wanted to bring this and the other issue (with the UFO output format for the EnKF) up sooner rather than later because the first NOAA milestone is this UFO in GSI EnKF step, and even when all of the JEDI pieces are ready, then there are some workflow/technical issues that will have to be tackled before it can be cycled successfully. In the meantime, I'm spending a bit of time here and there trying to learn more about the NCEP BUFR Fortran library so that once you have a Fortran IodaIO interface, we can have a stand alone converter.

[srherbener]

@CoryMartin-NOAA, sounds good and thanks for being flexible. I agree with bringing this up now, so thank you for doing that.

We are certainly open to using eccodes. It is open source and already has C++, Fortran and Python API's which makes it an attractive option. I think it's well worth investigating so I just want to let you know that we are not tied to the NCEP Fortran library.

[CoryMartin-NOAA]

@srherbener I had some issues getting eccodes to compile on Theia (but not in the singularity container, it worked there), however I'm sure it's just user error (and I didn't try very hard). I agree it's the more attractive option but using the NCEP BUFR might be the way to go if the short-term goal is getting it to run "operationally"

[ytremolet]

Can we close this? I think it is worked on elsewhere.

[aerorahul]

331 partially resolves this

[CoryMartin-NOAA]

and issue #350 will be closed by #331 , so I'll close this old issue.