Data Assimilation Monitoring for RRFS

[terraladwig]

Problem

There is a need for data assimilation monitoring graphics for real-time RRFS runs in the short-term. EMC/GSL RRFS developers are discussing the best path forward... (1) Dust off some existing EMC scripts to get basic "o-f" histograms and observation counts plotted. (2) Leverage the capabilities this project is developing.

Solution

We can discuss the timeline for features from this project at the June 13 demo. If generating static images from netcdf diag files on NOAA HPC (Jet/WCOSS) is an option this summer (roughly), it would be great to take advantage of them. On the other hand, we don't want this request to distract or disrupt this project.

No Gos

(This is Terra's first issue and she is unclear if this item should be an issue or an email.)

[ian-noaa]

Thanks for opening this @terraladwig! I think opening subjects either here or in the project's GitHub Discussions works for us! If this turns into more of a discussion Evan & I can always move it over there. (And vice-versa)

Are the needs you're describing here similar to what's described in #11 for 3D-RTMA? If so, we might be able to build on that work if we're able to get NetCDF versions of the RRFS diag files in AWS. They'd be viewable from a website and not as static imagery from the NOAA HPC systems.

Evan may have some thoughts here but if it is essential to have these tools in the short term, it'd probably make sense to dust off the existing EMC scripts so we don't block anyone's work.

[terraladwig]

Yes, it would be the same functionality described in #11. As you said, the added effort would be to access/transfer RRFS diag files and distinguishing the visualization as RRFS, not RTMA (maybe a separate website or a model selector?).

[esheehan-gsl]

@terraladwig is it a requirement that we generate static images, or would it be sufficient if people can load up the graphics in their browser as-needed, provided that NetCDF files remain available?

I can imagine a couple of reasons that static image would be a requirement:

• People need/want to keep copies of the graphics on their own computer as an archive (in case the NetCDF files age off of our S3 bucket)
• The graphics need to be used in presentations
• Some graphics need to be shared with people who would not have access to the system we’re building

I ask because making graphics downloadable for any of the above reasons might make the difference between being able to meet this summer (roughly) deadline or not. If…

• We don’t have to make images available as downloads
• Someone on RRFS is able to help us upload the NetCDF to S3
• The NetCDF files for RTMA and RRFS are similar (preferably identical)

it should be reasonable for us to provide labels and controls that support selecting one model or the other for your displays and appropriately labeling the output so that it’s obvious which model’s diagnostics you’re seeing.

But if static files are a real requirement, or if the diag files are substantially different, it might be best to revive your old EMC scripts.

[terraladwig]

I will need to ask a larger sample of developers to provide firm answers. But I can offer my perspective for now.

Static images will eventually be a requirement, for the reasons you suggested. But I think simply taking a screenshot would be sufficient for initial use.

Yes, the RRFS team (myself or others) could help with the transfer of RRFS diag files to S3.

Yes, the netcdf files for RRFS should be nearly identical to RTMA. I can verify that is the case.

[esheehan-gsl]

In that case, I’d say we can include this use case for you. Once the files are in S3, it shouldn’t be too much additional work beyond what we’re already working on for RTMA. I think it’s worth trying for, because it would be good to give people a reason to start using the tool so that they can help us improve the design based on actual use.