v2023-06-BCs, drop TROPOMI albedo filters, and Rocky 8 on Cannon

[nicholasbalasus]

Name and Institution

Name: Nick Balasus Institution: Harvard University

Describe the update

This update does the following:

• Allows the IMI to run on Cannon after the upgrade from CentOS 7 to Rocky 8 (used Bob's new environment files from #114 but left our imi_env.yml as is).
• Removes the albedo filters for TROPOMI data.
• Updates the boundary condition generation scripts so that if you generate boundary conditions up until day x, TROPOMI and GC data is used until day x+15 to make our temporal smoothing valid (and to make the BCs more reproducible). I generate out to 20 days past just to be careful (and because we will probably be doing this on a monthly basis).

[nicholasbalasus]

Comments addressed @laestrada

[sabourbaray]

Hi, I'm curious what the reasoning is for removing the albedo filters? Are there new recommendations for filtering? Thanks.

[djvaron]

Hi, I'm curious what the reasoning is for removing the albedo filters? Are there new recommendations for filtering? Thanks.

Hi Sabour, thanks for the question. There are a few reasons.

The latest TROPOMI product deals better with spectral variability in albedo and includes a post-processing albedo correction for low-albedo errors. And since the QA filtering already removes retrievals with very low SWIR albedo (a < 0.02 iirc), SRON recommended removing the additional SWIR filter we previously used (a < 0.05).

For the blended albedo filter (ba > 0.85), Lorente et al. (2021) showed that it removes retrievals in snowy scenes that would be subject to artifacts, but we've also seen it remove non-snowy retrievals at different latitudes and times of year. We aren't aware of any careful analysis of the filter's global effects, so we decided to remove it when constructing our global boundary condition fields.

What do you think? Do you have concerns about this for applications over Canadian domains? We're open to suggestions!

[sabourbaray]

Hi Daniel,

Thanks for the update.

The latest TROPOMI product deals better with spectral variability in albedo and includes a post-processing albedo correction for low-albedo errors. And since the QA filtering already removes retrievals with very low SWIR albedo (a < 0.02 iirc), SRON recommended removing the additional SWIR filter we previously used (a < 0.05).

That makes sense. In our case, allowing data with SWIR albedo 0.02 < a < 0.05 will give us more data to work with in upper latitudes.

For the blended albedo filter (ba > 0.85), Lorente et al. (2021) showed that it removes retrievals in snowy scenes that would be subject to artifacts, but we've also seen it remove non-snowy retrievals at different latitudes and times of year. We aren't aware of any careful analysis of the filter's global effects, so we decided to remove it when constructing our global boundary condition fields.

What do you think? Do you have concerns about this for applications over Canadian domains? We're open to suggestions!

This is helpful to know. I can include some sensitivity inversions in my work with and without the blended albedo filter. In my fork (under operational-eccc/v1.2), I'm using the surface classification to experiment with filters:

& ((tropomi_data["surface_classification"].astype(int) & 0x03) == 0)

This is likely too aggressive, but it helps us test without data that is contaminated by small lakes scattered in the domain. I think an ideal setup would be something in between, where we can select for data that is >80% land. This has been explored by de Foy et al. (2023), but it involves creating a high resolution water mask.

The surface classification variable also has a bit for "Land+Snow_Or_Ice", so something like:

& ((tropomi_data["surface_classification"].astype(int) & 0xF9) != 184)

Could also work.