evaluation and fix of surface emissivity adjoint in CRTMv3

[mingchen36]

@gmao-yzhu, @BenjaminTJohnson , @MingjingTong-NOAA , @emilyhcliu , @gmao-wgu , @quanhualiu, @mingchen36

I just noticed that the Common_RTSolution.f90 in this repository had already tried to adopt some of the "fix" codes from the release 2.4.x, where I highlighted the importance to discern the surface emissivity adjoint for use in the 1DVAR retrieval and the DA QC. Someone made a code structure adjustment in this V3 using different order of the IF statement blocks (IF SfcOptics%Compute vs .NOT. SfcOptics%Compute). This should not a problem logically. BUT it lost my fix of the surface emissivity adjoint for the SfcOptics%Compute case. Consequently, this version missed the key point I highlighted in the release 2.4.x.

CRTM_Surface_Emissivity_Jacobian_Final.pdf

I already created one branch hotfix/cm_surface_EmissAD with the updated Common_RTSolution.f90 and the FWD-TL-AD consistency testing programs.

Mark, can you add your testing results for the NOT. SfcOptics%Compute case?

[quanhualiu]

Ming, I didn't find the comparison table between K-matrix surface emissivity Jacobian and the finite difference surface emissivity Jacobian from your attached pdf file. For the case users input surface emissivity under scattering case, I provided the comparison table. For using CRTM MW ocean emissivity case even under clear-sky condition, it is complicated.

[mingchen36]

Mark, @gmao-yzhu, @BenjaminTJohnson , @MingjingTong-NOAA , @emilyhcliu , @gmao-wgu , @quanhualiu, @mingchen36 Instead of printing the example testing result, I provided the unit testing programs so that the user may use these codes as a template to test their own profile inputs.

I have a question on your Table. Mathematically, the FWD-TL-AD consistency testing need to provide two separate and systematic tests: the consistency between the FWD and TL, and the consistency the TL and AD (K-matrix). Your table just shows some kind of the consistency between TL and AD(K-matrix). So the Finite-Difference column is TL values? Not sure how you configured your testing? Can you share your testing codes?

Also, there are two conditions in the User's Input case, cloudy scattering vs clear-sky, can you attach also the testing result of the clear-sky?

[mingchen36]

@gmao-yzhu, @BenjaminTJohnson , @MingjingTong-NOAA , @emilyhcliu , @gmao-wgu , @quanhualiu, @mingchen36 Attached is the example FWD-TL-AD consistency testing results over ocean surface, where the emissivity is calculated from the physics model. Please note this testing is on clear-sky condition and with respect to the surface wind speed. As I mentioned in the meeting, the RTSolution%Surface_Emissivity is just the component at the sensor view angle. On cloudy scattering case, the general surface emissivity is a complex matrix of many other control variables at multiple angles. The FWD-TL-AD consistency of this general surface emissivity matrix was tested and already documented in the previous PDF file attached earlier, which ensured the completeness in the entire FWD_TL_AD_tests.pdf code consistency. The FWD-TL-AD consistency will not hold for single component.

[mingchen36]

typo error in the PDF: Ratio =Emiss_Finite-Differ / Emiss_TL Criteria for success/fail: Ratio -> 1.0 as wind_speed finite difference -> 0.0

[quanhualiu]

Did you use CRTM_Forward_Module.f90, CRTM_Tangent_Linear_Module.f90, and CRTM_K_Matrix_Module.f90 and got the results in FWD_TL_AD_test.pdf ? Users use CRTM_Forward_Module.f90, CRTM_Tangent_Linear_Module.f90, and CRTM_K_Matrix_Module.f90 and we need to test those code for the surface emissivity Jacobian.

If yes, which CRTM version I can use for the test?

[mingchen36]

Mark, you may use the branch hotfix/cm_surface_EmissAD. After your testing, I may create pull request. I use the adjoint module. K-matrix is the collection of the variables from the adjoint.

[quanhualiu]

Ming, The finite difference (4th column) in my table is the difference between two forward calculations divided by the surface emissivity perturbation. I need to add a column for TL using the latest CRTM version with the final change for surface emissivity Jacobian. I can see the difference between K-matrix and "finite difference" when I print out more digital numbers. Is there a test code in the branch/cm_surface_EmissAD for testing "finite difference" against K-matrix? The comparison in FWD_TL_AD_tests.pdf CH Emiss_TL Emiss Finite-Differ Ratio is for partial difference pd(emissivity)/pd(wind_speed) or for pd(Radiance)/pd(emissivity)? We also need the test for pd(Radiance)/pd(emissivity)

[quanhualiu]

The TLAD test code tests pd(Radiance)/pd(emissivity), right?

[mingchen36]

In short, neither pd(Radiance)/pd(emissivity), nor pd(emissivity)/pd(wind_speed), nor pd(Radiance)/pd(wind_speed) although it is so straightforward to get the pd(Radiance)/pd(wind_speed) from this test. K-matrix is nothing more than a transpose of the jacobian. It is essential to know the difference where a variable (e.g., surface emissivity) is a control variable or is a intermediate variable. Otherwise, it should be very difficult to understand the related testing algorithms.