I think I’ve found a bug in the gradient calculation. This affects both the polarized and the unpolarized pyshdom optimization of cloud microphysics.
One component of the gradient is calculated from the single-scatter contribution to the radiance. The single-scatter contribution to the radiance is calculated from a tabulation of phase functions as a function of scattering angle (dphasetab). These are computed in PRECOMPUTE_PHASE_CHECK (Line 1162 unpol, Line 1607 Pol, shdomsub4.f) using the legendre coefficients of the phase functions (LEGEN). The same subroutine is used to compute the derivatives of the phase function at each scattering angle by replacing LEGEN with the derivatives of the legendre coefficients (dlen). These two uses of PRECOMPUTE_PHASE_CHECK are at Line 988 rte_solver.py and Line 928 optimize.py, respectively.
However, PRECOMPUTE_PHASE_CHECK applies a correction to the phase function to ensure all of the delta-M scaling works out. To do this it needs to calculate the delta-M scaling factor (F) from LEGEN, however, it has been supplied dlen as LEGEN when calculating the derivatives and mistakenly calculates the wrong correction factor because of this and this component of the gradient becomes incorrect.
In particular, it becomes much too positively large. This bug was found from the public unpolarized version of pyshdom by noticing that the gradient with respect to the effective radius changed sign as a function of the angular resolution (which changes the delta-M scaling factor F).
As for a fix, the easiest way is to create a separate subroutine PRECOMPUTE_PHASE_CHECK_GRAD that uses the correct F factor and add it to shdomsub4.f for both polarized and unpolarized. I will do this.
I think I’ve found a bug in the gradient calculation. This affects both the polarized and the unpolarized pyshdom optimization of cloud microphysics.
One component of the gradient is calculated from the single-scatter contribution to the radiance. The single-scatter contribution to the radiance is calculated from a tabulation of phase functions as a function of scattering angle (dphasetab). These are computed in PRECOMPUTE_PHASE_CHECK (Line 1162 unpol, Line 1607 Pol, shdomsub4.f) using the legendre coefficients of the phase functions (LEGEN). The same subroutine is used to compute the derivatives of the phase function at each scattering angle by replacing LEGEN with the derivatives of the legendre coefficients (dlen). These two uses of PRECOMPUTE_PHASE_CHECK are at Line 988 rte_solver.py and Line 928 optimize.py, respectively.
However, PRECOMPUTE_PHASE_CHECK applies a correction to the phase function to ensure all of the delta-M scaling works out. To do this it needs to calculate the delta-M scaling factor (F) from LEGEN, however, it has been supplied dlen as LEGEN when calculating the derivatives and mistakenly calculates the wrong correction factor because of this and this component of the gradient becomes incorrect.
In particular, it becomes much too positively large. This bug was found from the public unpolarized version of pyshdom by noticing that the gradient with respect to the effective radius changed sign as a function of the angular resolution (which changes the delta-M scaling factor F).
As for a fix, the easiest way is to create a separate subroutine PRECOMPUTE_PHASE_CHECK_GRAD that uses the correct F factor and add it to shdomsub4.f for both polarized and unpolarized. I will do this.