generate_ens=T (in &hybrid_ensemble) causes segfault

[jswhit]

This is used to generate an internal ensemble from randomly sampling the static B.

! generate_ens - if true, then generate internal ensemble based on existing background error

It was originally added to test the ensemble Var solver, but I would like to add the capability to write out a random sample of the static B to use as additive inflation in the EnKF. Apparently it hasn't been used in a while, and changes to general_sub2grid and/or ckgcov have broken this option. The traceback is below.

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
global_gsi         0000000001F3632D  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7A759915D0  Unknown               Unknown  Unknown
libmpi.so.12.0     00002B7A74C4C2D2  Unknown               Unknown  Unknown
libmpi.so.12.0     00002B7A74A2981B  Unknown               Unknown  Unknown
libmpi.so.12.0     00002B7A74C68810  Unknown               Unknown  Unknown
libmpi.so.12.0     00002B7A74C3DE50  Unknown               Unknown  Unknown
libmpi.so.12.0     00002B7A74B79A25  Unknown               Unknown  Unknown
libmpi.so.12       00002B7A749EB274  PMPI_Alltoallv        Unknown  Unknown
libmpifort.so.12.  00002B7A74595B21  pmpi_alltoallv_       Unknown  Unknown
global_gsi         000000000050DD19  general_sub2grid_        1627  general_sub2grid_mod.f90
global_gsi         00000000013AAEA0  ckgcov_                   157  bkgcov.f90
global_gsi         0000000000E7BFF3  hybrid_ensemble_i        1502  hybrid_ensemble_isotropic.F90
global_gsi         0000000000E7A5FA  hybrid_ensemble_i        1245  hybrid_ensemble_isotropic.F90
global_gsi         0000000000E4F9C1  glbsoi_                   269  glbsoi.f90
global_gsi         000000000062A620  gsisub_                   201  gsisub.F90
global_gsi         000000000043AF7D  gsimod_mp_gsimain        1840  gsimod.F90
global_gsi         000000000043AEB7  MAIN__                    631  gsimain.f90
global_gsi         000000000043AE5E  Unknown               Unknown  Unknown
libc-2.17.so       00002B7A79407495  __libc_start_main     Unknown  Unknown
global_gsi         000000000043AD69  Unknown               Unknown  Unknown

[jswhit]

@aerorahul, @dtkleist have either of you used this option recently?

[aerorahul]

@jswhit I have not used this option for quite some time. Last was sometime in 2015-16, when trying to generate the ensemble with EVIL. I know it was broken then too.

[dtkleist]

@jswhit I'm in the same boat as @aerorahul here. I recall using this option very early in the EnVar development when we needed something to test the plumbing and provide a sanity check. Dave Parrish had done that work. In fact, most of the emails I can find about the use of this are from back in 2012/2013 or so. In addition to the EVIL work, I know that Xuguang had also been wanting to use this option at one point to write out perturbations from B. I do not think that anyone ever followed through and got all of this working. Given that the lagged forecast pairs seemed to more effective for additive inflation, we sort of lost track of this internal option.

[jswhit]

I'd really like to get this working to finish up the hybrid gain study we've been working on. The idea is to see if blending 3DVar and EnKF analysis increments is any different than simply adding random samples of the 3DVar covariance to the EnKF background ensemble. It looks like the updates to general_sub2grid require some changes to ckgcov, but I've spent a couple of days on this and haven't been able to figure it out. Is there anyone (besides Dave) who understands the general_subg2grid implementation who could potentially help?

[dtkleist]

I think you are probably stuck with either myself or @aerorahul. I can also ping on Xuguang to see if someone from her group had ever gotten it to work.

[aerorahul]

May be @WanshuWu-NOAA knows general_sub2grid better. She recently must have used it for her regional implementation of FV3.

[WanshuWu-NOAA]

general_sub2grid works fine. Please find out whether s2g_raf is defined with an ini call.

[jswhit]

I think I may have a solution - testing now. If it works, I'll create a PR

[jswhit]

Fix in https://github.com/NOAA-EMC/GSI/pull/42

[jswhit]

@dtkleist - what are the units for specific humidity in the 3dvar static B? I see perturbations that are O(1), so if it's specific humidity perhaps it's g/kg? Or is is relative humidity?

[dtkleist]

Control variable for static B is either pseudo RH (qoption 1) or normalized pseudo RH (qoption 2).

[jswhit]

so for qoption=1 multiplying by ges_qsat will convert to specific humidity?

[dtkleist]

Indeed, that should work. Alternatively, there is a routine "normal_rh_to_q" that should be able to get "q" regardless of qoption.

[jswhit]

Thanks for the tip @dtkleist - I've incorporated that into the PR.

[jswhit]

I noticed that normal_rh_to_q doesn't work currently if qoption=1. This is because compute_qvar3d is only called in prewgt if qoption=2 - and this is where the array dqdrh is defined. I went ahead and changed prewgt so that compute_qvar3d is always called, regardless of qoption. This seems to fix the issue, but there may be other unintended consequences.