wxflow not found in marine da jobs on wcoss2

[RussTreadon-NOAA]

What is wrong?

g-w CI C48mx500_3DVarAOWCDA fails on Cactus because module wxflow is not found. For example,

gdasprepoceanobs

+ JGLOBAL_PREP_OCEAN_OBS[21]: /lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/pr2641/scripts/exglobal_prep_ocean_obs.py
Traceback (most recent call last):
  File "/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/pr2641/scripts/exglobal_prep_ocean_obs.py", line 7, in <module>
    from wxflow import Logger, cast_strdict_as_dtypedict
ModuleNotFoundError: No module named 'wxflow'
+

gdasocnanalprep

+ JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP[46]: /lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/pr2641/sorc/gdas.cd/scripts/exgdas_global_marine_analysis_prep.py
Traceback (most recent call last):
  File "/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/pr2641/sorc/gdas.cd/scripts/exgdas_global_marine_analysis_prep.py", line 10, in <module>
    from soca import bkg_utils
  File "/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/pr2641/sorc/gdas.cd/ush/soca/bkg_utils.py", line 11, in <module>
    from wxflow import (Logger, FileHandler)
ModuleNotFoundError: No module named 'wxflow'

What should have happened?

Marine DA jobs should successfully run to completion on wcoss2

What machines are impacted?

WCOSS2

Steps to reproduce

1. clone and install g-w develop
2. enable C48mx500_3DVarAOWCDA CI on wcoss2
3. set up and run C48mx500_3DVarAOWCDA CI

20210324 18Z gdasprepoceanobs fails as noted above

Additional information

C48mx500_3DVarAOWCDA runs to completion on Hera but gdas.cd/modulefiles/GDAS/hera.intel.lua contains

-- hack for wxflow
prepend_path("PYTHONPATH", "/scratch1/NCEPDEV/da/python/gdasapp/wxflow/20240307/src")

Do you have a proposed solution?

Adding wxflow to the PYTHONPATH defined in jobs/rocoto/prepoceanobs.sh allows gdasprepoceanobs to run to completion

-PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}${HOMEgfs}/ush"
+wxflowPATH="${HOMEgfs}/ush/python:${HOMEgfs}/ush/python/wxflow/src"
+PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}${HOMEgfs}/ush:${wxflowPATH}"

Not sure if this is an acceptable solution.

An alternative would be to add something to GDASApp modulefiles/GDAS/wcoss2.intel.lua .

[RussTreadon-NOAA]

FYI @guillaumevernieres & @AndrewEichmann-NOAA

[RussTreadon-NOAA]

Tried another approach: Add wxflow to PYTHONPATH in ush/load_ufsda_modules.sh

@@ -48,6 +48,9 @@ case "${MACHINE_ID}" in
     ;;
 esac

+wxflowPATH="${HOMEgfs}/ush/python:${HOMEgfs}/ush/python/wxflow/src"
+PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}${HOMEgfs}/ush:${wxflowPATH}"
+

This change impacts all g-w jobs which execute ush/load_ufsda_modules.sh so extensive testing is required if this is deemed an acceptable solution.

[guillaumevernieres]

I have a fix for this @RussTreadon-NOAA , just a copy/paste of what you did in the rocoto bash script. I'm not quite sure why it worked before.