search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
24 stars 44 forks source link

ruamel for gmao-swell-env? #1253

Open Dooruk opened 3 weeks ago

Dooruk commented 3 weeks ago

In our templating and writing dictionaries to files, we noticed that the order of output YAMLs change and they are written alphabetically:

https://github.com/GEOS-ESM/swell/blob/96dd41c8a548b61e51334bda8477d3e481e964df/src/swell/tasks/run_jedi_variational_executable.py#L119C1-L122C89

This worked for us so far, however Ricardo noticed the scrambling of UFO filters might be causing answers to differ between GSI and SWELL, so we are considering multiple options to preserve order. I tested with a simple conversion to OrderedDict which works in a temporary way. I noticed EWOK uses NamedDict and ruamel package, which might be a more robust fix. So I was wondering if ruamel could be added to the gmao-swell-env and/or if it has too many dependencies?

https://github.com/JCSDA-internal/ewok/blob/d26a9f805e6068ab637a9004ba07084e8188cb02/src/yamltools/template.py#L18

More information here https://github.com/GEOS-ESM/swell/issues/409#issuecomment-2293740551

climbfuji commented 1 week ago

Since we build the unified environment on Discover, can you simply load the existing module for py-ruamel for the time being? It's available after you load stack-COMPILER, stack-MPI, stack-PYTHON.

Dooruk commented 6 days ago

Hmm, I don't see py-ruamel in my module list after loading via (or on SCU16):

module purge
module use /discover/swdev/gmao_SIteam/modulefiles-SLES15
module use /discover/swdev/jcsda/spack-stack/scu17/modulefiles
module load ecflow/5.11.4

module use /gpfsm/dswdev/jcsda/spack-stack/scu17/spack-stack-1.7.0/envs/ue-intel-2021.10.0/install/modulefiles/Core
module load stack-intel/2021.10.0
module load stack-intel-oneapi-mpi/2021.10.0
module load stack-python/3.10.13
climbfuji commented 6 days ago

Just to confirm, I have py-ruamel-yaml and py-ruamel-yaml-clib in my list of available modules. Thse are not what you are looking for?

Dooruk commented 6 days ago

Just to confirm, I have py-ruamel-yaml and py-ruamel-yaml-clib in my list of available modules. Thse are not what you are looking for?

Do you see those with module list after loading the same modules I listed above?

climbfuji commented 6 days ago

I can't log into discover at the moment, but I am checking the unified environment on my local machine (it is supposed to be identical)

climbfuji commented 5 days ago

@srherbener or @ashley314 Since I can't log into Discover at the moment (or not at all anymore?), can one of you check please if the modules py-ruamel-yaml and py-ruamel-yaml-clib are installed in the existing (1.7.0) and new (1.8.0) unified environments on Discover? Thanks very much!

ashley314 commented 4 days ago

@climbfuji I see both of those modules on discover in spack-stack 1.7.0 and 1.8.0

climbfuji commented 4 days ago

Thanks @ashley314 - @Dooruk can you check again, please?

Dooruk commented 4 days ago

@climbfuji I see both of those modules on discover in spack-stack 1.7.0 and 1.8.0

Thanks for checking this @ashley314. Could you test the following script that loads and tests ruamel with the following on SCU17 please? I'm getting ModuleNotFoundError: No module named 'ruamel'.

module purge
module use /discover/swdev/gmao_SIteam/modulefiles-SLES15
module use /discover/swdev/jcsda/spack-stack/scu17/modulefiles
module load ecflow/5.11.4

module use /gpfsm/dswdev/jcsda/spack-stack/scu17/spack-stack-1.7.0/envs/ue-intel-2021.10.0/install/modulefiles/Core
module load stack-intel/2021.10.0
module load stack-intel-oneapi-mpi/2021.10.0
module load stack-python/3.10.13
module load jedi-fv3-env
module load soca-env
module load gmao-swell-env/1.0.0

#Test importing ruamel
python3 -c "from ruamel.yaml import YAML"
Dooruk commented 4 days ago

Thanks @ashley314 - @Dooruk can you check again, please?

I must be missing an env load.

climbfuji commented 4 days ago

Please load the pu-ruamel-yaml etc modules explicitly after loading your other modules. It may be that the ruamel modules are part of ewok-env, but as I said in my comment https://github.com/JCSDA/spack-stack/issues/1253#issuecomment-2332769833 from last week, please load those py-ruamel modules directly.

Dooruk commented 4 days ago

It's there, my bad, I thought it should already be loaded not in the available modules!

climbfuji commented 4 days ago

Sorry for not being clear. We should still add the modules you want (which ones exactly) to gmao-swell-env for spack-stack-1.9.0, but until then you'll have to keep loading them manually. (I'll keep this issue open until we added the correct modules to gmao-swell-env).

Dooruk commented 4 days ago

Sounds good, thank you.