search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
27 stars 48 forks source link

Issue with Frontera spack-stack 1.6.0 #1388

Open uturuncoglu opened 4 days ago

uturuncoglu commented 4 days ago

Describe the bug Under UFS Coastal we are supporting Frontera for the ocean modeling community. We installed 1.6.0 previously without any issue and it was working fine until recent sync with the UFS WM. It seems that fms, g2 and g2tmpl modules are upgraded but still under 1.6.0 and then this requires updating our spack stack installation with those modules.

here are the new module version found in ufs.common fms/2024.01 g2/3.5.1 g2tmpl/1.13.0

and here are the module that we are using currently, fms/2023.04.lua g2/3.4.5.lua g2tmpl/1.10.2.lua

So, my question is that how this is happening. I was expecting to have release like 1.6.1 if there is some version change in modules. At this point, it is hard for me to update my 1.6.0 branch (has front support) to install these new versions since those new versions are available in newer version of spack and that is not compatible with our spack-stack 1.6.0 branch. Anyway, please let me know if there is 1.6.0 branch in Spack-stack side that has all those updated modules. If not, I think the kind of updates will create more headache in the UFS Coastal side in the future and somehow we need to find a consistent way to handle those kind of issues.

To Reproduce Steps to reproduce the behavior:

Expected behavior A clear and concise description of what you expected to happen. expecting to build the configuration

System: What system(s) are you running the code on? on Frontera

Additional context Add any other context about the problem here. No

climbfuji commented 4 days ago

@uturuncoglu Because some groups are less agile than others (NOAA/UFS, do you hear me?), they do not want to update spack-stack whenever a new release is made. But at the same time, they request updates to specific packages being made available in the existing stack. Bad practice imo, but that's what they insist on. Therefore, when there are no conflicts between packages and modules (especially duplicate module names), we go back and install new versions IN the existing environment OR as an addon/chained environment. The latter is preferred, since it leaves the original environment as-is.

The process generally is like this:

  1. Update the local clone of your spack-stack-1.6.0 install from the release/1.6.0 branches upstream (spack submodule and spack-stack parent).
  2. Create a chained environment that reuses whatever possible from the original environment and only installs the new packages.
  3. Run the install as usual, and add --upstream-modules to the spack module [tcl|lmod] refresh command.

I know this is unsatisfying, and I agree with you that this isn't a good situation. But NOAA&UFS want it this way.

uturuncoglu commented 4 days ago

@climbfuji Thanks for your help. Since UFS WM is used among different applications, it would be nice to follow some kind of standards in dependencies. I am not talking how community will adjust their self to these kind of changes. It is way more complicated for them. At this point, I solved the issue in UFS Coastal side by modifying Frontra module file and not pointing ufs.common in there (I know it is ugly solution but the fatest one). So, I could use the old version of those packages. In the meantime, I could try to follow the same approach like chaining and see what happens. Do you have detailed instruction of it? BTW, the spack that is found in spack-stack 1.6.0 branch is old and does not have newer version of those packages. When I try to update spack submodule, then it was giving error for me and complaining due to I think the inconsistencies between spack-stack and used version of spack. So, it would be time consuming to try installing newer version of those packages using spack-stack 1.6.0 as a base.

climbfuji commented 3 days ago

The "instructions" are here. It's a slightly different approach and requires copying the environment definition from Orion to Frontera: https://github.com/JCSDA/spack-stack/issues/1180#issuecomment-2251378587