JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
25 stars 44 forks source link

Update fms to 2022.04 #444

Closed climbfuji closed 1 year ago

climbfuji commented 1 year ago

Please describe the package or library you would like to add to spack-stack. The ufs-weather-model has updated to fms@2022.04, we need to do the same in our various package configs, templates, environment spack packages. We also need to update the version for JEDI (ufs-bundle). Not sure if the ufs-srweather-app is already on this version or not.

What applications will be using this package or library? UFS (ufs-weather-model)

Is there already a package or library in spack-stack that provides this, or related, functionality? fms@2022.04 is already available in spack (also in our fork)

Additional context n/a

Will This Package be Needed in a NOAA Operational Application? Yes - already approved on WCOSS2

WCOSS System Software Request Checklist n/a

mathomp4 commented 1 year ago

FYI to all those here: I need to make a PR about FMS as GEOS needs a variant not defined. I suppose I'll do this on mainline spack and then it can be brought down from them (which seems the "right" way).

AlexanderRichert-NOAA commented 1 year ago

I tested the ufs-weather-model-static template with FMS 2022.04 on Hera, and UFS WM built and ran fine (commit ea0b6e45). @GeorgeVandenberghe-NOAA

GeorgeVandenberghe-NOAA commented 1 year ago

Thanks. This will allow a full spack-stack dependency setup without my alligator clips.

On Wed, Jan 18, 2023 at 5:50 PM Alex Richert @.***> wrote:

I tested the ufs-weather-model-static template with FMS 2022.04 on Hera, and UFS WM built and ran fine (commit ea0b6e45). @GeorgeVandenberghe-NOAA https://github.com/GeorgeVandenberghe-NOAA

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1387475880, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FU7UPOLJPW5P5LF6YDWTAUNJANCNFSM6AAAAAAT6TGGP4 . You are receiving this because you were mentioned.Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

GeorgeVandenberghe-NOAA commented 1 year ago

" as GEOS needs a variant not defined." What variant is that. Something newer than 2022.04?

Hang-Lei-NOAA commented 1 year ago

@Alexander Richert - NOAA Affiliate @.***> for fms version 2022.03 and 2022.04 and later have to use the configure -DCONSTANTS=GFS. This has to be setted in the configs/templates/ufs-weather-model-static/spack.yaml since the spark package.py default is the constants of GFDL Otherwise, the model can run but the result will be wrong.

On Wed, Jan 18, 2023 at 1:04 PM GeorgeVandenberghe-NOAA < @.***> wrote:

" as GEOS needs a variant not defined." What variant is that. Something newer than 2022.04?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1387491960, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKWSMFFWS77KTW2ECYA5HRDWTAWBNANCNFSM6AAAAAAT6TGGP4 . You are receiving this because you were assigned.Message ID: @.***>

climbfuji commented 1 year ago

That’s a good point, I haven’t done that for my skylab-3.0.0 installs.

On Jan 18, 2023, at 11:34 AM, Hang-Lei-NOAA @.***> wrote:

@Alexander Richert - NOAA Affiliate @.***> for fms version 2022.03 and 2022.04 and later have to use the configure -DCONSTANTS=GFS. This has to be setted in the configs/templates/ufs-weather-model-static/spack.yaml since the spark package.py default is the constants of GFDL Otherwise, the model can run but the result will be wrong.

On Wed, Jan 18, 2023 at 1:04 PM GeorgeVandenberghe-NOAA < @.***> wrote:

" as GEOS needs a variant not defined." What variant is that. Something newer than 2022.04?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1387491960, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKWSMFFWS77KTW2ECYA5HRDWTAWBNANCNFSM6AAAAAAT6TGGP4 . You are receiving this because you were assigned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1387535531, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RN4KMOSKQA2FUWTZJDWTAZRVANCNFSM6AAAAAAT6TGGP4. You are receiving this because you were assigned.

GeorgeVandenberghe-NOAA commented 1 year ago

Wow! This is a wicked trap that could make GFS (or conversely GEOS or other) ports to new platforms very difficult. It could be several business days or maybe weeks to figure out why the model is running but giving grossly wrong answers. And although I am responding to spack stack to describe the gross catastrophic exposure this is really a FMS or UFS or both, design issue. Not knowing this could have tanked our UFS benchmark next quarter

^^ sigh!! ^^

On Wed, Jan 18, 2023 at 6:34 PM Hang-Lei-NOAA @.***> wrote:

@Alexander Richert - NOAA Affiliate @.***> for fms version 2022.03 and 2022.04 and later have to use the configure -DCONSTANTS=GFS. This has to be setted in the configs/templates/ufs-weather-model-static/spack.yaml since the spark package.py default is the constants of GFDL Otherwise, the model can run but the result will be wrong.

On Wed, Jan 18, 2023 at 1:04 PM GeorgeVandenberghe-NOAA < @.***> wrote:

" as GEOS needs a variant not defined." What variant is that. Something newer than 2022.04?

— Reply to this email directly, view it on GitHub < https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1387491960 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AKWSMFFWS77KTW2ECYA5HRDWTAWBNANCNFSM6AAAAAAT6TGGP4

. You are receiving this because you were assigned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1387535531, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FV5Y2FLL6W3RO5MDILWTAZRVANCNFSM6AAAAAAT6TGGP4 . You are receiving this because you were mentioned.Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

GeorgeVandenberghe-NOAA commented 1 year ago

I am thinking if we need to add an option to bring FMS back to the model. This option was in fms 2022.04 installation when we updated UFS WM with fms 2022.04.

On Wed, Jan 18, 2023 at 3:06 PM George Vandenberghe - NOAA Affiliate < @.***> wrote:

Wow! This is a wicked trap that could make GFS (or conversely GEOS or other) ports to new platforms very difficult. It could be several business days or maybe weeks to figure out why the model is running but giving grossly wrong answers. And although I am responding to spack stack to describe the gross catastrophic exposure this is really a FMS or UFS or both, design issue. Not knowing this could have tanked our UFS benchmark next quarter

^^ sigh!! ^^

On Wed, Jan 18, 2023 at 6:34 PM Hang-Lei-NOAA @.***> wrote:

@Alexander Richert - NOAA Affiliate @.***> for fms version 2022.03 and 2022.04 and later have to use the configure -DCONSTANTS=GFS. This has to be setted in the configs/templates/ufs-weather-model-static/spack.yaml since the spark package.py default is the constants of GFDL Otherwise, the model can run but the result will be wrong.

On Wed, Jan 18, 2023 at 1:04 PM GeorgeVandenberghe-NOAA < @.***> wrote:

" as GEOS needs a variant not defined." What variant is that. Something newer than 2022.04?

— Reply to this email directly, view it on GitHub < https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1387491960 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AKWSMFFWS77KTW2ECYA5HRDWTAWBNANCNFSM6AAAAAAT6TGGP4

. You are receiving this because you were assigned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1387535531, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FV5Y2FLL6W3RO5MDILWTAZRVANCNFSM6AAAAAAT6TGGP4 . You are receiving this because you were mentioned.Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

GeorgeVandenberghe-NOAA commented 1 year ago

I think we do but FMS should also be more robust, standard and stable. Look to NetCDF for a superb example of how that's done!

On Wed, Jan 18, 2023 at 8:24 PM Jun Wang - NOAA Federal @.***> wrote:

I am thinking if we need to add an option to bring FMS back to the model. This option was in fms 2022.04 installation when we updated UFS WM with fms 2022.04.

On Wed, Jan 18, 2023 at 3:06 PM George Vandenberghe - NOAA Affiliate < @.***> wrote:

Wow! This is a wicked trap that could make GFS (or conversely GEOS or other) ports to new platforms very difficult. It could be several business days or maybe weeks to figure out why the model is running but giving grossly wrong answers. And although I am responding to spack stack to describe the gross catastrophic exposure this is really a FMS or UFS or both, design issue. Not knowing this could have tanked our UFS benchmark next quarter

^^ sigh!! ^^

On Wed, Jan 18, 2023 at 6:34 PM Hang-Lei-NOAA @.***> wrote:

@Alexander Richert - NOAA Affiliate @.***> for fms version 2022.03 and 2022.04 and later have to use the configure -DCONSTANTS=GFS. This has to be setted in the configs/templates/ufs-weather-model-static/spack.yaml since the spark package.py default is the constants of GFDL Otherwise, the model can run but the result will be wrong.

On Wed, Jan 18, 2023 at 1:04 PM GeorgeVandenberghe-NOAA < @.***> wrote:

" as GEOS needs a variant not defined." What variant is that. Something newer than 2022.04?

— Reply to this email directly, view it on GitHub < https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1387491960 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AKWSMFFWS77KTW2ECYA5HRDWTAWBNANCNFSM6AAAAAAT6TGGP4

. You are receiving this because you were assigned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1387535531, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FV5Y2FLL6W3RO5MDILWTAZRVANCNFSM6AAAAAAT6TGGP4 . You are receiving this because you were mentioned.Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

mathomp4 commented 1 year ago

" as GEOS needs a variant not defined." What variant is that. Something newer than 2022.04?

@GeorgeVandenberghe-NOAA GEOS needs a 32bit variant as well as the 64bit. At least as I've tested so far. (We use FMS in an odd way and this sort of mirrors our current build of FMS in model.)

I made a PR here: https://github.com/spack/spack/pull/35010 to add it to mainline spack.

GeorgeVandenberghe-NOAA commented 1 year ago

Its looking more like we do need to fold FMS back into the model

On Wednesday, January 18, 2023, Matthew Thompson @.***> wrote:

" as GEOS needs a variant not defined." What variant is that. Something newer than 2022.04?

@GeorgeVandenberghe-NOAA GEOS needs a 32bit variant as well as the 64bit. At least as I've tested so far. (We use FMS in an odd way and this sort of mirrors our current build of FMS in model.)

I made a PR here: spack/spack#35010 to add it to mainline spack.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci6.googleusercontent.com/proxy/yJV_CFEBYS1mu3hwJK2Rog-f9Z0kbB4s7_iFSoyh2zZmUPnke2igw4MXPWwRIqDduViFFaBOKs5gycgnIZjwputDLS9343S5GsY4bVoOhz1mJFJZFlaTdBXkrRZCI2SoPLBVGXTPeLsffZ_-RsgcBDA_lrDzkHGoNO4d8Cdz7dO1qbhbiqbs2LEy8_fqUx6XW6lAxYgdGZaTgadJ3mvfFimg6I8k1JeRom6cCFFkQTjcWXIubQ=s0-d-e1-ft#https://github.com/notifications/beacon/ANDS4FUGPOM4Z22S6X5QH2LWTB36JA5CNFSM6AAAAAAT6TGGP6WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSTHC6SI.gif>Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

climbfuji commented 1 year ago

Please don’t, that’s bad idea. It took us a long time to get it out of there. It’s used by so many components outside the ufs-weather-model that you are only creating problems by adding it back in. On the contrary we should try to modularize as much as we can to leverage common dependencies. Maybe there is a way to work with the FMS developers instead to make the number of options a little less overwhelming.

On Jan 18, 2023, at 5:28 PM, GeorgeVandenberghe-NOAA @.***> wrote:

Its looking more like we do need to fold FMS back into the model

On Wednesday, January 18, 2023, Matthew Thompson @.***> wrote:

" as GEOS needs a variant not defined." What variant is that. Something newer than 2022.04?

@GeorgeVandenberghe-NOAA GEOS needs a 32bit variant as well as the 64bit. At least as I've tested so far. (We use FMS in an odd way and this sort of mirrors our current build of FMS in model.)

I made a PR here: spack/spack#35010 to add it to mainline spack.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci6.googleusercontent.com/proxy/yJV_CFEBYS1mu3hwJK2Rog-f9Z0kbB4s7_iFSoyh2zZmUPnke2igw4MXPWwRIqDduViFFaBOKs5gycgnIZjwputDLS9343S5GsY4bVoOhz1mJFJZFlaTdBXkrRZCI2SoPLBVGXTPeLsffZ_-RsgcBDA_lrDzkHGoNO4d8Cdz7dO1qbhbiqbs2LEy8_fqUx6XW6lAxYgdGZaTgadJ3mvfFimg6I8k1JeRom6cCFFkQTjcWXIubQ=s0-d-e1-ft#https://github.com/notifications/beacon/ANDS4FUGPOM4Z22S6X5QH2LWTB36JA5CNFSM6AAAAAAT6TGGP6WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSTHC6SI.gif>Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell) — Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1396280632, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RJIFFZBIWJDQ6VJSVLWTCDBXANCNFSM6AAAAAAT6TGGP4. You are receiving this because you were assigned.

GeorgeVandenberghe-NOAA commented 1 year ago

I agree its going to create problems and i threw this out as a warning to what might happen if the proliferation of incompatible component requirements isn't controlled. We now need a version compiled with GEOS constants and a version compiled with GFS constants for various projects' needs. There is also a requirement for 32 bit and 64 bit floating point bindings. Sometime this autumn, ufs weather model required FMS 2022.04 but earlier(summer) tags can't use this version because an external name in the summer tags' source has been removed from fms 2022.4. Perhaps freezing a standard for fms would help.

I am not a trained software engineer by any means but the above does suggest a growing problem . On Wednesday, January 18, 2023, Dom Heinzeller @.***> wrote:

Please don’t, that’s bad idea. It took us a long time to get it out of there. It’s used by so many components outside the ufs-weather-model that you are only creating problems by adding it back in. On the contrary we should try to modularize as much as we can to leverage common dependencies. Maybe there is a way to work with the FMS developers instead to make the number of options a little less overwhelming.

On Jan 18, 2023, at 5:28 PM, GeorgeVandenberghe-NOAA @.***> wrote:

Its looking more like we do need to fold FMS back into the model

On Wednesday, January 18, 2023, Matthew Thompson @.***> wrote:

" as GEOS needs a variant not defined." What variant is that. Something newer than 2022.04?

@GeorgeVandenberghe-NOAA GEOS needs a 32bit variant as well as the 64bit. At least as I've tested so far. (We use FMS in an odd way and this sort of mirrors our current build of FMS in model.)

I made a PR here: spack/spack#35010 to add it to mainline spack.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.<

https://ci6.googleusercontent.com/proxy/yJV_CFEBYS1mu3hwJK2Rog-f9Z0kbB4s7_iFSoyh2zZmUPnke2igw4MXPWwRIqDduViFFaBOKs5gycgnIZjwputDLS9343S5GsY4bVoOhz1mJFJZFlaTdBXkrRZCI2SoPLBVGXTPeLsffZ_-RsgcBDA_lrDzkHGoNO4d8Cdz7dO1qbhbiqbs2LEy8_fqUx6XW6lAxYgdGZaTgadJ3mvfFimg6I8k1JeRom6cCFFkQTjcWXIubQ=s0-d-e1-ft#https://github.com/notifications/beacon/ANDS4FUGPOM4Z22S6X5QH2LWTB36JA5CNFSM6AAAAAAT6TGGP6WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSTHC6SI.gif Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell) — Reply to this email directly, view it on GitHub < https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1396280632>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AB5C2RJIFFZBIWJDQ6VJSVLWTCDBXANCNFSM6AAAAAAT6TGGP4 . You are receiving this because you were assigned.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci3.googleusercontent.com/proxy/B1eIIdmutL1dMtNcUC7bqjp8j9fcyr4zFBvgtogUV_vMkC64FXMlv5fG455Ko9CmUMYjB-BDhDW6hJ6ESXBKLXn7JkFkU3kTnhRzdj7S9EzWJhpO2sGk7jGOQ0eMwzCnT_GJfb4CvrL7aMxsU9xhM2UuDRg7LTjRsqcrhPBJLaJzvTAuDXkwNXEq9LKHLLuUUNiz-8ST8KHX3hNeONhPAHskDM8MaQkqe8OFVs81b3-2QnqthA=s0-d-e1-ft#https://github.com/notifications/beacon/ANDS4FTYTZN3VQ45Q4ST2KLWTCDPFA5CNFSM6AAAAAAT6TGGP6WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSTHGM7E.gif>Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

climbfuji commented 1 year ago

I think we'll be able to handle this quite well in spack, there will probably be 2-3 installations of fms for each version, one with GFS, one with GFDL, one with GEOS constants. We build all of them in dual precision (32bit and 64bit), and make the choice of constants part of the module name.

GeorgeVandenberghe-NOAA commented 1 year ago

This sounds good. What will we do about the problem that old (pre 9/2022) tags of ufs-weather-model require fms2022:1 and will not build with fms2022:4 while newer ufs-weather-model tags require fms2022:4

On Thu, Jan 19, 2023 at 6:32 PM Dom Heinzeller @.***> wrote:

I think we'll be able to handle this quite well in spack, there will probably be 2-3 installations of fms for each version, one with GFS, one with GFDL, one with GEOS constants. We build all of them in dual precision (32bit and 64bit), and make the choice of constants part of the module name.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1397431923, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FUKTGVREGQWT3NLG43WTGCE3ANCNFSM6AAAAAAT6TGGP4 . You are receiving this because you were mentioned.Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

climbfuji commented 1 year ago

We can just install that version as well and load what is needed. It won’t have the constants decorator in the module name, that is only available since 2022.03.

On Jan 19, 2023, at 11:47 AM, GeorgeVandenberghe-NOAA @.***> wrote:

This sounds good. What will we do about the problem that old (pre 9/2022) tags of ufs-weather-model require fms2022:1 and will not build with fms2022:4 while newer ufs-weather-model tags require fms2022:4

On Thu, Jan 19, 2023 at 6:32 PM Dom Heinzeller @.***> wrote:

I think we'll be able to handle this quite well in spack, there will probably be 2-3 installations of fms for each version, one with GFS, one with GFDL, one with GEOS constants. We build all of them in dual precision (32bit and 64bit), and make the choice of constants part of the module name.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1397431923, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FUKTGVREGQWT3NLG43WTGCE3ANCNFSM6AAAAAAT6TGGP4 . You are receiving this because you were mentioned.Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell) — Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1397447068, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RJBWDN3WXGHIYR7BKLWTGD35ANCNFSM6AAAAAAT6TGGP4. You are receiving this because you were assigned.

AlexanderRichert-NOAA commented 1 year ago

I'll look at putting both versions in ufs-weather-model-static.

GeorgeVandenberghe-NOAA commented 1 year ago

The constants problem (GFDL:GFS:GEOS) will require three builds of the libraries and possibly includes. This would have been better handled upstream at the FMS API level by defining them with #IFDEF directives and having the various component build systems define which one they need with another -D in the cmake argument list.

On Thu, Jan 19, 2023 at 6:50 PM Alex Richert @.***> wrote:

I'll look at putting both versions in ufs-weather-model-static.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/spack-stack/issues/444#issuecomment-1397450406, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FUOEJA7G4IKHCOMK6DWTGEI3ANCNFSM6AAAAAAT6TGGP4 . You are receiving this because you were mentioned.Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)