search

NOAA-EMC / GDASApp

Global Data Assimilation System Application
GNU Lesser General Public License v2.1
15 stars 31 forks source link

Cloning gdasApp on wcoss2 fails #1303

Open aerorahul opened 1 week ago

aerorahul commented 1 week ago

GDASApp fails to be cloned on WCOSS2

Steps to reproduce:

❯❯❯ git clone https://github.com/NOAA-EMC/gdasapp
Cloning into 'gdasapp'...
remote: Enumerating objects: 10663, done.
remote: Counting objects: 100% (2375/2375), done.
remote: Compressing objects: 100% (1154/1154), done.
remote: Total 10663 (delta 1492), reused 1869 (delta 1166), pack-reused 8288 (from 1)
Receiving objects: 100% (10663/10663), 2.83 MiB | 807.00 KiB/s, done.
Resolving deltas: 100% (7059/7059), done.

❯❯❯ cd gdasapp/

❯❯❯  git submodule update --init --recursive
Submodule 'parm/jcb-algorithms' (https://github.com/noaa-emc/jcb-algorithms) registered for path 'parm/jcb-algorithms'
Submodule 'parm/jcb-gdas' (https://github.com/noaa-emc/jcb-gdas) registered for path 'parm/jcb-gdas'
Submodule 'sorc/bufr-query' (https://github.com/noaa-emc/bufr-query.git) registered for path 'sorc/bufr-query'
Submodule 'sorc/crtm' (https://github.com/jcsda/crtm.git) registered for path 'sorc/crtm'
Submodule 'sorc/da-utils' (https://github.com/noaa-emc/da-utils.git) registered for path 'sorc/da-utils'
Submodule 'sorc/femps' (https://github.com/jcsda/femps.git) registered for path 'sorc/femps'
Submodule 'sorc/fv3' (https://github.com/jcsda/GFDL_atmos_cubed_sphere.git) registered for path 'sorc/fv3'
Submodule 'sorc/fv3-jedi' (https://github.com/jcsda/fv3-jedi.git) registered for path 'sorc/fv3-jedi'
Submodule 'sorc/fv3-jedi-lm' (https://github.com/jcsda/fv3-jedi-linearmodel.git) registered for path 'sorc/fv3-jedi-lm'
Submodule 'sorc/gsibec' (https://github.com/GEOS-ESM/GSIbec.git) registered for path 'sorc/gsibec'
Submodule 'sorc/gsw' (https://github.com/jcsda-internal/GSW-Fortran.git) registered for path 'sorc/gsw'
Submodule 'sorc/icepack' (https://github.com/JCSDA-internal/Icepack.git) registered for path 'sorc/icepack'
Submodule 'sorc/ioda' (https://github.com/jcsda/ioda.git) registered for path 'sorc/ioda'
Submodule 'sorc/iodaconv' (https://github.com/JCSDA-internal/ioda-converters.git) registered for path 'sorc/iodaconv'
Submodule 'sorc/jcb' (https://github.com/noaa-emc/jcb.git) registered for path 'sorc/jcb'
Submodule 'sorc/jedicmake' (https://github.com/jcsda/jedi-cmake.git) registered for path 'sorc/jedicmake'
Submodule 'sorc/land-imsproc' (https://github.com/NOAA-PSL/land-IMS_proc.git) registered for path 'sorc/land-imsproc'
Submodule 'sorc/land-jediincr' (https://github.com/NOAA-PSL/land-apply_jedi_incr.git) registered for path 'sorc/land-jediincr'
Submodule 'sorc/oops' (https://github.com/jcsda/oops.git) registered for path 'sorc/oops'
Submodule 'sorc/saber' (https://github.com/jcsda/saber.git) registered for path 'sorc/saber'
Submodule 'sorc/soca' (https://github.com/jcsda-internal/soca.git) registered for path 'sorc/soca'
Submodule 'sorc/ufo' (https://github.com/jcsda/ufo.git) registered for path 'sorc/ufo'
Submodule 'sorc/vader' (https://github.com/jcsda/vader.git) registered for path 'sorc/vader'
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/parm/jcb-algorithms'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/parm/jcb-gdas'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/bufr-query'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/crtm'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/da-utils'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/femps'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/fv3'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/fv3-jedi'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/fv3-jedi-lm'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/gsibec'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/gsw'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/icepack'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/ioda'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/iodaconv'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/jcb'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/jedicmake'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/land-imsproc'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/land-jediincr'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/oops'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/saber'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/soca'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/ufo'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/vader'...
Submodule path 'parm/jcb-algorithms': checked out 'a6822d8c1e72f6b1bf951e378b153cb6df1faee5'
Submodule path 'parm/jcb-gdas': checked out '272d6d73c0ceea5332fee005f59b3270527d10c7'
Submodule path 'sorc/bufr-query': checked out '97367fcd59adf4863aba1a52189e20f9f66451af'
git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly
Submodule path 'sorc/da-utils': checked out 'a2da7a84c76c03cbede5aa13f25549417bd30c43'
git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly
Submodule path 'sorc/fv3': checked out 'ab25dc09d955271f34ca6a3fa83af1093c85d9f7'
git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly
Submodule path 'sorc/fv3-jedi-lm': checked out 'a6e97d76ed7c0b2a27cf97512893a93d7e2b44bc'
Submodule path 'sorc/gsibec': checked out 'c8ac58d9b43eb8f890e565d12c88f1b0579c9ccd'
Submodule path 'sorc/gsw': checked out '697cbeb7605d70ed3857664c5f54a5c05346e31f'
Submodule path 'sorc/icepack': checked out '73136ee8dcdbe378821e540488a5980a03d8abe6'
Submodule path 'sorc/ioda': checked out 'b8281e5a9e04c07bd871a275eacd4ecd697b9434'
git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly
Submodule path 'sorc/jcb': checked out '8269ce474fe864fdd6dc51913ccce9daca630ac2'
Submodule path 'sorc/jedicmake': checked out '36fc99bdff5d3d8835480b37a3dcc75e5f8da256'
Submodule 'CMakeModules' (https://github.com/NOAA-EMC/CMakeModules) registered for path 'sorc/jedicmake/CMakeModules'
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/jedicmake/CMakeModules'...
Submodule path 'sorc/jedicmake/CMakeModules': checked out 'cabd7753ae17f7bfcc6dad56daf10868aa51c3f4'
Submodule path 'sorc/land-imsproc': checked out 'bbe3f311dc310020319c313cde1a2ee8a3c47a36'
Submodule path 'sorc/land-jediincr': checked out '7c1f6a3f8f949e376786eef7dba55a9e10e9778d'
Submodule path 'sorc/oops': checked out 'e6485c0a659103f0daa2b7e2cece39a15bfb0d60'
git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly
git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly
git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly
git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly
fatal: Unable to checkout 'bb7adbfc4f3deadedac98743bcb4e18a2bca11f9' in submodule path 'sorc/crtm'
fatal: Unable to checkout '4f12677d345e683bf910b5f76f0df120ad27482d' in submodule path 'sorc/femps'
fatal: Unable to checkout '731fcf4cbf541f37ac0531b2504fcc4108e1f6ee' in submodule path 'sorc/fv3-jedi'
fatal: Unable to checkout '3c437f464572ce8d5ab076b4ad713f7f7f24dd04' in submodule path 'sorc/iodaconv'
fatal: Unable to checkout 'bf5339c5ffe1828efab728e89bbc545ac22ddc88' in submodule path 'sorc/saber'
fatal: Unable to checkout '92519ab72b89a4c3b802501e71b7b66349fc8cc8' in submodule path 'sorc/soca'
fatal: Unable to checkout 'cd66505007b1559d79cb158bd6dc018a3943c1e7' in submodule path 'sorc/ufo'
fatal: Unable to checkout 'c4df3182de5c050b430250b5beae198af6eeb717' in submodule path 'sorc/vader'

My environment does not load any modules and am using git from the system provided path

❯❯❯  module list

Currently Loaded Modules:
  1) craype-x86-rome (H)   2) libfabric/1.11.0.0. (H)   3) craype-network-ofi (H)   4) envvar/1.0

  Where:
   H:  Hidden Module

❯❯❯  git --version
git version 2.35.3
aerorahul commented 1 week ago

git-lfs is available on WCOSS2. It requires loading gcc module. A ticket has been opened with WCOSS2 helpdesk to allow loading git-lfs alongside git without needing to load the gcc module.

Loading the git-lfs module allows successful cloning of the gdasapp

❯❯❯ module list

Currently Loaded Modules:
  1) craype-x86-rome (H)   2) libfabric/1.11.0.0. (H)   3) craype-network-ofi (H)   4) envvar/1.0   5) git/2.29.0   6) gcc/12.1.0   7) git-lfs/2.11.0

  Where:
   H:  Hidden Module
❯❯❯ time git submodule update --init --recursive
Submodule 'parm/jcb-algorithms' (https://github.com/noaa-emc/jcb-algorithms) registered for path 'parm/jcb-algorithms'
Submodule 'parm/jcb-gdas' (https://github.com/noaa-emc/jcb-gdas) registered for path 'parm/jcb-gdas'
Submodule 'sorc/bufr-query' (https://github.com/noaa-emc/bufr-query.git) registered for path 'sorc/bufr-query'
Submodule 'sorc/crtm' (https://github.com/jcsda/crtm.git) registered for path 'sorc/crtm'
Submodule 'sorc/da-utils' (https://github.com/noaa-emc/da-utils.git) registered for path 'sorc/da-utils'
Submodule 'sorc/femps' (https://github.com/jcsda/femps.git) registered for path 'sorc/femps'
Submodule 'sorc/fv3' (https://github.com/jcsda/GFDL_atmos_cubed_sphere.git) registered for path 'sorc/fv3'
Submodule 'sorc/fv3-jedi' (https://github.com/jcsda/fv3-jedi.git) registered for path 'sorc/fv3-jedi'
Submodule 'sorc/fv3-jedi-lm' (https://github.com/jcsda/fv3-jedi-linearmodel.git) registered for path 'sorc/fv3-jedi-lm'
Submodule 'sorc/gsibec' (https://github.com/GEOS-ESM/GSIbec.git) registered for path 'sorc/gsibec'
Submodule 'sorc/gsw' (https://github.com/jcsda-internal/GSW-Fortran.git) registered for path 'sorc/gsw'
Submodule 'sorc/icepack' (https://github.com/JCSDA-internal/Icepack.git) registered for path 'sorc/icepack'
Submodule 'sorc/ioda' (https://github.com/jcsda/ioda.git) registered for path 'sorc/ioda'
Submodule 'sorc/iodaconv' (https://github.com/JCSDA-internal/ioda-converters.git) registered for path 'sorc/iodaconv'
Submodule 'sorc/jcb' (https://github.com/noaa-emc/jcb.git) registered for path 'sorc/jcb'
Submodule 'sorc/jedicmake' (https://github.com/jcsda/jedi-cmake.git) registered for path 'sorc/jedicmake'
Submodule 'sorc/land-imsproc' (https://github.com/NOAA-PSL/land-IMS_proc.git) registered for path 'sorc/land-imsproc'
Submodule 'sorc/land-jediincr' (https://github.com/NOAA-PSL/land-apply_jedi_incr.git) registered for path 'sorc/land-jediincr'
Submodule 'sorc/oops' (https://github.com/jcsda/oops.git) registered for path 'sorc/oops'
Submodule 'sorc/saber' (https://github.com/jcsda/saber.git) registered for path 'sorc/saber'
Submodule 'sorc/soca' (https://github.com/jcsda-internal/soca.git) registered for path 'sorc/soca'
Submodule 'sorc/ufo' (https://github.com/jcsda/ufo.git) registered for path 'sorc/ufo'
Submodule 'sorc/vader' (https://github.com/jcsda/vader.git) registered for path 'sorc/vader'
Submodule path 'parm/jcb-algorithms': checked out 'a6822d8c1e72f6b1bf951e378b153cb6df1faee5'
Submodule path 'parm/jcb-gdas': checked out '272d6d73c0ceea5332fee005f59b3270527d10c7'
Submodule path 'sorc/bufr-query': checked out '97367fcd59adf4863aba1a52189e20f9f66451af'
Submodule path 'sorc/crtm': checked out 'bb7adbfc4f3deadedac98743bcb4e18a2bca11f9'
Submodule path 'sorc/da-utils': checked out 'a2da7a84c76c03cbede5aa13f25549417bd30c43'
Filtering content: 100% (5/5), 2.36 MiB | 553.00 KiB/s, done.
Submodule path 'sorc/femps': checked out '4f12677d345e683bf910b5f76f0df120ad27482d'
Submodule path 'sorc/fv3': checked out 'ab25dc09d955271f34ca6a3fa83af1093c85d9f7'
Filtering content: 100% (8/8), 2.39 MiB | 734.00 KiB/s, done.
Submodule path 'sorc/fv3-jedi': checked out '731fcf4cbf541f37ac0531b2504fcc4108e1f6ee'
Submodule path 'sorc/fv3-jedi-lm': checked out 'a6e97d76ed7c0b2a27cf97512893a93d7e2b44bc'
Submodule path 'sorc/gsibec': checked out 'c8ac58d9b43eb8f890e565d12c88f1b0579c9ccd'
Submodule path 'sorc/gsw': checked out '697cbeb7605d70ed3857664c5f54a5c05346e31f'
Submodule path 'sorc/icepack': checked out '73136ee8dcdbe378821e540488a5980a03d8abe6'
Submodule path 'sorc/ioda': checked out 'b8281e5a9e04c07bd871a275eacd4ecd697b9434'
Filtering content: 100% (368/368), 281.83 MiB | 872.00 KiB/s, done.
Submodule path 'sorc/iodaconv': checked out '3c437f464572ce8d5ab076b4ad713f7f7f24dd04'
Submodule path 'sorc/jcb': checked out '8269ce474fe864fdd6dc51913ccce9daca630ac2'
Submodule path 'sorc/jedicmake': checked out '36fc99bdff5d3d8835480b37a3dcc75e5f8da256'
Submodule path 'sorc/jedicmake/CMakeModules': checked out 'cabd7753ae17f7bfcc6dad56daf10868aa51c3f4'
Submodule path 'sorc/land-imsproc': checked out 'bbe3f311dc310020319c313cde1a2ee8a3c47a36'
Submodule path 'sorc/land-jediincr': checked out '7c1f6a3f8f949e376786eef7dba55a9e10e9778d'
Submodule path 'sorc/oops': checked out 'e6485c0a659103f0daa2b7e2cece39a15bfb0d60'
Filtering content: 100% (15/15), 150.01 MiB | 867.00 KiB/s, done.
Submodule path 'sorc/saber': checked out 'bf5339c5ffe1828efab728e89bbc545ac22ddc88'
Filtering content: 100% (56/56), 56.60 MiB | 840.00 KiB/s, done.
Submodule path 'sorc/soca': checked out '92519ab72b89a4c3b802501e71b7b66349fc8cc8'
Submodule 'mom6/MOM6' (https://github.com/NOAA-EMC/MOM6) registered for path 'sorc/soca/mom6/MOM6'
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/soca/mom6/MOM6'...
Submodule path 'sorc/soca/mom6/MOM6': checked out '10521a921d2f442de19a0cda240d912fd918c40c'
Submodule 'pkg/CVMix-src' (https://github.com/mom-ocean/CVMix-src.git) registered for path 'sorc/soca/mom6/MOM6/pkg/CVMix-src'
Submodule 'pkg/GSW-Fortran' (https://github.com/mom-ocean/GSW-Fortran.git) registered for path 'sorc/soca/mom6/MOM6/pkg/GSW-Fortran'
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/soca/mom6/MOM6/pkg/CVMix-src'...
Cloning into '/lfs/h2/emc/eib/noscrub/rahul.mahajan/gwWork/gdasapp/sorc/soca/mom6/MOM6/pkg/GSW-Fortran'...
Submodule path 'sorc/soca/mom6/MOM6/pkg/CVMix-src': checked out '9423197f894112edfcb1502245f7d7b873d551f9'
Submodule path 'sorc/soca/mom6/MOM6/pkg/GSW-Fortran': checked out '29e64d652786e1d076a05128c920f394202bfe10'
Filtering content: 100% (14/14), 6.08 MiB | 921.00 KiB/s, done.
Submodule path 'sorc/ufo': checked out 'cd66505007b1559d79cb158bd6dc018a3943c1e7'
Filtering content: 100% (7/7), 143.64 MiB | 865.00 KiB/s, done.
Submodule path 'sorc/vader': checked out 'c4df3182de5c050b430250b5beae198af6eeb717'

real    14m19.602s
RussTreadon-NOAA commented 1 week ago

Similar failure in GDASApp clone observed on Hercules and Orion when git-lfs is not loaded in the user environment

Hercules

git-lfs filter-process: line 1: git-lfs: command not found
fatal: the remote end hung up unexpectedly
git-lfs filter-process: line 1: git-lfs: command not found
fatal: the remote end hung up unexpectedly
git-lfs filter-process: line 1: git-lfs: command not found
fatal: the remote end hung up unexpectedly
git-lfs filter-process: line 1: git-lfs: command not found
fatal: the remote end hung up unexpectedly
Unable to checkout 'bb7adbfc4f3deadedac98743bcb4e18a2bca11f9' in submodule path 'sorc/crtm'
Unable to checkout '4f12677d345e683bf910b5f76f0df120ad27482d' in submodule path 'sorc/femps'
Unable to checkout '731fcf4cbf541f37ac0531b2504fcc4108e1f6ee' in submodule path 'sorc/fv3-jedi'
Unable to checkout '3c437f464572ce8d5ab076b4ad713f7f7f24dd04' in submodule path 'sorc/iodaconv'
Unable to checkout 'bf5339c5ffe1828efab728e89bbc545ac22ddc88' in submodule path 'sorc/saber'
Unable to checkout '92519ab72b89a4c3b802501e71b7b66349fc8cc8' in submodule path 'sorc/soca'
Unable to checkout 'cd66505007b1559d79cb158bd6dc018a3943c1e7' in submodule path 'sorc/ufo'
Unable to checkout 'c4df3182de5c050b430250b5beae198af6eeb717' in submodule path 'sorc/vader'

Orion

git-lfs filter-process: line 1: git-lfs: command not found
fatal: the remote end hung up unexpectedly
git-lfs filter-process: line 1: git-lfs: command not found
fatal: the remote end hung up unexpectedly
git-lfs filter-process: line 1: git-lfs: command not found
fatal: the remote end hung up unexpectedly
git-lfs filter-process: line 1: git-lfs: command not found
fatal: the remote end hung up unexpectedly
Unable to checkout 'bb7adbfc4f3deadedac98743bcb4e18a2bca11f9' in submodule path 'sorc/crtm'
Unable to checkout '4f12677d345e683bf910b5f76f0df120ad27482d' in submodule path 'sorc/femps'
Unable to checkout '731fcf4cbf541f37ac0531b2504fcc4108e1f6ee' in submodule path 'sorc/fv3-jedi'
Unable to checkout '3c437f464572ce8d5ab076b4ad713f7f7f24dd04' in submodule path 'sorc/iodaconv'
Unable to checkout 'bf5339c5ffe1828efab728e89bbc545ac22ddc88' in submodule path 'sorc/saber'
Unable to checkout '92519ab72b89a4c3b802501e71b7b66349fc8cc8' in submodule path 'sorc/soca'
Unable to checkout 'cd66505007b1559d79cb158bd6dc018a3943c1e7' in submodule path 'sorc/ufo'
Unable to checkout 'c4df3182de5c050b430250b5beae198af6eeb717' in submodule path 'sorc/vader'

Clone works on Hercules and Orion when git-lfs module loaded. Both machines have git-lfs/3.1.2.

git-lfs is part of the system path, /usr/bin/git-lfs, on Hera. This is git-lfs/3.2.0. Thus, no modules need to be loaded to successfully clone GDASApp on Hera.

git-lfs must be available in the user environment to fully clone GDASApp with submodules.