search

NREL / SOWFA-6

31 stars 35 forks source link

OpenFOAM 6 with SOWFA 6 via Spack? #47

Open mikerenfro opened 2 years ago

mikerenfro commented 2 years ago

Hey, folks. I'm neither an OpenFOAM nor a SOWFA user, but I'm trying to get both installed on our HPC running Rocky and Spack, and as an intermedia step, I'm trying to install it on a Vagrant VM to iterate more quickly.

The Vagrantfile I'm using:

Vagrant.configure("2") do |config|
  config.vm.box = "rockylinux/8"
  config.vm.box_version = "4.0"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "16384"
    vb.cpus = 4
  end
  config.vm.provision "shell", inline: <<-SHELLPRIV
    yum -y remove 'iwl*'
    yum -y update
    yum -y install redhat-lsb-core git gcc-c++ gcc-gfortran python3
  SHELLPRIV
  config.vm.provision "shell", privileged: false, inline: <<-SHELLUNPRIV
    git clone -c feature.manyFiles=true https://github.com/spack/spack.git
    git clone https://github.com/NREL/SOWFA-6.git
    ( cd SOWFA-6 ; git checkout dev )
    . spack/share/spack/setup-env.sh
    spack compiler find
    spack install -j4 openfoam-org@6
  SHELLUNPRIV
  # Spack for pmix needs Python, not listed as dependency
end

After logging back in as the vagrant user and loading the spack settings via sourcing spack/share/spack/setup-env.sh, I try to run Allwclean and Allwmake from the SOWFA-6 directory.

Allwclean returns:

wclean src/miscTools
wclean src/TurbulenceModels/incompressible
wclean src/TurbulenceModels/turbulenceModels
wclean src/boundaryConditions
OPENFAST_DIR is not set. Not cleaning OpenFAST interface.
wclean src/meshTools
wclean src/ABLForcing
wclean src/sampling
wclean src/fileFormats
wclean src/functionObjects/field
wclean src/functionObjects/utilities
wclean applications/utilities/setFieldsABL
wclean applications/solvers/superDeliciousVanilla
OPENFAST_DIR is not set. Not cleaning OpenFASTcoupled solver.

which I don't think is a problem, while Allwmake returns:

wmake src/miscTools
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file geometricTransformations.C
Making dependency list for source file windRoseToCartesian.C
g++ -std=c++11 -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -O3 -Wl,-rpath,/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/platforms/linux64GccDPInt32-spack/lib -DNoRepository -ftemplate-depth-100 -I/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/src/meshTools/lnInclude -I/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/src/finiteVolume/lnInclude -IlnInclude -I. -I/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/src/OpenFOAM/lnInclude -I/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/src/OSspecific/POSIX/lnInclude   -fPIC -c windRoseToCartesian/windRoseToCartesian.C -o Make/linux64GccDPInt32-spack/windRoseToCartesian/windRoseToCartesian.o
g++ -std=c++11 -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -O3 -Wl,-rpath,/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/platforms/linux64GccDPInt32-spack/lib -DNoRepository -ftemplate-depth-100 -I/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/src/meshTools/lnInclude -I/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/src/finiteVolume/lnInclude -IlnInclude -I. -I/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/src/OpenFOAM/lnInclude -I/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/src/OSspecific/POSIX/lnInclude   -fPIC -c geometricTransformations/geometricTransformations.C -o Make/linux64GccDPInt32-spack/geometricTransformations/geometricTransformations.o
mkdir: cannot create directory ‘’: No such file or directory
make: *** [/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/wmake/makefiles/general:167: /libSOWFAmiscTools.so] Error 1

and this pattern repeats for several other wmake commands. Line 167 is @$(WM_SCRIPTS)/makeTargetDir $(LIB) and line 80 defines LIB = libNULL, but that value must have been overridden somewhere before line 167.

What other information can I provide to help figure out what's going on? My current environment variables via export are:

declare -x ACLOCAL_PATH="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/util-macros-1.19.3-cbg6rwljtnw4ks7momqiypu4antmqxlv/share/aclocal:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libtool-2.4.7-uv6utq5fcsoxoq67hdqj6domvomgzsgc/share/aclocal:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/gettext-0.21-rvdrywf34tw757crvxseh7vfs42insur/share/aclocal:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libxml2-2.9.13-fh4iwewj76yykzxyefgorzldzy557a2t/share/aclocal:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/bison-3.8.2-fexlv7zu4g5qlllgafsc3ob2qpkmrwft/share/aclocal:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/automake-1.16.5-ztu2zzdcr32japl4jd5gichynwa6mggi/share/aclocal:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/cmake-3.23.3-rlboixbp3tv2lr26cxvgoaq7dr2ugwgz/share/aclocal:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pkgconf-1.8.0-os7jljyxz37e4ni4zi67xwo6k5e2ivkj/share/aclocal"
declare -x CMAKE_PREFIX_PATH="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/scotch-7.0.1-fldpglss2xejvz6sr4ajddbboq63iv3m:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pmix-4.1.2-qeczppkrcnyzcjdbhxtyax2bpuaxmwbz:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libevent-2.1.12-mtro3r6zg3ok4gjom7v2qaxc3ehthmgx:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openssh-9.0p1-thdbknwgmojuf7sdhxspchomfkxh6r2z:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libedit-3.1-20210216-2xle4h4epawrbhxfxt5eq5n63bfosycn:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/krb5-1.19.3-bevzzjbflfsi6busqr3eu7vweidtjz2n:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/numactl-2.0.14-h2m5lu5karad2t4dmbncpazkwf2rmvhb:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/hwloc-2.8.0-hdzs5uenwyrkk3jrcd2cd2qzpgxvccff:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libpciaccess-0.16-oy7evzedaypskacv3fxthcmhp6rvzymy:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/util-macros-1.19.3-cbg6rwljtnw4ks7momqiypu4antmqxlv:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/flex-2.6.4-5y2uye7ko2p6u6qyywcdecm7ijj7qmcm:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libtool-2.4.7-uv6utq5fcsoxoq67hdqj6domvomgzsgc:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/help2man-1.47.16-jm6xhcbjozgvqaphc6soyjb33ofgmvwx:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/gettext-0.21-rvdrywf34tw757crvxseh7vfs42insur:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/tar-1.34-4ijsvxdefkdjdyeod57hwegg6emmmle5:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/zstd-1.5.2-wsfzoxluyzq2s6c463tj4qfltfjiwl2v:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pigz-2.7-zhfs2i7674sapcllgjgcauh73euc5ovi:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libxml2-2.9.13-fh4iwewj76yykzxyefgorzldzy557a2t:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/xz-5.2.5-jqegoncucvxcryk4t3ay4cqfgl4yp5um:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/findutils-4.9.0-xuppk3u3bcwjxurcahunhvvr63f4it73:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/bison-3.8.2-fexlv7zu4g5qlllgafsc3ob2qpkmrwft:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/automake-1.16.5-ztu2zzdcr32japl4jd5gichynwa6mggi:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/autoconf-2.69-zt4tcsx45jrkkqllohozvc7mu4gri7bl:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/m4-1.4.19-3qjci5fw4vi5bq4mm2dopfjclz7aczu6:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libsigsegv-2.13-g3g26ugiy2oudh4pzlbpepf4ahvgihdw:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/cmake-3.23.3-rlboixbp3tv2lr26cxvgoaq7dr2ugwgz:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openssl-1.1.1q-66n75bdhfn77yrujghu36wgxn6zypl2k:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/perl-5.34.1-ngplx3jzbydjcfh25ipta5vsuct64tmb:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/zlib-1.2.12-7m2fsilyutgiiyblyy6yd2tcpib4cvds:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/gdbm-1.19-lvevadigsoqfzobcvsic5idn57dvt74k:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/readline-8.1.2-rh7upcssc5pmgitoijytsqa64hwiiqlr:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/bzip2-1.0.8-crujvlcawmzyti2vjlazyp7zfmko5my5:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/diffutils-3.8-omdg6cusi57u4g3c5fquyiphfj4mictx:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libiconv-1.16-i6cnkc73c3uvx6gu3jnosrtbiadurhqz:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/berkeley-db-18.1.40-2gie7vuqjqzr2vay3dotfyxrnwupufaw:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/ca-certificates-mozilla-2022-07-19-yvgly67hrhryzg7vstjyhuvqhxtu44yp:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/ncurses-6.3-mdk4c7oyvvfn5jn3jwtuqphqjudoacmr:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pkgconf-1.8.0-os7jljyxz37e4ni4zi67xwo6k5e2ivkj"
declare -x DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"
declare -x FOAM_APP="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/applications"
declare -x FOAM_APPBIN="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/platforms/linux64GccDPInt32-spack/bin"
declare -x FOAM_ETC="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/etc"
declare -x FOAM_EXT_LIBBIN="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/ThirdParty/platforms/linux64GccDPInt32/lib"
declare -x FOAM_INST_DIR="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0"
declare -x FOAM_JOB_DIR="/home/vagrant/.OpenFOAM/jobControl"
declare -x FOAM_LIBBIN="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/platforms/linux64GccDPInt32-spack/lib"
declare -x FOAM_MPI="mpi-system"
declare -x FOAM_RUN="/home/vagrant/OpenFOAM/vagrant-6/run"
declare -x FOAM_SETTINGS=""
declare -x FOAM_SIGFPE=""
declare -x FOAM_SITE_APPBIN="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/site/6/platforms/linux64GccDPInt32-spack/bin"
declare -x FOAM_SITE_LIBBIN="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/site/6/platforms/linux64GccDPInt32-spack/lib"
declare -x FOAM_SOLVERS="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/applications/solvers"
declare -x FOAM_SRC="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/src"
declare -x FOAM_TUTORIALS="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/tutorials"
declare -x FOAM_USER_APPBIN="/home/vagrant/OpenFOAM/vagrant-6/platforms/linux64GccDPInt32-spack/bin"
declare -x FOAM_USER_LIBBIN="/home/vagrant/OpenFOAM/vagrant-6/platforms/linux64GccDPInt32-spack/lib"
declare -x FOAM_UTILITIES="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/applications/utilities"
declare -x HISTCONTROL="ignoredups"
declare -x HISTSIZE="1000"
declare -x HOME="/home/vagrant"
declare -x HOSTNAME="localhost.localdomain"
declare -x LANG="en_US.UTF-8"
declare -x LD_LIBRARY_PATH="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/platforms/linux64GccDPInt32-spack/lib/mpi-system:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/ThirdParty/platforms/linux64GccDPInt32/lib/mpi-system:/home/vagrant/OpenFOAM/vagrant-6/platforms/linux64GccDPInt32-spack/lib:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/site/6/platforms/linux64GccDPInt32-spack/lib:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/platforms/linux64GccDPInt32-spack/lib:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/ThirdParty/platforms/linux64GccDPInt32/lib:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/platforms/linux64GccDPInt32-spack/lib/dummy"
declare -x LESSOPEN="||/usr/bin/lesspipe.sh %s"
declare -x LOGNAME="vagrant"
declare -x LS_COLORS="rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;05;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.m4a=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.oga=38;5;45:*.opus=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:"
declare -x M4="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/m4-1.4.19-3qjci5fw4vi5bq4mm2dopfjclz7aczu6/bin/m4"
declare -x MAIL="/var/spool/mail/vagrant"
declare -x MANPATH="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/scotch-7.0.1-fldpglss2xejvz6sr4ajddbboq63iv3m/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pmix-4.1.2-qeczppkrcnyzcjdbhxtyax2bpuaxmwbz/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openssh-9.0p1-thdbknwgmojuf7sdhxspchomfkxh6r2z/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libedit-3.1-20210216-2xle4h4epawrbhxfxt5eq5n63bfosycn/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/krb5-1.19.3-bevzzjbflfsi6busqr3eu7vweidtjz2n/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/numactl-2.0.14-h2m5lu5karad2t4dmbncpazkwf2rmvhb/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/hwloc-2.8.0-hdzs5uenwyrkk3jrcd2cd2qzpgxvccff/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/flex-2.6.4-5y2uye7ko2p6u6qyywcdecm7ijj7qmcm/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libtool-2.4.7-uv6utq5fcsoxoq67hdqj6domvomgzsgc/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/help2man-1.47.16-jm6xhcbjozgvqaphc6soyjb33ofgmvwx/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/gettext-0.21-rvdrywf34tw757crvxseh7vfs42insur/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/tar-1.34-4ijsvxdefkdjdyeod57hwegg6emmmle5/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/zstd-1.5.2-wsfzoxluyzq2s6c463tj4qfltfjiwl2v/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pigz-2.7-zhfs2i7674sapcllgjgcauh73euc5ovi/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libxml2-2.9.13-fh4iwewj76yykzxyefgorzldzy557a2t/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/xz-5.2.5-jqegoncucvxcryk4t3ay4cqfgl4yp5um/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/findutils-4.9.0-xuppk3u3bcwjxurcahunhvvr63f4it73/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/bison-3.8.2-fexlv7zu4g5qlllgafsc3ob2qpkmrwft/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/automake-1.16.5-ztu2zzdcr32japl4jd5gichynwa6mggi/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/autoconf-2.69-zt4tcsx45jrkkqllohozvc7mu4gri7bl/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/m4-1.4.19-3qjci5fw4vi5bq4mm2dopfjclz7aczu6/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/perl-5.34.1-ngplx3jzbydjcfh25ipta5vsuct64tmb/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/zlib-1.2.12-7m2fsilyutgiiyblyy6yd2tcpib4cvds/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/gdbm-1.19-lvevadigsoqfzobcvsic5idn57dvt74k/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/readline-8.1.2-rh7upcssc5pmgitoijytsqa64hwiiqlr/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/bzip2-1.0.8-crujvlcawmzyti2vjlazyp7zfmko5my5/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/diffutils-3.8-omdg6cusi57u4g3c5fquyiphfj4mictx/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libiconv-1.16-i6cnkc73c3uvx6gu3jnosrtbiadurhqz/share/man:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pkgconf-1.8.0-os7jljyxz37e4ni4zi67xwo6k5e2ivkj/share/man:"
declare -x MODULEPATH="/home/vagrant/spack/share/spack/modules/linux-rocky8-sandybridge"
declare -x MPICC="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf/bin/mpicc"
declare -x MPICXX="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf/bin/mpic++"
declare -x MPIF77="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf/bin/mpif77"
declare -x MPIF90="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf/bin/mpif90"
declare -x MPI_ARCH_FLAGS="-DOMPI_SKIP_MPICXX -DMPICH_SKIP_MPICXX"
declare -x MPI_ARCH_INC="-I/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf/include"
declare -x MPI_ARCH_LIBS="-L/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf/lib -lmpi"
declare -x MPI_ARCH_PATH="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf"
declare -x MPI_BUFFER_SIZE="20000000"
declare -x MPI_ROOT="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf"
declare -x OLDPWD="/home/vagrant"
declare -x PATH="/home/vagrant/OpenFOAM/vagrant-6/platforms/linux64GccDPInt32-spack/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/site/6/platforms/linux64GccDPInt32-spack/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/platforms/linux64GccDPInt32-spack/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/wmake:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/scotch-7.0.1-fldpglss2xejvz6sr4ajddbboq63iv3m/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pmix-4.1.2-qeczppkrcnyzcjdbhxtyax2bpuaxmwbz/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libevent-2.1.12-mtro3r6zg3ok4gjom7v2qaxc3ehthmgx/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openssh-9.0p1-thdbknwgmojuf7sdhxspchomfkxh6r2z/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/krb5-1.19.3-bevzzjbflfsi6busqr3eu7vweidtjz2n/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/numactl-2.0.14-h2m5lu5karad2t4dmbncpazkwf2rmvhb/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/hwloc-2.8.0-hdzs5uenwyrkk3jrcd2cd2qzpgxvccff/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/flex-2.6.4-5y2uye7ko2p6u6qyywcdecm7ijj7qmcm/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libtool-2.4.7-uv6utq5fcsoxoq67hdqj6domvomgzsgc/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/help2man-1.47.16-jm6xhcbjozgvqaphc6soyjb33ofgmvwx/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/gettext-0.21-rvdrywf34tw757crvxseh7vfs42insur/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/tar-1.34-4ijsvxdefkdjdyeod57hwegg6emmmle5/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/zstd-1.5.2-wsfzoxluyzq2s6c463tj4qfltfjiwl2v/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pigz-2.7-zhfs2i7674sapcllgjgcauh73euc5ovi/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libxml2-2.9.13-fh4iwewj76yykzxyefgorzldzy557a2t/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/xz-5.2.5-jqegoncucvxcryk4t3ay4cqfgl4yp5um/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/findutils-4.9.0-xuppk3u3bcwjxurcahunhvvr63f4it73/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/bison-3.8.2-fexlv7zu4g5qlllgafsc3ob2qpkmrwft/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/automake-1.16.5-ztu2zzdcr32japl4jd5gichynwa6mggi/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/autoconf-2.69-zt4tcsx45jrkkqllohozvc7mu4gri7bl/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/m4-1.4.19-3qjci5fw4vi5bq4mm2dopfjclz7aczu6/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/cmake-3.23.3-rlboixbp3tv2lr26cxvgoaq7dr2ugwgz/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openssl-1.1.1q-66n75bdhfn77yrujghu36wgxn6zypl2k/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/perl-5.34.1-ngplx3jzbydjcfh25ipta5vsuct64tmb/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/gdbm-1.19-lvevadigsoqfzobcvsic5idn57dvt74k/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/readline-8.1.2-rh7upcssc5pmgitoijytsqa64hwiiqlr/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/bzip2-1.0.8-crujvlcawmzyti2vjlazyp7zfmko5my5/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/diffutils-3.8-omdg6cusi57u4g3c5fquyiphfj4mictx/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libiconv-1.16-i6cnkc73c3uvx6gu3jnosrtbiadurhqz/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/berkeley-db-18.1.40-2gie7vuqjqzr2vay3dotfyxrnwupufaw/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/ncurses-6.3-mdk4c7oyvvfn5jn3jwtuqphqjudoacmr/bin:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pkgconf-1.8.0-os7jljyxz37e4ni4zi67xwo6k5e2ivkj/bin:/home/vagrant/spack/bin:/home/vagrant/.local/bin:/home/vagrant/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
declare -x PKG_CONFIG_PATH="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openmpi-4.1.4-pxrvymyorqh7qxi4jrwmchi2yuuq4agf/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pmix-4.1.2-qeczppkrcnyzcjdbhxtyax2bpuaxmwbz/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libevent-2.1.12-mtro3r6zg3ok4gjom7v2qaxc3ehthmgx/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libedit-3.1-20210216-2xle4h4epawrbhxfxt5eq5n63bfosycn/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/krb5-1.19.3-bevzzjbflfsi6busqr3eu7vweidtjz2n/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/numactl-2.0.14-h2m5lu5karad2t4dmbncpazkwf2rmvhb/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/hwloc-2.8.0-hdzs5uenwyrkk3jrcd2cd2qzpgxvccff/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libpciaccess-0.16-oy7evzedaypskacv3fxthcmhp6rvzymy/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/util-macros-1.19.3-cbg6rwljtnw4ks7momqiypu4antmqxlv/share/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/zstd-1.5.2-wsfzoxluyzq2s6c463tj4qfltfjiwl2v/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/libxml2-2.9.13-fh4iwewj76yykzxyefgorzldzy557a2t/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/xz-5.2.5-jqegoncucvxcryk4t3ay4cqfgl4yp5um/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openssl-1.1.1q-66n75bdhfn77yrujghu36wgxn6zypl2k/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/zlib-1.2.12-7m2fsilyutgiiyblyy6yd2tcpib4cvds/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/readline-8.1.2-rh7upcssc5pmgitoijytsqa64hwiiqlr/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/ncurses-6.3-mdk4c7oyvvfn5jn3jwtuqphqjudoacmr/lib/pkgconfig:/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/pkgconf-1.8.0-os7jljyxz37e4ni4zi67xwo6k5e2ivkj/lib/pkgconfig"
declare -x PWD="/home/vagrant/SOWFA-6"
declare -x SELINUX_LEVEL_REQUESTED=""
declare -x SELINUX_ROLE_REQUESTED=""
declare -x SELINUX_USE_CURRENT_RANGE=""
declare -x SHELL="/bin/bash"
declare -x SHLVL="1"
declare -x SPACK_LOADED_HASHES="wqswyfp3himrawavuaf4oassytyn7mf2:fldpglss2xejvz6sr4ajddbboq63iv3m:pxrvymyorqh7qxi4jrwmchi2yuuq4agf:qeczppkrcnyzcjdbhxtyax2bpuaxmwbz:mtro3r6zg3ok4gjom7v2qaxc3ehthmgx:thdbknwgmojuf7sdhxspchomfkxh6r2z:2xle4h4epawrbhxfxt5eq5n63bfosycn:bevzzjbflfsi6busqr3eu7vweidtjz2n:h2m5lu5karad2t4dmbncpazkwf2rmvhb:hdzs5uenwyrkk3jrcd2cd2qzpgxvccff:oy7evzedaypskacv3fxthcmhp6rvzymy:cbg6rwljtnw4ks7momqiypu4antmqxlv:5y2uye7ko2p6u6qyywcdecm7ijj7qmcm:uv6utq5fcsoxoq67hdqj6domvomgzsgc:jm6xhcbjozgvqaphc6soyjb33ofgmvwx:rvdrywf34tw757crvxseh7vfs42insur:4ijsvxdefkdjdyeod57hwegg6emmmle5:wsfzoxluyzq2s6c463tj4qfltfjiwl2v:zhfs2i7674sapcllgjgcauh73euc5ovi:fh4iwewj76yykzxyefgorzldzy557a2t:jqegoncucvxcryk4t3ay4cqfgl4yp5um:xuppk3u3bcwjxurcahunhvvr63f4it73:fexlv7zu4g5qlllgafsc3ob2qpkmrwft:ztu2zzdcr32japl4jd5gichynwa6mggi:zt4tcsx45jrkkqllohozvc7mu4gri7bl:3qjci5fw4vi5bq4mm2dopfjclz7aczu6:g3g26ugiy2oudh4pzlbpepf4ahvgihdw:rlboixbp3tv2lr26cxvgoaq7dr2ugwgz:66n75bdhfn77yrujghu36wgxn6zypl2k:ngplx3jzbydjcfh25ipta5vsuct64tmb:7m2fsilyutgiiyblyy6yd2tcpib4cvds:lvevadigsoqfzobcvsic5idn57dvt74k:rh7upcssc5pmgitoijytsqa64hwiiqlr:crujvlcawmzyti2vjlazyp7zfmko5my5:omdg6cusi57u4g3c5fquyiphfj4mictx:i6cnkc73c3uvx6gu3jnosrtbiadurhqz:2gie7vuqjqzr2vay3dotfyxrnwupufaw:yvgly67hrhryzg7vstjyhuvqhxtu44yp:mdk4c7oyvvfn5jn3jwtuqphqjudoacmr:os7jljyxz37e4ni4zi67xwo6k5e2ivkj"
declare -x SPACK_PYTHON="/usr/bin/python3"
declare -x SPACK_ROOT="/home/vagrant/spack"
declare -x SSH_AUTH_SOCK="/tmp/ssh-Z6U2mPJD7E/agent.2092"
declare -x SSH_CLIENT="10.0.2.2 51143 22"
declare -x SSH_CONNECTION="10.0.2.2 51143 10.0.2.15 22"
declare -x SSH_TTY="/dev/pts/0"
declare -x TERM="xterm-256color"
declare -x USER="vagrant"
declare -x WM_ARCH="linux64"
declare -x WM_ARCH_OPTION="64"
declare -x WM_CC="gcc"
declare -x WM_CFLAGS="-m64 -fPIC"
declare -x WM_COMPILER="Gcc"
declare -x WM_COMPILER_LIB_ARCH="64"
declare -x WM_COMPILER_TYPE="system"
declare -x WM_COMPILE_OPTION="-spack"
declare -x WM_CXX="g++"
declare -x WM_CXXFLAGS="-m64 -fPIC -std=c++0x"
declare -x WM_DIR="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/wmake"
declare -x WM_LABEL_OPTION="Int32"
declare -x WM_LABEL_SIZE="32"
declare -x WM_LDFLAGS="-m64"
declare -x WM_LINK_LANGUAGE="c++"
declare -x WM_MPLIB="SYSTEMMPI"
declare -x WM_OPTIONS="linux64GccDPInt32-spack"
declare -x WM_OSTYPE="POSIX"
declare -x WM_PRECISION_OPTION="DP"
declare -x WM_PROJECT="OpenFOAM"
declare -x WM_PROJECT_DIR="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2"
declare -x WM_PROJECT_INST_DIR="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0"
declare -x WM_PROJECT_USER_DIR="/home/vagrant/OpenFOAM/vagrant-6"
declare -x WM_PROJECT_VERSION="6"
declare -x WM_THIRD_PARTY_DIR="/home/vagrant/spack/opt/spack/linux-rocky8-sandybridge/gcc-8.5.0/openfoam-org-6-wqswyfp3himrawavuaf4oassytyn7mf2/ThirdParty"
declare -x XDG_RUNTIME_DIR="/run/user/1000"
declare -x XDG_SESSION_ID="5"
declare -x _sp_initializing
declare -x which_declare="declare -f"
ewquon commented 2 years ago

Hi Mike,

I haven't tried to use spack and a VM to create an OpenFOAM environment for SOWFA. However, I just verified that I was able to clone and build SOWFA without any issue. If you don't have OpenFAST (i.e., OPENFAST_DIR is not set), then it will not compile the turbine solver (but you'll still have the ABL LES solver, superDeliciousVanilla).

I'm guessing manually running wmake inside of the src subdirectories also doesn't work? You should be able to run wmake in any directory that contains a "Make" subdirectory. AFAIK, the only time mkdir would be called is to create the directory to hold the compiled objects, e.g., "Make/linux64GccDPInt64Opt" where the directory name depends on your OpenFOAM system configuration. I suspect something isn't set up properly in your OpenFOAM environment...

Eliot

mikerenfro commented 2 years ago

I've made another VM without Spack to test, same results, including if I do a wmake for a given folder:

Last login: Sun Aug 21 19:20:55 2022 from 10.0.2.2
[vagrant@localhost ~]$ module load mpi/openmpi-x86_64
[vagrant@localhost ~]$ source /opt/OpenFOAM-6/etc/bashrc
[vagrant@localhost ~]$ cd /opt/SOWFA-6/
[vagrant@localhost SOWFA-6]$ wmake src/miscTools
wmake src/miscTools
mkdir: cannot create directory ‘’: No such file or directory
make: *** [/opt/OpenFOAM-6/wmake/makefiles/general:167: /libSOWFAmiscTools.so] Error 1
[vagrant@localhost SOWFA-6]$ cd src/miscTools/
[vagrant@localhost miscTools]$ wmake
mkdir: cannot create directory ‘’: No such file or directory
make: *** [/opt/OpenFOAM-6/wmake/makefiles/general:167: /libSOWFAmiscTools.so] Error 1

And as of 2022-08-22, an entirely standard Rocky VM, without Vagrant and without Spack, fails the same way.

juleskerley commented 2 years ago

This was all tested on a fresh, standard Rocky VM, without Vagrant and without Spack, and it still fails the same way. To narrow down where the errors were sourced from, we modified the OpenFOAM-6 makeTargetDir file to show the $target and $dir variable values. 

[sw-arkerley@foam OpenFOAM-6]$ git diff wmake/scripts/makeTargetDir
diff --git a/wmake/scripts/makeTargetDir b/wmake/scripts/makeTargetDir
index de6722843..c55706f26 100755
--- a/wmake/scripts/makeTargetDir
+++ b/wmake/scripts/makeTargetDir
@@ -31,10 +31,11 @@
 #     Usage: makeTargetDir <directory>
 #
 #------------------------------------------------------------------------------
-
 for target
 do
+    echo $target is TARGET
     dir=${target%/*}
+    echo $dir is DIR
     [ -d "$dir" ] || [ "$dir" = "$target" ] || mkdir -p "$dir"
 done

From that, we can see that the error appears when $target is only one level of directory. The ones that work are all prefixed with Make and their architecture folder. The ones that fail are just the one level of folder and leading / which leaves $dir as an empty string. 

[sw-arkerley@foam SOWFA-6]$ ./Allwmake
In OpenFoam's wmake/wmake
wmake src/miscTools
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file geometricTransformations.C
Make/linux64GccDPInt32Opt/geometricTransformations/geometricTransformations.C.dep is TARGET
Make/linux64GccDPInt32Opt/geometricTransformations is DIR
Making dependency list for source file windRoseToCartesian.C
Make/linux64GccDPInt32Opt/windRoseToCartesian/windRoseToCartesian.C.dep is TARGET
Make/linux64GccDPInt32Opt/windRoseToCartesian is DIR
g++ -std=c++11 -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -O3  -DNoRepository -ftemplate-depth-100 -I/home/sw-arkerley/OpenFOAM-6/src/meshTools/lnInclude -I/home/sw-arkerley/OpenFOAM-6/src/finiteVolume/lnInclude -IlnInclude -I. -I/home/sw-arkerley/OpenFOAM-6/src/OpenFOAM/lnInclude -I/home/sw-arkerley/OpenFOAM-6/src/OSspecific/POSIX/lnInclude   -fPIC -c windRoseToCartesian/windRoseToCartesian.C -o Make/linux64GccDPInt32Opt/windRoseToCartesian/windRoseToCartesian.o
g++ -std=c++11 -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -O3  -DNoRepository -ftemplate-depth-100 -I/home/sw-arkerley/OpenFOAM-6/src/meshTools/lnInclude -I/home/sw-arkerley/OpenFOAM-6/src/finiteVolume/lnInclude -IlnInclude -I. -I/home/sw-arkerley/OpenFOAM-6/src/OpenFOAM/lnInclude -I/home/sw-arkerley/OpenFOAM-6/src/OSspecific/POSIX/lnInclude   -fPIC -c geometricTransformations/geometricTransformations.C -o Make/linux64GccDPInt32Opt/geometricTransformations/geometricTransformations.o
/libSOWFAmiscTools is TARGET
is DIR
mkdir: cannot create directory ‘’: No such file or directory
make: *** [/home/sw-arkerley/OpenFOAM-6/wmake/makefiles/general:167: /libSOWFAmiscTools.so] Error 1
ewquon commented 2 years ago

I should've asked this first, but did you source the OpenFOAM environment (OpenFOAM/etc/bashrc)? You should compile OpenFOAM first and sourcing the environment file would have been the first step.

mikerenfro commented 2 years ago

We are, both on the from-scratch, manually-installed VM and the Vagrant version. I'm defaulting to using Vagrant-provisioned VMs for the repeatability and automation, but this behavior has shown up in every Rocky 8 installation we've tried. Current Vagrantfile, so you can see what commands we're running both as root and as the unprivileged user compiling OpenFOAM and SOWFA:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "rockylinux/8"
  config.disksize.size = '50GB'
  config.vm.synced_folder '.', '/vagrant', disabled: true
  config.vm.provider "virtualbox" do |vb|
    vb.memory = "16384"
    vb.cpus = 4
  end
  config.vm.provision "shell", inline: <<-SHELLPRIV
    # make new partition and filesystem for /opt
    echo "n\np\n2\n\n\nw" | fdisk /dev/sda
    mkfs.xfs /dev/sda2
    echo /dev/sda2 /opt xfs defaults 0 0 >> /etc/fstab
    mount /opt
    chown -R vagrant:vagrant /opt

    yum -y update
    yum -y install redhat-lsb-core git gcc-c++ gcc-gfortran python3 openmpi flex bison zlib-devel
  SHELLPRIV
  config.vm.provision "shell", privileged: false, inline: <<-SHELLUNPRIV
    cd /opt
    git clone https://github.com/OpenFOAM/OpenFOAM-6.git
    git clone https://github.com/OpenFOAM/ThirdParty-6.git
    git clone https://github.com/NREL/SOWFA-6.git
    module load mpi/openmpi-x86_64
    cd OpenFOAM-6 ; source etc/bashrc ; cd ..
    cd ThirdParty-6 ; ./Allwmake -j >& ../ThirdParty-6.log ; cd ..
    cd OpenFOAM-6 ; ./Allwmake -j >& ../OpenFOAM-6.log ; cd ..
    cd SOWFA-6 ; git checkout dev ; ./Allwmake -j >& ../SOWFA-6.log
  SHELLUNPRIV
end

From this set of commands, we've got the same behavior in the first few lines of the /opt/SOWFA-6.log:

wmake src/miscTools
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file geometricTransformations.C
Making dependency list for source file windRoseToCartesian.C
g++ -std=c++11 -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -O3  -DNoRepository -ftemplate-depth-100 -I/opt/OpenFOAM-6/src/meshTools/lnInclude -I/opt/OpenFOAM-6/src/finiteVolume/lnInclude -IlnInclude -I. -I/opt/OpenFOAM-6/src/OpenFOAM/lnInclude -I/opt/OpenFOAM-6/src/OSspecific/POSIX/lnInclude   -fPIC -c windRoseToCartesian/windRoseToCartesian.C -o Make/linux64GccDPInt32Opt/windRoseToCartesian/windRoseToCartesian.o
g++ -std=c++11 -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -O3  -DNoRepository -ftemplate-depth-100 -I/opt/OpenFOAM-6/src/meshTools/lnInclude -I/opt/OpenFOAM-6/src/finiteVolume/lnInclude -IlnInclude -I. -I/opt/OpenFOAM-6/src/OpenFOAM/lnInclude -I/opt/OpenFOAM-6/src/OSspecific/POSIX/lnInclude   -fPIC -c geometricTransformations/geometricTransformations.C -o Make/linux64GccDPInt32Opt/geometricTransformations/geometricTransformations.o
mkdir: cannot create directory ‘’: No such file or directory
make: *** [/opt/OpenFOAM-6/wmake/makefiles/general:167: /libSOWFAmiscTools.so] Error 1
...
ewquon commented 2 years ago

I'm sorry I'm not familiar with VMs. Have you tried running through these commands in a non-VM environment before deploying to Vagrant? We've had numerous researchers compile SOWFA6 in different computing environments without issue, so I suspect there is some nuance to your VM setup that needs to be addressed.

Simon7537 commented 1 year ago

I'm sorry I'm not familiar with VMs. Have you tried running through these commands in a non-VM environment before deploying to Vagrant? We've had numerous researchers compile SOWFA6 in different computing environments without issue, so I suspect there is some nuance to your VM setup that needs to be addressed.

I got the same problem during compiling SOWFA-6 in a non-VM environment(I ran wmake inside the src directory):

mkdir: cannot create directory ‘’: No such file or directory make: *** [/libSOWFAincompressibleTurbulenceModels.so] Error 1

Actually I didn't use VM or spack, just compiled OpenFOAM-6 with icc. Perhaps it is not a VM problem? Has anyone got the same problem and successfully fixed it?