Open mikerenfro opened 2 years ago
Hi Mike,
I haven't tried to use spack and a VM to create an OpenFOAM environment for SOWFA. However, I just verified that I was able to clone and build SOWFA without any issue. If you don't have OpenFAST (i.e., OPENFAST_DIR
is not set), then it will not compile the turbine solver (but you'll still have the ABL LES solver, superDeliciousVanilla
).
I'm guessing manually running wmake
inside of the src subdirectories also doesn't work? You should be able to run wmake
in any directory that contains a "Make" subdirectory. AFAIK, the only time mkdir
would be called is to create the directory to hold the compiled objects, e.g., "Make/linux64GccDPInt64Opt" where the directory name depends on your OpenFOAM system configuration. I suspect something isn't set up properly in your OpenFOAM environment...
Eliot
I've made another VM without Spack to test, same results, including if I do a wmake
for a given folder:
Last login: Sun Aug 21 19:20:55 2022 from 10.0.2.2
[vagrant@localhost ~]$ module load mpi/openmpi-x86_64
[vagrant@localhost ~]$ source /opt/OpenFOAM-6/etc/bashrc
[vagrant@localhost ~]$ cd /opt/SOWFA-6/
[vagrant@localhost SOWFA-6]$ wmake src/miscTools
wmake src/miscTools
mkdir: cannot create directory ‘’: No such file or directory
make: *** [/opt/OpenFOAM-6/wmake/makefiles/general:167: /libSOWFAmiscTools.so] Error 1
[vagrant@localhost SOWFA-6]$ cd src/miscTools/
[vagrant@localhost miscTools]$ wmake
mkdir: cannot create directory ‘’: No such file or directory
make: *** [/opt/OpenFOAM-6/wmake/makefiles/general:167: /libSOWFAmiscTools.so] Error 1
And as of 2022-08-22, an entirely standard Rocky VM, without Vagrant and without Spack, fails the same way.
This was all tested on a fresh, standard Rocky VM, without Vagrant and without Spack, and it still fails the same way.
To narrow down where the errors were sourced from, we modified the OpenFOAM-6 makeTargetDir
file to show the $target
and $dir
variable values.
[sw-arkerley@foam OpenFOAM-6]$ git diff wmake/scripts/makeTargetDir
diff --git a/wmake/scripts/makeTargetDir b/wmake/scripts/makeTargetDir
index de6722843..c55706f26 100755
--- a/wmake/scripts/makeTargetDir
+++ b/wmake/scripts/makeTargetDir
@@ -31,10 +31,11 @@
# Usage: makeTargetDir <directory>
#
#------------------------------------------------------------------------------
-
for target
do
+ echo $target is TARGET
dir=${target%/*}
+ echo $dir is DIR
[ -d "$dir" ] || [ "$dir" = "$target" ] || mkdir -p "$dir"
done
From that, we can see that the error appears when $target
is only one level of directory. The ones that work are all prefixed with Make and their architecture folder. The ones that fail are just the one level of folder and leading /
which leaves $dir
as an empty string.
[sw-arkerley@foam SOWFA-6]$ ./Allwmake
In OpenFoam's wmake/wmake
wmake src/miscTools
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file geometricTransformations.C
Make/linux64GccDPInt32Opt/geometricTransformations/geometricTransformations.C.dep is TARGET
Make/linux64GccDPInt32Opt/geometricTransformations is DIR
Making dependency list for source file windRoseToCartesian.C
Make/linux64GccDPInt32Opt/windRoseToCartesian/windRoseToCartesian.C.dep is TARGET
Make/linux64GccDPInt32Opt/windRoseToCartesian is DIR
g++ -std=c++11 -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -O3 -DNoRepository -ftemplate-depth-100 -I/home/sw-arkerley/OpenFOAM-6/src/meshTools/lnInclude -I/home/sw-arkerley/OpenFOAM-6/src/finiteVolume/lnInclude -IlnInclude -I. -I/home/sw-arkerley/OpenFOAM-6/src/OpenFOAM/lnInclude -I/home/sw-arkerley/OpenFOAM-6/src/OSspecific/POSIX/lnInclude -fPIC -c windRoseToCartesian/windRoseToCartesian.C -o Make/linux64GccDPInt32Opt/windRoseToCartesian/windRoseToCartesian.o
g++ -std=c++11 -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -O3 -DNoRepository -ftemplate-depth-100 -I/home/sw-arkerley/OpenFOAM-6/src/meshTools/lnInclude -I/home/sw-arkerley/OpenFOAM-6/src/finiteVolume/lnInclude -IlnInclude -I. -I/home/sw-arkerley/OpenFOAM-6/src/OpenFOAM/lnInclude -I/home/sw-arkerley/OpenFOAM-6/src/OSspecific/POSIX/lnInclude -fPIC -c geometricTransformations/geometricTransformations.C -o Make/linux64GccDPInt32Opt/geometricTransformations/geometricTransformations.o
/libSOWFAmiscTools is TARGET
is DIR
mkdir: cannot create directory ‘’: No such file or directory
make: *** [/home/sw-arkerley/OpenFOAM-6/wmake/makefiles/general:167: /libSOWFAmiscTools.so] Error 1
I should've asked this first, but did you source the OpenFOAM environment (OpenFOAM/etc/bashrc
)? You should compile OpenFOAM first and sourcing the environment file would have been the first step.
We are, both on the from-scratch, manually-installed VM and the Vagrant version. I'm defaulting to using Vagrant-provisioned VMs for the repeatability and automation, but this behavior has shown up in every Rocky 8 installation we've tried. Current Vagrantfile, so you can see what commands we're running both as root and as the unprivileged user compiling OpenFOAM and SOWFA:
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
config.vm.box = "rockylinux/8"
config.disksize.size = '50GB'
config.vm.synced_folder '.', '/vagrant', disabled: true
config.vm.provider "virtualbox" do |vb|
vb.memory = "16384"
vb.cpus = 4
end
config.vm.provision "shell", inline: <<-SHELLPRIV
# make new partition and filesystem for /opt
echo "n\np\n2\n\n\nw" | fdisk /dev/sda
mkfs.xfs /dev/sda2
echo /dev/sda2 /opt xfs defaults 0 0 >> /etc/fstab
mount /opt
chown -R vagrant:vagrant /opt
yum -y update
yum -y install redhat-lsb-core git gcc-c++ gcc-gfortran python3 openmpi flex bison zlib-devel
SHELLPRIV
config.vm.provision "shell", privileged: false, inline: <<-SHELLUNPRIV
cd /opt
git clone https://github.com/OpenFOAM/OpenFOAM-6.git
git clone https://github.com/OpenFOAM/ThirdParty-6.git
git clone https://github.com/NREL/SOWFA-6.git
module load mpi/openmpi-x86_64
cd OpenFOAM-6 ; source etc/bashrc ; cd ..
cd ThirdParty-6 ; ./Allwmake -j >& ../ThirdParty-6.log ; cd ..
cd OpenFOAM-6 ; ./Allwmake -j >& ../OpenFOAM-6.log ; cd ..
cd SOWFA-6 ; git checkout dev ; ./Allwmake -j >& ../SOWFA-6.log
SHELLUNPRIV
end
From this set of commands, we've got the same behavior in the first few lines of the /opt/SOWFA-6.log:
wmake src/miscTools
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file geometricTransformations.C
Making dependency list for source file windRoseToCartesian.C
g++ -std=c++11 -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -O3 -DNoRepository -ftemplate-depth-100 -I/opt/OpenFOAM-6/src/meshTools/lnInclude -I/opt/OpenFOAM-6/src/finiteVolume/lnInclude -IlnInclude -I. -I/opt/OpenFOAM-6/src/OpenFOAM/lnInclude -I/opt/OpenFOAM-6/src/OSspecific/POSIX/lnInclude -fPIC -c windRoseToCartesian/windRoseToCartesian.C -o Make/linux64GccDPInt32Opt/windRoseToCartesian/windRoseToCartesian.o
g++ -std=c++11 -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -O3 -DNoRepository -ftemplate-depth-100 -I/opt/OpenFOAM-6/src/meshTools/lnInclude -I/opt/OpenFOAM-6/src/finiteVolume/lnInclude -IlnInclude -I. -I/opt/OpenFOAM-6/src/OpenFOAM/lnInclude -I/opt/OpenFOAM-6/src/OSspecific/POSIX/lnInclude -fPIC -c geometricTransformations/geometricTransformations.C -o Make/linux64GccDPInt32Opt/geometricTransformations/geometricTransformations.o
mkdir: cannot create directory ‘’: No such file or directory
make: *** [/opt/OpenFOAM-6/wmake/makefiles/general:167: /libSOWFAmiscTools.so] Error 1
...
I'm sorry I'm not familiar with VMs. Have you tried running through these commands in a non-VM environment before deploying to Vagrant? We've had numerous researchers compile SOWFA6 in different computing environments without issue, so I suspect there is some nuance to your VM setup that needs to be addressed.
I'm sorry I'm not familiar with VMs. Have you tried running through these commands in a non-VM environment before deploying to Vagrant? We've had numerous researchers compile SOWFA6 in different computing environments without issue, so I suspect there is some nuance to your VM setup that needs to be addressed.
I got the same problem during compiling SOWFA-6 in a non-VM environment(I ran wmake inside the src directory):
mkdir: cannot create directory ‘’: No such file or directory make: *** [/libSOWFAincompressibleTurbulenceModels.so] Error 1
Actually I didn't use VM or spack, just compiled OpenFOAM-6 with icc. Perhaps it is not a VM problem? Has anyone got the same problem and successfully fixed it?
Hey, folks. I'm neither an OpenFOAM nor a SOWFA user, but I'm trying to get both installed on our HPC running Rocky and Spack, and as an intermedia step, I'm trying to install it on a Vagrant VM to iterate more quickly.
The Vagrantfile I'm using:
After logging back in as the vagrant user and loading the spack settings via sourcing spack/share/spack/setup-env.sh, I try to run
Allwclean
andAllwmake
from the SOWFA-6 directory.Allwclean
returns:which I don't think is a problem, while
Allwmake
returns:and this pattern repeats for several other
wmake
commands. Line 167 is@$(WM_SCRIPTS)/makeTargetDir $(LIB)
and line 80 definesLIB = libNULL
, but that value must have been overridden somewhere before line 167.What other information can I provide to help figure out what's going on? My current environment variables via
export
are: