Open santiagoMonedero opened 1 year ago
Hi, after some digging into this I think I have a working solution. I have done it directly in the container but I guess you can add it to the Dockerfile or the source code.
--allow-run-as-root
into the mpirun
command. Probably here: https://github.com/firelab/windninja/blob/2c971aeeaa66b7f2614d895eaac23f9cd537d974/src/ninja/ninjafoam.cpp#L1216
1.B exporting OMPI_ALLOW_RUN_AS_ROOT=1
and OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
which is what I did. Before doing so however run the docker interactively and execute mpirun
to get the following scary message which, to be honest, I have absolutely no clue how relevant it is or it is not being inside a Docker (but it is definitely very scary):
root@1d33a1a7e569:/data# mpirun
--------------------------------------------------------------------------
mpirun has detected an attempt to run as root.
Running as root is strongly discouraged as any mistake (e.g., in defining TMPDIR) or bug can result in catastrophic damage to the OS file system, leaving your system in an unusable state.
We strongly suggest that you run mpirun as a non-root user.
2. Once you have root privileges you can run **OpenFOAM** with **MPI** but you will get a convergence problem when trying their test cases like $`FOAM_TUTORIALS/incompressible/simpleFoam/motorBike`, or you will get stucked into the `(moveDynamicMesh) 100% complete...` in Windninja without ever getting into the domain reconstruction part. Apparently this is an MPI known issue (see https://github.com/open-mpi/ompi/issues/4948) which can be fixed by adding the environmental variable `export OMPI_MCA_btl_vader_single_copy_mechanism=none`
And that's it !, we have the Docker container running with momentum conservation and 25 threads
root@d0a02ed7692b:/home/wind/example# WindNinja_cli cli_momentumSolver_diurnal.cfg Run 0: Reading elevation file... Run 0: Simulation time is 2011-Sep-23 13:30:00 MDT Run 0: Run number 0 started with 25 threads. Run 0: Writing OpenFOAM files... Run 0: Converting DEM to STL format... Run 0: Transforming surface points to output wind height... Run 0: Generating mesh... Run 0: Running blockMesh... Run 0: Decomposing domain for parallel mesh calculations... Run 0: Running moveDynamicMesh... Run 0: (moveDynamicMesh) 4% complete... Run 0: (moveDynamicMesh) 12% complete... Run 0: (moveDynamicMesh) 20% complete... Run 0: (moveDynamicMesh) 26% complete... Run 0: (moveDynamicMesh) 34% complete... Run 0: (moveDynamicMesh) 42% complete... Run 0: (moveDynamicMesh) 48% complete... Run 0: (moveDynamicMesh) 56% complete... Run 0: (moveDynamicMesh) 62% complete... Run 0: (moveDynamicMesh) 70% complete... Run 0: (moveDynamicMesh) 78% complete... Run 0: (moveDynamicMesh) 84% complete... Run 0: (moveDynamicMesh) 92% complete... Run 0: (moveDynamicMesh) 98% complete... Run 0: Reconstructing domain... Run 0: Refining surface cells in mesh... Run 0: (refineMesh) 10% complete... Run 0: (refineMesh) 99% complete... Run 0: Renumbering mesh... Run 0: Applying initial conditions... Run 0: Decomposing domain for parallel flow calculations... Run 0: Solving for the flow field... Run 0 (solver): 2% complete Run 0 (solver): 2% complete Run 0 (solver): 2% complete
Notice that I am not an expert on this and this is simply a working solution I got by googling and playing a bit with the Docker.
PS: The Docker approach is important for us to run Azure HPC Batching with momentum conservation. For our local HPC I will have to do some testing because we use **Singularity** and it is more restrictive with privileges ... in case it fails maybe using openFoam image as base instead of Ubuntu 20.04 may be a solution (well..... just thinking out loud)
@santiagoMonedero Glad you got something working for now. I'm also not a Docker or MPI expert so am not sure off the top of my head if there is a better way to handle this. I'll leave this ticket open and will try to take a closer look soon. Thanks for reporting your fix here.
Hi, I am trying to run the example file
cli_momentumSolver_diurnal.cfg
using the Dockerfile in the respository (v3.9) but it fails when using more than 1 thread. All I have done is 1) clone repository, 2) build dockerfile and 3) run the docker interactively to access WindNinja_cli.Surprisingly it gives different errors when creating and using the image in Ubuntu 20.04 through WSL2 in windows 11, and directly using Ubuntu 18.04 without WSL
Ubuntu 18.04
Ubuntu 20.04 with WSL on windows 11
PS: I know this was a known issue in a previous windninja version but just wanted to give it a try on the new release and check if there is no work around it yet. Thanks!!