Bug during running a code in GPU when Resolution is increased but runs well in CPU

[atanuchaudhury]

Dear OpenLB team, I was running the Cylinder3d program while I am a new stl file which is extended at Y-axis and rest are the same. But while I am increasing the resolution (N) from 20 and above it does not run in ‘GPU’ but it is running well in ‘CPU’. When I used the ‘Gpu only’ config file it is giving an error: out of memory, so, I used the ‘Gpu openmpi’ config file and its giving error:

root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d# mpirun –allow-run-as-root -np 2 ./cylinder3d ./cylinder3d: symbol lookup error: /lib/x86_64-linux-gnu/libmpi_cxx.so.40: undefined symbol: ompi_mpi_errors_throw_exceptions ./cylinder3d: symbol lookup error: /lib/x86_64-linux-gnu/libmpi_cxx.so.40: undefined symbol: ompi_mpi_errors_throw_exceptions ————————————————————————– prterun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was:

Process name: [prterun-acmt-gpu-192644@1,0] Exit code: 127 ————————————————————————– root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d# mpicc -o cylinder3d cylinder3d.c gcc: error: cylinder3d.c: No such file or directory root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d# mpirun –version mpirun (Open MPI) 5.0.5

Report bugs to https://www.open-mpi.org/community/help/

Please help me to solve this bug.

[wenduwan]

@atanuchaudhury The error appears to be a linking issue between the application and MPI. Typically this should be mitigated on the application side. Have you reached out to Cylinder3d community?

root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d# mpirun –allow-run-as-root -np 2 ./cylinder3d
./cylinder3d: symbol lookup error: /lib/x86_64-linux-gnu/libmpi_cxx.so.40: undefined symbol: ompi_mpi_errors_throw_exceptions
./cylinder3d: symbol lookup error: /lib/x86_64-linux-gnu/libmpi_cxx.so.40: undefined symbol: ompi_mpi_errors_throw_exceptions

[atanuchaudhury]

I used it but the error now it gives: when I run this code in GPU it gives an error: [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [prepareLattice] Prepare Lattice ... OK [main] starting simulation... [setBoundaryValues] step=0; maxVel=0.0499999 [Timer] step=0; percent=0; passedTime=7.411; remTime=94853.4; MLUPs=0 [LatticeStatistics] step=0; t=0; uMax=0.0499999; avEnergy=2.49386e-06; avRho=1.0 0011 [getResults] pressure1=0; pressure2=0; pressureDrop=0; drag=0; lift=0 cylinder3d: ../../../src/utilities/vectorHelpers.h:204: std::vector<_Tp> olb::ut il::normalize(const std::vector<_Tp>&) [with T = float]: Assertion `scale>0' fai led. Aborted (core dumped)

[atanuchaudhury]

@atanuchaudhury The error appears to be a linking issue between the application and MPI. Typically this should be mitigated on the application side. Have you reached out to Cylinder3d community?

root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d# mpirun –allow-run-as-root -np 2 ./cylinder3d
./cylinder3d: symbol lookup error: /lib/x86_64-linux-gnu/libmpi_cxx.so.40: undefined symbol: ompi_mpi_errors_throw_exceptions
./cylinder3d: symbol lookup error: /lib/x86_64-linux-gnu/libmpi_cxx.so.40: undefined symbol: ompi_mpi_errors_throw_exceptions

Actually I have extended the along the Y axis which is upto 2 meter and rest are the same. So do I need to change the prepareGeometry setup? What changes should I make? Please reply

[bosilca]

I think you are mixing two versions of Open MPI. ompi_mpi_errors_throw_exceptions exists in the 4.x but not in the 5.0. So the question is how did you compile and link the cylinder3d application ? Can you do an ldd cylinder3d ? Also please add the output of ompi_info.

[atanuchaudhury]

I think you are mixing two versions of Open MPI. ompi_mpi_errors_throw_exceptions exists in the 4.x but not in the 5.0. So the question is how did you compile and link the cylinder3d application ? Can you do an ldd cylinder3d ? Also please add the output of ompi_info.

I am giving you the details what I have done till now.

my openmpi version: root@acmt-gpu:/home/achaudhury/olb-1.7r0# mpirun --version mpirun (Open MPI) 5.0.5

First the changes that I made in the Cylinder3D code: const int N = 50; // resolution of the model const T Re = 100.; // Reynolds number // //Uniform velocity profile instead of Poisullie velocity profile //New Vector<T,3> inletVelocity(3.,0.,0.) ; inletVelocity[0] = converter.getCharLatticeVelocity(); AnalyticalConst<3,T,T> uniformVelocity(inletVelocity); sLattice.defineU(superGeometry,3,uniformVelocity);

clout << "step=" << iT << "; maxVel=" << inletVelocity[0] << std::endl;

sLattice.setProcessingContext<Array<momenta::FixedVelocityMomentumGeneric::VELOCITY>>(
  ProcessingContext::Simulation);

} } //Rest are the same

Now I have made a new stl file in which I extended the Y axis to 2000 mm from the previous 410 mm and rest remains the same. So with Resolution, N = 10, it runs smoothly but as I increase the resolution say N = 20, 50 , 100 etc. and run it in GPU, the simulation gets abborted and shows error. I posted it to the Openlb forum but they are telling that its not the issue of mpirun. Please help me to solve it. The error is given below : root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d# mpirun --a llow-run-as-root -np 2 ./cylinder3d [ThreadPool] Sucessfully initialized, numThreads=1 [ThreadPool] Sucessfully initialized, numThreads=1 [GPU_CUDA] Found 4 CUDA devices but only one can be used per MPI process. [GPU_CUDA] Found 4 CUDA devices but only one can be used per MPI process. [UnitConverter] ----------------- UnitConverter information ----------------- [UnitConverter] -- Parameters: [UnitConverter] Resolution: N= 50 [UnitConverter] ----------------- UnitConverter information ----------------- [UnitConverter] -- Parameters: [UnitConverter] Resolution: N= 50 [UnitConverter] Lattice velocity: latticeU= 0.02 [UnitConverter] Lattice velocity: latticeU= 0.02 [UnitConverter] Lattice relaxation frequency: omega= 1.88679 [UnitConverter] Lattice relaxation time: tau= 0.53 [UnitConverter] Characteristical length(m): charL= 0.1 [UnitConverter] Characteristical speed(m/s): charU= 0.2 [UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 0.0002 [UnitConverter] Phys. density(kg/m^d): charRho= 1 [UnitConverter] Characteristical pressure(N/m^2): charPressure= 0 [UnitConverter] Mach number: machNumber= 0.034641 [UnitConverter] Reynolds number: reynoldsNumber= 100 [UnitConverter] Knudsen number: knudsenNumber= 0.00034641 [UnitConverter] [UnitConverter] -- Conversion factors: [UnitConverter] Voxel length(m): physDeltaX= 0.002 [UnitConverter] Time step(s): physDeltaT= 0.0002 [UnitConverter] Velocity factor(m/s): physVelocity= 10 [UnitConverter] Lattice relaxation frequency: omega= 1.88679 [UnitConverter] Lattice relaxation time: tau= 0.53 [UnitConverter] Characteristical length(m): charL= 0.1 [UnitConverter] Characteristical speed(m/s): charU= 0.2 [UnitConverter] Density factor(kg/m^3): physDensity= 1 [UnitConverter] Mass factor(kg): physMass= 8e-09 [UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.02 [UnitConverter] Force factor(N): physForce= 0.000400001 [UnitConverter] Pressure factor(N/m^2): physPressure= 100 [UnitConverter] ------------------------------------------------------------- [UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 0.0002 [UnitConverter] Phys. density(kg/m^d): charRho= 1 [UnitConverter] Characteristical pressure(N/m^2): charPressure= 0 [UnitConverter] Mach number: machNumber= 0.034641 [UnitConverter] Reynolds number: reynoldsNumber= 100 [UnitConverter] Knudsen number: knudsenNumber= 0.00034641 [UnitConverter] [UnitConverter] -- Conversion factors: [UnitConverter] Voxel length(m): physDeltaX= 0.002 [UnitConverter] Time step(s): physDeltaT= 0.0002 [UnitConverter] Velocity factor(m/s): physVelocity= 10 [UnitConverter] Density factor(kg/m^3): physDensity= 1 [UnitConverter] Mass factor(kg): physMass= 8e-09 [UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.02 [UnitConverter] Force factor(N): physForce= 0.000400001 [UnitConverter] Pressure factor(N/m^2): physPressure= 100 [UnitConverter] ------------------------------------------------------------- [prepareGeometry] Prepare Geometry ... [prepareGeometry] Prepare Geometry ... [SuperGeometry3D] cleaned 0 outer boundary voxel(s) [SuperGeometry3D] cleaned 0 outer boundary voxel(s) [SuperGeometry3D] cleaned 0 outer boundary voxel(s) [SuperGeometry3D] cleaned 0 outer boundary voxel(s) [SuperGeometryStatistics3D] updated [SuperGeometryStatistics3D] updated [SuperGeometry3D] the model is correct! [CuboidGeometry3D] ---Cuboid Stucture Statistics--- [CuboidGeometry3D] Number of Cuboids: 14 [CuboidGeometry3D] Delta (min): 0.002 [CuboidGeometry3D] (max): 0.002 [CuboidGeometry3D] Ratio (min): 0.578212 [CuboidGeometry3D] (max): 1.61353 [CuboidGeometry3D] Nodes (min): 18474750 [CuboidGeometry3D] (max): 18600606 [CuboidGeometry3D] Weight (min): 18351378 [CuboidGeometry3D] (max): 18598122 [CuboidGeometry3D] -------------------------------- [SuperGeometryStatistics3D] materialNumber=0; count=401274; minPhysR=(0.00450908 ,0.0425057,-0.000500045); maxPhysR=(0.546509,1.04651,0.4115) [SuperGeometryStatistics3D] materialNumber=1; count=255796267; minPhysR=(0.00050 908,0.000505742,0.00149996); maxPhysR=(2.49851,1.99851,0.4095) [SuperGeometryStatistics3D] materialNumber=2; count=3018812; minPhysR=(-0.001490 92,-0.00149426,-0.000500045); maxPhysR=(2.50051,2.00051,0.4115) [SuperGeometryStatistics3D] materialNumber=3; count=205000; minPhysR=(-0.0014909 2,0.000505742,0.00149996); maxPhysR=(-0.00149092,1.99851,0.4095) [SuperGeometryStatistics3D] materialNumber=4; count=205000; minPhysR=(2.50051,0. 000505742,0.00149996); maxPhysR=(2.50051,1.99851,0.4095) [SuperGeometryStatistics3D] materialNumber=5; count=55975; minPhysR=(0.00250908, 0.0405057,0.00149996); maxPhysR=(0.548509,1.04851,0.4095) [SuperGeometryStatistics3D] countTotal[1e6]=259.682 [prepareGeometry] Prepare Geometry ... OK [SuperGeometry3D] the model is correct! [CuboidGeometry3D] ---Cuboid Stucture Statistics--- [CuboidGeometry3D] Number of Cuboids: 14 [CuboidGeometry3D] Delta (min): 0.002 [CuboidGeometry3D] (max): 0.002 [CuboidGeometry3D] Ratio (min): 0.578212 [CuboidGeometry3D] (max): 1.61353 [CuboidGeometry3D] Nodes (min): 18474750 [CuboidGeometry3D] (max): 18600606 [CuboidGeometry3D] Weight (min): 18351378 [CuboidGeometry3D] (max): 18598122 [CuboidGeometry3D] -------------------------------- [SuperGeometryStatistics3D] materialNumber=0; count=401274; minPhysR=(0.00450908 ,0.0425057,-0.000500045); maxPhysR=(0.546509,1.04651,0.4115) [SuperGeometryStatistics3D] materialNumber=1; count=255796267; minPhysR=(0.00050 908,0.000505742,0.00149996); maxPhysR=(2.49851,1.99851,0.4095) [SuperGeometryStatistics3D] materialNumber=2; count=3018812; minPhysR=(-0.001490 92,-0.00149426,-0.000500045); maxPhysR=(2.50051,2.00051,0.4115) [SuperGeometryStatistics3D] materialNumber=3; count=205000; minPhysR=(-0.0014909 2,0.000505742,0.00149996); maxPhysR=(-0.00149092,1.99851,0.4095) [SuperGeometryStatistics3D] materialNumber=4; count=205000; minPhysR=(2.50051,0. 000505742,0.00149996); maxPhysR=(2.50051,1.99851,0.4095) [SuperGeometryStatistics3D] materialNumber=5; count=55975; minPhysR=(0.00250908, 0.0405057,0.00149996); maxPhysR=(0.548509,1.04851,0.4095) [SuperGeometryStatistics3D] countTotal[1e6]=259.682 [prepareGeometry] Prepare Geometry ... OK terminate called after throwing an instance of 'std::runtime_error' what(): out of memory

prterun noticed that process rank 0 with PID 10295 on node acmt-gpu exited on signal 6 (Aborted).

Please help me to solve it.

[bosilca]

Unfortunately, there is little we can help with.

1. You are running with Open MPI 5.0.5 but you have compiled, either your application or one of the libraries, with an older version, most certainly 4.x (which was that last version exposing ompi_mpi_errors_throw_exceptions.
2. you last run terminated with out of memory, which seems to indicate you are running a problem to large for the memory available, either on the nodes or on the GPUs.

[ggouaillardet]

@atanuchaudhury please run the following commands and post the outputs:

ldd cylinder3d
type mpirun
mpirun -V

[atanuchaudhury]

@atanuchaudhury please run the following commands and post the outputs:

ldd cylinder3d
type mpirun
mpirun -V

Ityped those commands and got these messages: root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d# mpirun -V mpirun (Open MPI) 5.0.5

Report bugs to http://www.open-mpi.org/community/help/ root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d# ldd cylinder3d linux-vdso.so.1 (0x00007fff35dd6000) libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1 (0x00007f18b6e00000) libcudart.so.11.0 => /usr/local/cuda-11.4/targets/x86_64-linux/lib/libcudart.so.11.0 (0x00007f18b6a00000) libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f18b67d4000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f18b6d19000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f18b85bc000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f18b65ab000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f18b85b5000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f18b85b0000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f18b85ab000) /lib64/ld-linux-x86-64.so.2 (0x00007f18b85eb000) root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d# mpirun --version mpirun (Open MPI) 5.0.5

Report bugs to http://www.open-mpi.org/community/help/ root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d#

[ggouaillardet]

this ticket started with Open MPI 5.0.5 and now it is 4.1.6. cylinder3d does not depend on libmpi.so, so unless it is doing something tricky, it is not an MPI program, and you should not start it with mpirun.

[atanuchaudhury]

this ticket started with Open MPI 5.0.5 and now it is 4.1.6. cylinder3d does not depend on libmpi.so, so unless it is doing something tricky, it is not an MPI program, and you should not start it with mpirun.

No sorry actually its not. I forgot to load the openmpi in the server thats why. The actual OpenMpi is 5.0.5.

[bosilca]

This mismatch between different versions of Open MPI is consistent with the error that started this issue. Please make sure your entire stack is compiled and run with a single version of Open MPI.

[atanuchaudhury]

This mismatch between different versions of Open MPI is consistent with the error that started this issue. Please make sure your entire stack is compiled and run with a single version of Open MPI.

I compiled the code Openmpi 5.0.5 version and also added "setNewSlipBoundary" and periodicity to the code: CuboidGeometry3D cuboidGeometry( extendedDomain, converter.getConversionFactorLength(), noOfCuboids ); //Added cuboidGeometry.setPeriodicity(true, false, false); But its giving an error. How to solve it? Please help me. root@acmt-gpu:/home/achaudhury/olb-1.7r0/examples/laminar/cylinder3d# ./cylinder3d [ThreadPool] Sucessfully initialized, numThreads=1 [GPU_CUDA] Found 4 CUDA devices but only one can be used per MPI process. [UnitConverter] ----------------- UnitConverter information ----------------- [UnitConverter] -- Parameters: [UnitConverter] Resolution: N= 20 [UnitConverter] Lattice velocity: latticeU= 0.0499999 [UnitConverter] Lattice relaxation frequency: omega= 1.88679 [UnitConverter] Lattice relaxation time: tau= 0.53 [UnitConverter] Characteristical length(m): charL= 0.1 [UnitConverter] Characteristical speed(m/s): charU= 0.2 [UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 0.0002 [UnitConverter] Phys. density(kg/m^d): charRho= 1 [UnitConverter] Characteristical pressure(N/m^2): charPressure= 0 [UnitConverter] Mach number: machNumber= 0.0866024 [UnitConverter] Reynolds number: reynoldsNumber= 100 [UnitConverter] Knudsen number: knudsenNumber= 0.000866025 [UnitConverter] [UnitConverter] -- Conversion factors: [UnitConverter] Voxel length(m): physDeltaX= 0.005 [UnitConverter] Time step(s): physDeltaT= 0.00125 [UnitConverter] Velocity factor(m/s): physVelocity= 4 [UnitConverter] Density factor(kg/m^3): physDensity= 1 [UnitConverter] Mass factor(kg): physMass= 1.25e-07 [UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.02 [UnitConverter] Force factor(N): physForce= 0.000400001 [UnitConverter] Pressure factor(N/m^2): physPressure= 16 [UnitConverter] ------------------------------------------------------------- [prepareGeometry] Prepare Geometry ... [SuperGeometry3D] cleaned 0 outer boundary voxel(s) [SuperGeometry3D] cleaned 0 outer boundary voxel(s) [SuperGeometryStatistics3D] updated [SuperGeometry3D] the model is correct! [CuboidGeometry3D] ---Cuboid Stucture Statistics--- [CuboidGeometry3D] Number of Cuboids: 7 [CuboidGeometry3D] Delta (min): 0.005 [CuboidGeometry3D] (max): 0.005 [CuboidGeometry3D] Ratio (min): 0.334661 [CuboidGeometry3D] (max): 2.18261 [CuboidGeometry3D] Nodes (min): 2412816 [CuboidGeometry3D] (max): 2427264 [CuboidGeometry3D] Weight (min): 2404944 [CuboidGeometry3D] (max): 2427264 [CuboidGeometry3D] -------------------------------- [SuperGeometryStatistics3D] materialNumber=0; count=19716; minPhysR=(0.461251,0.956251,-0.00374993); maxPhysR=(0.541251,1.04125,0.41125) [SuperGeometryStatistics3D] materialNumber=1; count=16374733; minPhysR=(0.00125108,0.00125084,0.00125007); maxPhysR=(2.49625,1.99625,0.41125) [SuperGeometryStatistics3D] materialNumber=2; count=484963; minPhysR=(-0.00374892,-0.00374916,-0.00374993); maxPhysR=(2.50125,2.00125,0.41125) [SuperGeometryStatistics3D] materialNumber=3; count=32800; minPhysR=(-0.00374892,0.00125084,0.00125007); maxPhysR=(-0.00374892,1.99625,0.40625) [SuperGeometryStatistics3D] materialNumber=4; count=32800; minPhysR=(2.50125,0.00125084,0.00125007); maxPhysR=(2.50125,1.99625,0.40625) [SuperGeometryStatistics3D] materialNumber=5; count=6524; minPhysR=(0.0812511,0.176251,0.00125007); maxPhysR=(0.546251,1.04625,0.40625) [SuperGeometryStatistics3D] countTotal[1e6]=16.9515 [prepareGeometry] Prepare Geometry ... OK [prepareLattice] Prepare Lattice ... [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [BlockGeometryStatistics3D] WARNING: no discreteNormal is found [prepareLattice] Prepare Lattice ... OK [main] starting simulation... [setBoundaryValues] step=0; maxVel=0.0499999 [Timer] step=0; percent=0; passedTime=7.506; remTime=96069.3; MLUPs=0 [LatticeStatistics] step=0; t=0; uMax=0.0499999; avEnergy=2.49386e-06; avRho=1.00011 [getResults] pressure1=0; pressure2=0; pressureDrop=0; drag=0; lift=0 cylinder3d: ../../../src/utilities/vectorHelpers.h:204: std::vector<_Tp> olb::util::normalize(const std::vector<_Tp>&) [with T = float]: Assertion `scale>0' failed. Aborted (core dumped)