Closed rodionstepanov closed 9 months ago
Hi @rodionstepanov,
it seems you have an older version of the gcc
compiler that does not yet support C++17. You need at least gcc
version 9; check with gcc --version
.
Possible solutions:
gcc
version, so in make.sh
instead of g++
, write g++-9
or g++-11
.module load gcc
.utilities.hpp
line 10: #define UTILITIES_NO_CPP17
. Then it should compile also with an older version of gcc
. The only C++17 function that otherwise is used is to automatically create missing directories for file export with std::filesystem
. If this is deactivated, all directories where files are exported have to be manually created beforehand, or otherwise files will not be written to the hard drive and the data is lost.Please let me know if this works.
Kind regards, Moritz
thank you @ProjectPhysX! Indeed there is only gcc (GCC) 8.5.0
I uncommented #define UTILITIES_NO_CPP17
. The code then compiles successfully. I've got a strange behavior in memory allocation. Is it because of old compiler?
Result for run on single GPU with const uint memory = 2088u;
.-----------------------------------------------------------------------------.
| __ __ |
| \ ____ | | ____ / |
| \ \ | | | | / / |
| \ \ | | | | / / |
| \ \ | | | | / / |
| \ _.-" | | "-./ / |
| \ .-" "-. / |
| .-" .-" "-. "-./ |
| .-" .-"-. "-. |
| \ v" "v / |
| \ \ / / |
| \ \ / / |
| \ \ / / |
| \ ' / |
| \ / |
| \ / FluidX3D Version 2.6 |
| ' Copyright (c) Moritz Lehmann |
|----------------.------------------------------------------------------------|
| Device ID 0 | Tesla M2090 |
|----------------'------------------------------------------------------------|
|----------------.------------------------------------------------------------|
| Device ID | 0 |
| Device Name | Tesla M2090 |
| Device Vendor | NVIDIA Corporation |
| Device Driver | 390.132 |
| OpenCL Version | OpenCL C 1.1 |
| Compute Units | 16 at 1301 MHz (2048 cores, 5.329 TFLOPs/s) |
| Memory, Cache | 6067 MB, 256 KB global / 48 KB local |
| Buffer Limits | 1516 MB global, 64 KB constant |
|----------------'------------------------------------------------------------|
| Info: OpenCL C code successfully compiled. |
|-----------------.-----------------------------------------------------------|
| Grid Resolution | 286 x 286 x 286 = 23393656 |
| Grid Domains | 1 x 1 x 1 = 1 |
| LBM Type | D3Q19 SRT (FP32/FP32) |
| Memory Usage | CPU 379 MB, GPU 1x 2073 MB |
| Max Alloc Size | 1695 MB |
| Time Steps | 10 |
| Kin. Viscosity | 1.00000000 |
| Relaxation Time | 3.50000000 |
| Reynolds Number | Re < 165 |
|---------.-------'-----.-----------.-------------------.---------------------|
| MLUPs | Bandwidth | Steps/s | Current Step | Time Remaining |
| 832 | 127 GB/s | 36 | 998 80% | 0s |
|---------'-------------'-----------'-------------------'---------------------|
| Info: Peak MLUPs/s = 832
Result for two GPUs
.-----------------------------------------------------------------------------. | __ __ | | \ ____ | | ____ / | | \ \ | | | | / / | | \ \ | | | | / / | | \ \ | | | | / / | | \ _.-" | | "-./ / | | \ .-" "-. / | | .-" .-" "-. "-./ | | .-" .-"-. "-. | | \ v" "v / | | \ \ / / | | \ \ / / | | \ \ / / | | \ ' / | | \ / | | \ / FluidX3D Version 2.6 | | ' Copyright (c) Moritz Lehmann | |----------------.------------------------------------------------------------| | Device ID 0 | Tesla M2090 | | Device ID 1 | Tesla M2090 | |----------------'------------------------------------------------------------| |----------------.------------------------------------------------------------| | Device ID | 0 | | Device Name | Tesla M2090 | | Device Vendor | NVIDIA Corporation | | Device Driver | 390.132 | | OpenCL Version | OpenCL C 1.1 | | Compute Units | 16 at 1301 MHz (2048 cores, 5.329 TFLOPs/s) | | Memory, Cache | 6067 MB, 256 KB global / 48 KB local | | Buffer Limits | 1516 MB global, 64 KB constant | |----------------'------------------------------------------------------------| | Info: OpenCL C code successfully compiled. | |----------------.------------------------------------------------------------| | Device ID | 1 | | Device Name | Tesla M2090 | | Device Vendor | NVIDIA Corporation | | Device Driver | 390.132 | | OpenCL Version | OpenCL C 1.1 | | Compute Units | 16 at 1301 MHz (2048 cores, 5.329 TFLOPs/s) | | Memory, Cache | 6067 MB, 256 KB global / 48 KB local | | Buffer Limits | 1516 MB global, 64 KB constant | |----------------'------------------------------------------------------------| | Info: OpenCL C code successfully compiled. | |-----------------.-----------------------------------------------------------| | Grid Resolution | 572 x 286 x 286 = 46787312 | | Grid Domains | 2 x 1 x 1 = 2 | | LBM Type | D3Q19 SRT (FP32/FP32) | | Memory Usage | CPU 758 MB, GPU 2x 2089 MB | | Max Alloc Size | 1695 MB | | Time Steps | 10 | | Kin. Viscosity | 1.00000000 | | Relaxation Time | 3.50000000 | | Reynolds Number | Re < 165 | |---------.-------'-----.-----------.-------------------.---------------------| | MLUPs | Bandwidth | Steps/s | Current Step | Time Remaining | | 1078 | 165 GB/s | 23 | 1000 100% | 0s | |---------'-------------'-----------'-------------------'---------------------| | Info: Peak MLUPs/s = 1104 |
Run for 4 GPUs was failed .-----------------------------------------------------------------------------. | __ __ | | \ ____ | | ____ / | | \ \ | | | | / / | | \ \ | | | | / / | | \ \ | | | | / / | | \ _.-" | | "-./ / | | \ .-" "-. / | | .-" .-" "-. "-./ | | .-" .-"-. "-. | | \ v" "v / | | \ \ / / | | \ \ / / | | \ \ / / | | \ ' / | | \ / | | \ / FluidX3D Version 2.6 | | ' Copyright (c) Moritz Lehmann | |----------------.------------------------------------------------------------| | Device ID 0 | Tesla M2090 | | Device ID 1 | Tesla M2090 | | Device ID 2 | Tesla M2090 | | Device ID 3 | Tesla M2090 | |----------------'------------------------------------------------------------| |----------------.------------------------------------------------------------| | Device ID | 0 | | Device Name | Tesla M2090 | | Device Vendor | NVIDIA Corporation | | Device Driver | 390.132 | | OpenCL Version | OpenCL C 1.1 | | Compute Units | 16 at 1301 MHz (2048 cores, 5.329 TFLOPs/s) | | Memory, Cache | 6067 MB, 256 KB global / 48 KB local | | Buffer Limits | 1516 MB global, 64 KB constant | |----------------'------------------------------------------------------------| | Info: OpenCL C code successfully compiled. | |----------------.------------------------------------------------------------| | Device ID | 1 | | Device Name | Tesla M2090 | | Device Vendor | NVIDIA Corporation | | Device Driver | 390.132 | | OpenCL Version | OpenCL C 1.1 | | Compute Units | 16 at 1301 MHz (2048 cores, 5.329 TFLOPs/s) | | Memory, Cache | 6067 MB, 256 KB global / 48 KB local | | Buffer Limits | 1516 MB global, 64 KB constant | |----------------'------------------------------------------------------------| | Info: OpenCL C code successfully compiled. | |----------------.------------------------------------------------------------| | Device ID | 2 | | Devicslurmstepd: error: Detected 1 oom-kill event(s) in step 16316445.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. srun: error: tesla43: task 0: Out Of Memory
Wow you have some awesome old hardware! The oom-kill event
means that Linux did run out of memory (oom) and had to kill the process in order to keep the operating system running. Do you have enough CPU RAM installed?
yes, rather old but why not have a fun :)
This node has:
Two 8-core Intel® Xeon® E5-2660 (2.2 GHz) processors
main memory 96 GB
2 x 20 MB cache Level 2 cache
8 Tesla M2090 GPUs (6GB Global Memory)
Do you have enough CPU RAM installed?
@ProjectPhysX I think, yes, I do. Since the run on 1 GPU didn't crush. But for 4 GPU means, total memory 4*memory, exceeds 6GB. I don't understand. Is it strange memory allocation because of old compiler?
Hi @rodionstepanov,
I think this an issue with slurm limiting the amount of CPU memory that the application can occupy. Add the option --mem-per-cpu, or --mem (per node) to make slurm request more memory for the job. Let me know it this works!
Kind regards, Moritz
When trying to compile on Linux (without X11) below errors appear. What is wrong?
/tmp/ccDlRpNU.o: In function, std::allocator >, std::filesystem:: cxx11::path>(std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::filesystem:: cxx11::path::format)':
lbm.cpp:(.text._ZNSt10filesystem7cxx114pathC2INSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_EERKT_NS1_6formatE[_ZNSt10filesystem7cxx114pathC5INSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_EERKT_NS1_6formatE]+0x64): undefined reference to `std::filesystem::__cxx11::path::_M_split_cmpts()'
collect2: error: ld returned 1 exit status
std::filesystem::exists(std::filesystem::__cxx11::path const&)': lbm.cpp:(.text._ZNSt10filesystem6existsERKNS_7__cxx114pathE[_ZNSt10filesystem6existsERKNS_7__cxx114pathE]+0x14): undefined reference to
std::filesystem::status(std::filesystem::cxx11::path const&)' /tmp/ccDlRpNU.o: In function `std::filesystem::is_directory(std::filesystem::__cxx11::path const&)': lbm.cpp:(.text._ZNSt10filesystem12is_directoryERKNS_7cxx114pathE[_ZNSt10filesystem12is_directoryERKNS_7cxx114pathE]+0x14): undefined reference to `std::filesystem::status(std::filesystem::cxx11::path const&)' /tmp/ccDlRpNU.o: In functioncreate_folder(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)': lbm.cpp:(.text._Z13create_folderRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_Z13create_folderRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x13e): undefined reference to
std::filesystem::create_directories(std::filesystem::cxx11::path const&)' /tmp/ccDlRpNU.o: In function `std::filesystem::cxx11::path::path<std::cxx11::basic_string<char, std::char_traits