MarjanLukezic commented 6 years ago

Problem with solver glmat in fds, FDS SMV ver.:

Current Date : February 11, 2018 20:47:53 Version : FDS 6.6.0 Revision : FDS6.6.0-131-g88ae75a-HEAD Revision Date : Wed Nov 1 16:03:29 2017 -0400 Compiler : Intel ifort 17.0.4 Compilation Date : Nov 01, 2017 21:03:34

My HW info: AMD Ryzen Threadripper 1950X 16-Core Processor MemTotal: 16353488 kB

Software: OS: CentOS 7.4 (64 bit)

When I run ulimit -a I got this set of info.: [mlukezic@localhost ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63490 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

Please, don't do anything else. First, paste these lines into your .bashrc file.

============= FDS environment =======

export PATH=/home/mlukezic/FDS/FDS6/bin:$PATH export LD_LIBRARY_PATH=/usr/lib64:/home/mlukezic/FDS/FDS6/bin/LIB64:$LD_LIBRARY$ export OMP_NUM_THREADS=8

ulimit -s unlimited

Salah Benkorichi adviced this code for bash I_MPI_SHM_LMT=shm

============= FDS environment =======

Don't put the nbr of threads to 16, to me that appears you are having hyperthreadening, and this will only slow the run instead of increasing it. If you want to run single MPI, then keep it to 8, if you want use MPI, then reduce it to 1, or 2 depending on how many MPI processes you're intending to use.

log off and login again. Run the case again and report what you get.

When I run: $ fds glmat.fds I got notice: Mesh 1 is assigned to MPI Process 0 OpenMP thread 0 of 7 assigned to MPI process 0 of 0 OpenMP thread 5 of 7 assigned to MPI process 0 of 0 OpenMP thread 7 of 7 assigned to MPI process 0 of 0 OpenMP thread 3 of 7 assigned to MPI process 0 of 0 OpenMP thread 1 of 7 assigned to MPI process 0 of 0 OpenMP thread 2 of 7 assigned to MPI process 0 of 0 OpenMP thread 4 of 7 assigned to MPI process 0 of 0 OpenMP thread 6 of 7 assigned to MPI process 0 of 0 Completed Initialization Step 1 Completed Initialization Step 2 Completed Initialization Step 3 Completed Initialization Step 4

Using GLMAT as pressure solver. List of H unknown numbers per proc: MYID= 0, NUNKH_LOCAL= 256000

forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source
fds 00000000059AAB24 Unknown Unknown Unknown libpthread-2.17.s 00002B99466B45E0 Unknown Unknown Unknown libc-2.17.so 00002B99481204FC cfree Unknown Unknown ld-2.17.so 00002B99460E2F22 Unknown Unknown Unknown ld-2.17.so 00002B99460E5712 Unknown Unknown Unknown ld-2.17.so 00002B99460F0964 Unknown Unknown Unknown ld-2.17.so 00002B99460EC2D4 Unknown Unknown Unknown ld-2.17.so 00002B99460F02CB Unknown Unknown Unknown libdl-2.17.so 00002B9946BC3FBB Unknown Unknown Unknown ld-2.17.so 00002B99460EC2D4 Unknown Unknown Unknown libdl-2.17.so 00002B9946BC45BD Unknown Unknown Unknown libdl-2.17.so 00002B9946BC4051 dlopen Unknown Unknown

fds 000000000044D9A7 Unknown Unknown Unknown fds 000000000044DCBD Unknown Unknown Unknown fds 00000000005404EB Unknown Unknown Unknown fds 000000000045BC37 Unknown Unknown Unknown fds 0000000000407C7F Unknown Unknown Unknown fds 000000000610478B Unknown Unknown Unknown fds 0000000005A8DEFD MAIN 252 main.f90 fds 000000000040784E Unknown Unknown Unknown libc-2.17.so 00002B99480C1C05 libc_start_main Unknown Unknown fds 0000000000407729 Unknown

Even with this code: &HEAD CHID='glmat', TITLE='Compart_Test'/ &TIME T_END=360.0/

&MESH IJK=21,10,10, XB=0.0,2.1,0.0,1.0,0.0,1.0 /

&REAC ID='React_Methane', FUEL='METHANE', CO_YIELD=0.1, SOOT_YIELD=0.01/

&PRES SOLVER='GLMAT'/

&TAIL / I got the same notice.

sbenkorichi commented 6 years ago

Thanks we'll take a look. @marcosvanella This example works fine on my end on both windows and Linux, can you check please why this user is receiving such error. Thanks !

marcosvanella commented 6 years ago

Hi Marjan, thank you for reporting. Thank you Salah for checking on your end. I haven't been able to reproduce the problem for this simplified test case you've submitted. I tested the 6.6.0 release versions on a local linux machine which has CentOS 6.2, and also on my Mac. A couple of tests I'd like you to do:

Try setting OMP_NUM_THREADS=1 and re-running the simplified test.
Try invoking mpiexec -n 1 fds case.fds. I've noted in the past different behavior when using only fds, or invoking mpiexec from bundle.
Try putting an vent of type 'OPEN'anywhere on the boundary of the domain. Let me know what happens.

Salah, you use Ubuntu on you linux machines, correct? Thank you.

sbenkorichi commented 6 years ago

@marcosvanella Might be worth checking how the discussion started here This one of the weirdos behaviors that happens. Other examples run fine, but when using GLMAT solver it just doesn't work. Something might be related to his machine. Yes, I use Linux mint 18.3 I haven't tried though using the debugged version around the compiled bundle date to check just in case. I will try it at later time and let you know.

marcosvanella commented 6 years ago

Marjan, I see your stack size is not set to unlimited (stack size (kbytes, -s) 8192) even though you set ulimit -s unlimited make sure this gets set properly.

sbenkorichi commented 6 years ago

Marcos, Actually, it's already set after I suggested so, here:

Outuput:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63490
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Check the link of the discussion above. I don't believe it's related to stack size. I did ask him to run a simple case beside glmat, it worked, and then when I asked him to use glmat on it, it failed.

marcosvanella commented 6 years ago

All right, lets see what Marjan finds. Marjan also try this, go to your FDS6/bin dir, check that you have the fds executable there and type: $ldd fds

report back what it says. Thank you.

marcosvanella commented 6 years ago

Marjan, I'm creating a Centos 7 virtual machine on my Mac to see if I can reproduce your problem. Might take a while.

marcosvanella commented 6 years ago

Hi Marjan, I installed a Centos 7 VM and deployed the linux bundle on it.

I was able to reproduce your problem. It seems related to the dynamic library glibc present in Centos 7 (glibc2.17), and libraries it calls. The glibc we used to make the bundle is glibc 2.12). I tested the code using Ubuntu linux with glibc2.19 and things work fine. This is the expected behavior, as glibc should be backwards compatible. The problem appears when invoking the Intel Math Kernel Library we link statically, where the sparse cluster linear solver lives (used by glmat).

I haven't been able to even downgrade or upgrade glibc on Centos 7. Maybe you have a better idea on how to go about that?

Next steps:

Try to compile the code from github repository in Centos 7 and link to Intel libs (impi, MKL). See if it runs correctly.
Place a ticket with intel about this, if need be.

marcosvanella commented 6 years ago

Hi Marjan, looks like your best bet at this point is to compile FDS from the repository. I compiled FDS on the Centos 7 VM, using gfortran 7.2, openmpi3.0, and the freely available Intel MKL library. The resulting executable runs correctly.

I will place a ticket with intel support next Monday, to see if they can help out understanding this issue.

To get gfortran 7.2 on Centos 7, I installed gcc from devtoolset-7. In your command line type: $sudo yum install devtoolset-7-gcc*

Then in your ~/.bashrc add the following lines:

export PATH=/opt/rh/devtoolset-7/root/bin:$PATH
export LD_LIBRARY_PATH=/opt/rh/devtoolset-7/root/lib/gcc/x86_64-redhat-linux/7:$LD_LIBRARY_PATH

After sourcing .bashrc you should be able to see what version of gfortran you have at hand. The openmpi and MKL installations follow what is discussed in the following wiki:

https://github.com/firemodels/fds/wiki/Compiling-FDS-with-GNU-Fortran,-OpenMPI-and-Intel-Performance-Libraries-in-Ubuntu-Linux

We can discuss on Monday if you want to proceed this way.

MarjanLukezic commented 6 years ago

Hi Mr. Marco!

Thank you for your time. Thank you for all your help and suggestions. I can not try all your suggestion right now, because I run one simulation. I will try as soon as possible.

Best regards,

Marjan Lukezic

On Sat, Feb 24, 2018 at 1:22 AM, marcosvanella notifications@github.com wrote:

Hi Marjan, looks like your best bet at this point is to compile FDS from the repository. I compiled FDS on the Centos 7 VM, using gfortran 7.2, openmpi3.0, and the freely available Intel MKL library. The resulting executable runs correctly.

I will place a ticket with intel support next Monday, to see if they can help out understanding this issue.

To get gfortran 7.2 on Centos 7, I installed gcc from devtoolset-7. In your command line type: $sudo yum install devtoolset-7-gcc*

Then in your ~/.bashrc add the following lines:

export PATH=/opt/rh/devtoolset-7/root/bin:$PATH export LD_LIBRARY_PATH=/opt/rh/devtoolset-7/root/lib/gcc/x86_64-redhat-linux/7:$LD_LIBRARY_PATH

After sourcing .bashrc you should be able to see what version of gfortran you have at hand. The openmpi and MKL installations follow what is discussed in the following wiki:

https://github.com/firemodels/fds/wiki/Compiling-FDS-with- GNU-Fortran,-OpenMPI-and-Intel-Performance-Libraries-in-Ubuntu-Linux

We can discuss on Monday if you want to proceed this way.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-368177310, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbr33DlAozB9IADfcqPdYGPw9SuX42ks5tX1ZHgaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

marcosvanella commented 6 years ago

Hi Marjan, I sent a support ticket to Intel to see if they can help out with this. I'm waiting for their response. The crash you see is a segfault on the library glibc (malloc.c), when invoked from the Intel MKL/MPI library set. Glenn compiled a debug version of the FDS bundle (thank you Glenn!), to test this on the CenOS 7 virtual machine. Again, compiling the code from repo in your machine will give you an fds executable that works correctly (I've tested this). Let me know if you want to do that.

Here is the information on the failure provided using the gdb debugger:

[marcos@localhost test_glmat]$ gdb fds_db
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/marcos/FDS/bin/fds_db...done.
(gdb) run glmat.fds
Starting program: /home/marcos/FDS/bin/fds_db glmat.fds
Missing separate debuginfo for /home/marcos/FDS/bin/LIB64/libiomp5.so
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
Missing separate debuginfo for /home/marcos/FDS/bin/LIB64/libmpi.so.12
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/1f/66ea852b505c5c8dfda327ff3d890036d18b74.debug
[New Thread 0x2aaaae314780 (LWP 24686)]
[New Thread 0x2aaaae715800 (LWP 24687)]
[New Thread 0x2aaaaeb16880 (LWP 24688)]

 Using GLMAT as pressure solver. List of H unknown numbers per proc:
 MYID=       0, NUNKH_LOCAL=    2100

Program received signal SIGSEGV, Segmentation fault.
0x00002aaaacaee4fc in __GI___libc_free (mem=0x2aaaaaace730) at malloc.c:2949
2949      ar_ptr = arena_for_chunk(p);
(gdb) bt
#0  0x00002aaaacaee4fc in __GI___libc_free (mem=0x2aaaaaace730) at malloc.c:2949
#1  0x00002aaaaaab0f22 in open_path (name=name@entry=0x7fffffff9f30 "libittnotify.so", namelen=namelen@entry=16,
    secure=secure@entry=0, sps=0x2aaaaacce4f8, realname=realname@entry=0x7fffffff6758, fbp=fbp@entry=0x7fffffff6768,
    loader=loader@entry=0x2aaaaacce150, whatcode=whatcode@entry=4,
    found_other_class=found_other_class@entry=0x7fffffff6750) at dl-load.c:2077
#2  0x00002aaaaaab3712 in _dl_map_object (loader=loader@entry=0x2aaaaacce150,
    name=name@entry=0x7fffffff9f30 "libittnotify.so", type=type@entry=2, trace_mode=trace_mode@entry=0,
    mode=mode@entry=-1879047935, nsid=<optimized out>) at dl-load.c:2246
#3  0x00002aaaaaabe964 in dl_open_worker (a=a@entry=0x7fffffff6ce8) at dl-open.c:232
#4  0x00002aaaaaaba2d4 in _dl_catch_error (objname=objname@entry=0x7fffffff6cd8, errstring=errstring@entry=0x7fffffff6ce0,
    mallocedp=mallocedp@entry=0x7fffffff6cd0, operate=operate@entry=0x2aaaaaabe820 <dl_open_worker>,
    args=args@entry=0x7fffffff6ce8) at dl-error.c:177
#5  0x00002aaaaaabe2cb in _dl_open (file=0x7fffffff9f30 "libittnotify.so", mode=-2147483391,
    caller_dlopen=<optimized out>, nsid=-2, argc=2, argv=0x7fffffffd988, env=0x9c28a90) at dl-open.c:650
#6  0x00002aaaab591fbb in dlopen_doit (a=a@entry=0x7fffffff6ef0) at dlopen.c:66
#7  0x00002aaaaaaba2d4 in _dl_catch_error (objname=0x9c14040, errstring=0x9c14048, mallocedp=0x9c14038,
    operate=0x2aaaab591f60 <dlopen_doit>, args=0x7fffffff6ef0) at dl-error.c:177
#8  0x00002aaaab5925bd in _dlerror_run (operate=operate@entry=0x2aaaab591f60 <dlopen_doit>, args=args@entry=0x7fffffff6ef0)
    at dlerror.c:163
#9  0x00002aaaab592051 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#10 0x00000000038bc357 in mkl_serv_load_inspector ()
#11 0x00000000038bc66d in mkl_serv_lock ()
#12 0x00000000039b0d1b in mkl_serv_get_mpi_wrappers ()
#13 0x00000000038cc467 in mkl_pds_lp64_cpardiso_mpi_comm_rank ()
#14 0x000000000387662f in mkl_pds_lp64_cluster_sparse_solver ()
#15 0x000000000355d8a4 in globalmatrix_solver::get_h_matrix_ludcmp () at ../../Source/pres.f90:2394
#16 0x0000000003540a4a in globalmatrix_solver::glmat_solver_setup_h (stage_flag=3) at ../../Source/pres.f90:1841
#17 0x00000000037223bf in fds () at ../../Source/main.f90:246
#18 0x000000000054a29e in main ()
#19 0x00002aaaaca8fc05 in __libc_start_main (main=0x54a270 <main>, argc=2, ubp_av=0x7fffffffd988, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd978) at ../csu/libc-start.c:274
#20 0x000000000054a129 in _start ()
(gdb)

MarjanLukezic commented 6 years ago

Hi Marco!

I am going to install FDS from source code on my Centos 7.4 on Friday. Thank you for you detail explanation and for send support ticket to Intel developers. Thank you for your time. Best regards,

Marjan

On Wed, Feb 28, 2018 at 8:52 PM, marcosvanella notifications@github.com wrote:

Hi Marjan, I sent a support ticket to Intel to see if they can help out with this. I'm waiting for their response. The crash you see is a segfault on the library glibc (malloc.c), when invoked from the Intel MKL/MPI library set. Glenn compiled a debug version of the FDS bundle (thank you Glenn!), to test this on the CenOS 7 virtual machine. Again, compiling the code from repo in your machine will give you an fds executable that works correctly (I've tested this). Let me know if you want to do that.

Here is the information on the failure provided using the gdb debugger:

[marcos@localhost test_glmat]$ gdb fds_db GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /home/marcos/FDS/bin/fds_db...done. (gdb) run glmat.fds Starting program: /home/marcos/FDS/bin/fds_db glmat.fds Missing separate debuginfo for /home/marcos/FDS/bin/LIB64/libiomp5.so [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib64/libthread_db.so.1". Missing separate debuginfo for /home/marcos/FDS/bin/LIB64/libmpi.so.12 Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/1f/66ea852b505c5c8dfda327ff3d890036d18b74.debug [New Thread 0x2aaaae314780 (LWP 24686)] [New Thread 0x2aaaae715800 (LWP 24687)] [New Thread 0x2aaaaeb16880 (LWP 24688)]

Using GLMAT as pressure solver. List of H unknown numbers per proc: MYID= 0, NUNKH_LOCAL= 2100

Program received signal SIGSEGV, Segmentation fault. 0x00002aaaacaee4fc in __GI___libc_free (mem=0x2aaaaaace730) at malloc.c:2949 2949 ar_ptr = arena_for_chunk(p); (gdb) bt

0 0x00002aaaacaee4fc in __GI___libc_free (mem=0x2aaaaaace730) at malloc.c:2949

1 0x00002aaaaaab0f22 in open_path (name=name@entry=0x7fffffff9f30 "libittnotify.so", namelen=namelen@entry=16,
secure=secure@entry=0, sps=0x2aaaaacce4f8, realname=realname@entry=0x7fffffff6758, fbp=fbp@entry=0x7fffffff6768,
loader=loader@entry=0x2aaaaacce150, whatcode=whatcode@entry=4,
found_other_class=found_other_class@entry=0x7fffffff6750) at dl-load.c:2077
2 0x00002aaaaaab3712 in _dl_map_object (loader=loader@entry=0x2aaaaacce150,
name=name@entry=0x7fffffff9f30 "libittnotify.so", type=type@entry=2, trace_mode=trace_mode@entry=0,
mode=mode@entry=-1879047935, nsid=<optimized out>) at dl-load.c:2246
3 0x00002aaaaaabe964 in dl_open_worker (a=a@entry=0x7fffffff6ce8) at dl-open.c:232

4 0x00002aaaaaaba2d4 in _dl_catch_error (objname=objname@entry=0x7fffffff6cd8, errstring=errstring@entry=0x7fffffff6ce0,
mallocedp=mallocedp@entry=0x7fffffff6cd0, operate=operate@entry=0x2aaaaaabe820 <dl_open_worker>,
args=args@entry=0x7fffffff6ce8) at dl-error.c:177
5 0x00002aaaaaabe2cb in _dl_open (file=0x7fffffff9f30 "libittnotify.so", mode=-2147483391,
caller_dlopen=<optimized out>, nsid=-2, argc=2, argv=0x7fffffffd988, env=0x9c28a90) at dl-open.c:650
6 0x00002aaaab591fbb in dlopen_doit (a=a@entry=0x7fffffff6ef0) at dlopen.c:66

7 0x00002aaaaaaba2d4 in _dl_catch_error (objname=0x9c14040, errstring=0x9c14048, mallocedp=0x9c14038,
operate=0x2aaaab591f60 <dlopen_doit>, args=0x7fffffff6ef0) at dl-error.c:177
8 0x00002aaaab5925bd in _dlerror_run (operate=operate@entry=0x2aaaab591f60 , args=args@entry=0x7fffffff6ef0)
at dlerror.c:163
9 0x00002aaaab592051 in __dlopen (file=, mode=) at dlopen.c:87

10 0x00000000038bc357 in mkl_serv_load_inspector ()

11 0x00000000038bc66d in mkl_serv_lock ()

12 0x00000000039b0d1b in mkl_serv_get_mpi_wrappers ()

13 0x00000000038cc467 in mkl_pds_lp64_cpardiso_mpi_comm_rank ()

14 0x000000000387662f in mkl_pds_lp64_cluster_sparse_solver ()

15 0x000000000355d8a4 in globalmatrix_solver::get_h_matrix_ludcmp () at ../../Source/pres.f90:2394

16 0x0000000003540a4a in globalmatrix_solver::glmat_solver_setup_h (stage_flag=3) at ../../Source/pres.f90:1841

17 0x00000000037223bf in fds () at ../../Source/main.f90:246

18 0x000000000054a29e in main ()

19 0x00002aaaaca8fc05 in __libc_start_main (main=0x54a270
, argc=2, ubp_av=0x7fffffffd988, init=,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd978) at ../csu/libc-start.c:274
20 0x000000000054a129 in _start ()

(gdb)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-369361747, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbr5JA4OG2Yan_FgXCDtdWaVUFCuROks5tZa5ugaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

MarjanLukezic commented 6 years ago

Hi Marco!

I'm trying instructions for compiling fds code from source:

https://github.com/firemodels/fds/wiki/Compiling-FDS-with-GNU-Fortran,-OpenMPI-and-Intel-Performance-Libraries-in-Ubuntu-Linux

I can not proceed to the end. I had problem with this instruction set in CentOS 7.4:

Then make and install:

$ make -j 2 $ sudo make install

Your mpifort and mpirun executables will be located in /shared/openmpi_64/bin, and the associated libraries to link FDS to will be located in /shared/openmpi_64/lib

What should I do? May be I can not transfer GitHub source code to local computer correctly. Where should I put GitHub repository?

[mlukezic@localhost Build]$ make mpi_intel_linux_64 fatal: Not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). usage: git diff [--no-index] fatal: Not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: Not a git repository (or any parent up to mount point /home) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). /bin/bash: ../../Utilities/Scripts/intelmpi_compversion.sh: No such file or directory /bin/bash: ../../Utilities/Scripts/openmpi_compversion.sh: No such file or directory /bin/bash: ../../Utilities/Scripts/gnu_compversion.sh: No such file or directory mpifort -c -m64 -O2 -ipo -traceback -no-wrap-margin -fpp -DGITHASH_PP=\"-\" -DGITDATE_PP=\""\"" -DBUILDDATE_PP=\""Mar 07, 2018 03:49:06\"" -DCOMPVER_PP=\"\" ../Source/prec.f90 /bin/bash: mpifort: command not found make: *** [prec.o] Error 127

I also install Intel Math Kernel Library correctly.

Thank you for help and best regards,

Marjan Lukezic

On Wed, Feb 28, 2018 at 9:13 PM, Marjan Lukezic marjan.lukezic@gmail.com wrote:

Hi Marco!

I am going to install FDS from source code on my Centos 7.4 on Friday. Thank you for you detail explanation and for send support ticket to Intel developers. Thank you for your time. Best regards,

Marjan

On Wed, Feb 28, 2018 at 8:52 PM, marcosvanella notifications@github.com wrote:
Hi Marjan, I sent a support ticket to Intel to see if they can help out with this. I'm waiting for their response. The crash you see is a segfault on the library glibc (malloc.c), when invoked from the Intel MKL/MPI library set. Glenn compiled a debug version of the FDS bundle (thank you Glenn!), to test this on the CenOS 7 virtual machine. Again, compiling the code from repo in your machine will give you an fds executable that works correctly (I've tested this). Let me know if you want to do that.

Here is the information on the failure provided using the gdb debugger:

[marcos@localhost test_glmat]$ gdb fds_db GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /home/marcos/FDS/bin/fds_db...done. (gdb) run glmat.fds Starting program: /home/marcos/FDS/bin/fds_db glmat.fds Missing separate debuginfo for /home/marcos/FDS/bin/LIB64/libiomp5.so [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib64/libthread_db.so.1". Missing separate debuginfo for /home/marcos/FDS/bin/LIB64/libmpi.so.12 Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/1f/66ea852b505c5c8dfda327ff3d890036d18b74.debug [New Thread 0x2aaaae314780 (LWP 24686)] [New Thread 0x2aaaae715800 (LWP 24687)] [New Thread 0x2aaaaeb16880 (LWP 24688)]

Using GLMAT as pressure solver. List of H unknown numbers per proc: MYID= 0, NUNKH_LOCAL= 2100

Program received signal SIGSEGV, Segmentation fault. 0x00002aaaacaee4fc in __GI___libc_free (mem=0x2aaaaaace730) at malloc.c:2949 2949 ar_ptr = arena_for_chunk(p); (gdb) bt

0 0x00002aaaacaee4fc in __GI___libc_free (mem=0x2aaaaaace730) at malloc.c:2949

1 0x00002aaaaaab0f22 in open_path (name=name@entry=0x7fffffff9f30 "libittnotify.so", namelen=namelen@entry=16,
secure=secure@entry=0, sps=0x2aaaaacce4f8, realname=realname@entry=0x7fffffff6758, fbp=fbp@entry=0x7fffffff6768,
loader=loader@entry=0x2aaaaacce150, whatcode=whatcode@entry=4,
found_other_class=found_other_class@entry=0x7fffffff6750) at dl-load.c:2077
2 0x00002aaaaaab3712 in _dl_map_object (loader=loader@entry=0x2aaaaacce150,
name=name@entry=0x7fffffff9f30 "libittnotify.so", type=type@entry=2, trace_mode=trace_mode@entry=0,
mode=mode@entry=-1879047935, nsid=<optimized out>) at dl-load.c:2246
3 0x00002aaaaaabe964 in dl_open_worker (a=a@entry=0x7fffffff6ce8) at dl-open.c:232

4 0x00002aaaaaaba2d4 in _dl_catch_error (objname=objname@entry=0x7fffffff6cd8, errstring=errstring@entry=0x7fffffff6ce0,
mallocedp=mallocedp@entry=0x7fffffff6cd0, operate=operate@entry=0x2aaaaaabe820 <dl_open_worker>,
args=args@entry=0x7fffffff6ce8) at dl-error.c:177
5 0x00002aaaaaabe2cb in _dl_open (file=0x7fffffff9f30 "libittnotify.so", mode=-2147483391,
caller_dlopen=<optimized out>, nsid=-2, argc=2, argv=0x7fffffffd988, env=0x9c28a90) at dl-open.c:650
6 0x00002aaaab591fbb in dlopen_doit (a=a@entry=0x7fffffff6ef0) at dlopen.c:66

7 0x00002aaaaaaba2d4 in _dl_catch_error (objname=0x9c14040, errstring=0x9c14048, mallocedp=0x9c14038,
operate=0x2aaaab591f60 <dlopen_doit>, args=0x7fffffff6ef0) at dl-error.c:177
8 0x00002aaaab5925bd in _dlerror_run (operate=operate@entry=0x2aaaab591f60 , args=args@entry=0x7fffffff6ef0)
at dlerror.c:163
9 0x00002aaaab592051 in __dlopen (file=, mode=) at dlopen.c:87

10 0x00000000038bc357 in mkl_serv_load_inspector ()

11 0x00000000038bc66d in mkl_serv_lock ()

12 0x00000000039b0d1b in mkl_serv_get_mpi_wrappers ()

13 0x00000000038cc467 in mkl_pds_lp64_cpardiso_mpi_comm_rank ()

14 0x000000000387662f in mkl_pds_lp64_cluster_sparse_solver ()

15 0x000000000355d8a4 in globalmatrix_solver::get_h_matrix_ludcmp () at ../../Source/pres.f90:2394

16 0x0000000003540a4a in globalmatrix_solver::glmat_solver_setup_h (stage_flag=3) at ../../Source/pres.f90:1841

17 0x00000000037223bf in fds () at ../../Source/main.f90:246

18 0x000000000054a29e in main ()

19 0x00002aaaaca8fc05 in __libc_start_main (main=0x54a270
, argc=2, ubp_av=0x7fffffffd988, init=,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd978) at ../csu/libc-start.c:274
20 0x000000000054a129 in _start ()

(gdb)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-369361747, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbr5JA4OG2Yan_FgXCDtdWaVUFCuROks5tZa5ugaJpZM4SQfOC .
--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

marcosvanella commented 6 years ago

Hi Marjan, lets go one step at a time with this:

Could you install the devtoolset-7-gcc* libraries and gfortran 7.2 on your Centos? Did you add to your ~/.bashrc:


export PATH=/opt/rh/devtoolset-7/root/bin:$PATH
export LD_LIBRARY_PATH=/opt/rh/devtoolset-7/root/lib/gcc/x86_64-redhat-linux/7:$LD_LIBRARY_PATH


In your ~/.bashrc comment the lines related to the installed bundle FDS6. You won't be using it (well just smokeview, for which you can make an alias).

2. Could you compile openmpi with this gfortran 7.2? If so did you add the paths to the install /bin and /lib directories to your ~/.bashrc?
Something like:

export MPIDIST=/shared/openmpi_64 export PATH=$MPIDIST/bin:$PATH export LD_LIBRARY_PATH=$MPIDIST/lib:$LD_LIBRARY_PATH

And source this .bashrc. If your environment is setup correctly, typing:
`$mpifort -v`

will tell you the gfortran 7.2 info, which you used to compile openmpi.

3. It seems you were able to clone firemodels/fds from github correct? Go to fds/Build/mpi_gnu_linux_64/

and type:

`$./make_fds.sh`

See if that works. The targets mpi_intel_linux_64* are defined for ifort (the intel fortran compiler, which I assume you don't have). Make sure that the flag -WITH_MKL is coming up in the compile lines for souce files.

The executable produced will be in fds/Build/mpi_gnu_linux_64/fds_mpi_gnu_linux_64, so provided compilation works fine, if in that directory you type:
`$./fds_mpi_gnu_linux_64`

You should get back the fds header (default behavior).

Then you can make in your ~/.bashrc an alias, something like:

alias fds=PATH_TO_FIREMODELS/fds/Build/mpi_gnu_linux_64/fds_mpi_gnu_linux_64 alias smv=PATH_TO_FDS6/bin/smokeview


Again source your ~/.bashrc .
As time goes by, to get the latest changes being done on FDS you can update your firemodels/fds repo and recompile the code.

I can't access my Centos VM here, but we can review tomorrow. Also, I'm in discussion with Intel support on this issue. I'm waiting for their response now, but looks like I'll need to make a more simplified case besides pointing them to the bundle. I'll let you know what comes up with this.

Marcos

MarjanLukezic commented 6 years ago

Hi Marco, I had problem with compiling openmpi 3.0.0.

root@localhost openmpi-3.0.0]# ./configure FC=gfortran-7 CC=gcc-7 --prefix=/shared/openmpi_64 --enable-mpirun-prefix-by-default --enable-mpi-fortran --enable-static --disable-shared checking for perl... perl

============================================================================ == Configuring Open MPI

*** Startup tests checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking for gcc... gcc-7 checking whether the C compiler works... no configure: error: in /home/mlukezic/Downloads/openmpi-3.0.0': configure: error: C compiler cannot create executables Seeconfig.log' for more details

[root@localhost openmpi-3.0.0]# gcc --version gcc (GCC) 7.2.1 20170829 (Red Hat 7.2.1-1) Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I really need your help.

Best regards,

Marjan Lukezic

On Sat, Feb 24, 2018 at 1:22 AM, marcosvanella notifications@github.com wrote:

Hi Marjan, looks like your best bet at this point is to compile FDS from the repository. I compiled FDS on the Centos 7 VM, using gfortran 7.2, openmpi3.0, and the freely available Intel MKL library. The resulting executable runs correctly.

I will place a ticket with intel support next Monday, to see if they can help out understanding this issue.

To get gfortran 7.2 on Centos 7, I installed gcc from devtoolset-7. In your command line type: $sudo yum install devtoolset-7-gcc*

Then in your ~/.bashrc add the following lines:

export PATH=/opt/rh/devtoolset-7/root/bin:$PATH export LD_LIBRARY_PATH=/opt/rh/devtoolset-7/root/lib/gcc/x86_64-redhat-linux/7:$LD_LIBRARY_PATH

After sourcing .bashrc you should be able to see what version of gfortran you have at hand. The openmpi and MKL installations follow what is discussed in the following wiki:

https://github.com/firemodels/fds/wiki/Compiling-FDS-with- GNU-Fortran,-OpenMPI-and-Intel-Performance-Libraries-in-Ubuntu-Linux

We can discuss on Monday if you want to proceed this way.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-368177310, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbr33DlAozB9IADfcqPdYGPw9SuX42ks5tX1ZHgaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

marcosvanella commented 6 years ago

Marjan, try this:

$ ./configure FC=gfortran CXX=g++ CC=gcc --prefix=/shared/openmpi_64 --enable-mpirun-prefix-by-default --enable-mpi-fortran --enable-static --disable-shared

The GNU C and Fortran compilers in your system are called gcc and gfortran (different than Ubuntu).

sbenkorichi commented 6 years ago

This is how I used to compile it under Centos 7 ./configure FC=gfortran CC=gcc --enable-mpi-fortran --enable-static --disable-shared --prefix=/shared/openmpi_64

One thing you should keep in mind, you are installing in a root / So, you should first make sure that the /shared has permission to write in it. Just do $ sudo chmod -R 0777 /shared/ before you start the configuration. Then, also make sure that openmpi_64 exists in it.

If the configuration works, then do:

$ make
$ sudo make install

Try to follow this order.

MarjanLukezic commented 6 years ago

Dear Marco and Salah,

Thank your for your detailed instruction and help. Now I got this massage:

[mlukezic@localhost ~]$ which mpirun /shared/openmpi_64/bin/mpirun

Tomorrow I am going to make next steps.

Best regards, Marjan

On Tue, Mar 13, 2018 at 10:00 PM, Salah Benkorichi <notifications@github.com

wrote:

This is how I used to compile it under Centos 7 ./configure FC=gfortran CC=gcc --enable-mpi-fortran --enable-static --disable-shared --prefix=/shared/openmpi_64

One thing you should keep in mind, you are installing in a root / So, you should first make sure that the /shared has permission to write in it. Just do $ sudo chmod -R 0777 /shared/ before you start the configuration. Then, also make sure that openmpi_64 exists in it.

If the configuration works, then do:

$ make $ sudo make install

Try to follow this order.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-372816643, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbr_43ALzgBWz0gFS0iD6GEUtmyuguks5teDNxgaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

MarjanLukezic commented 6 years ago

Dear Marco and Salah,

I managed to install fds code from source completely. Thank you very much for your kindness and help.

Best regards,

Marjan Lukezic

On Wed, Mar 14, 2018 at 8:45 PM, Marjan Lukezic marjan.lukezic@gmail.com wrote:

Dear Marco and Salah,

Thank your for your detailed instruction and help. Now I got this massage:

[mlukezic@localhost ~]$ which mpirun /shared/openmpi_64/bin/mpirun

Tomorrow I am going to make next steps.

Best regards, Marjan

On Tue, Mar 13, 2018 at 10:00 PM, Salah Benkorichi < notifications@github.com> wrote:

This is how I used to compile it under Centos 7 ./configure FC=gfortran CC=gcc --enable-mpi-fortran --enable-static --disable-shared --prefix=/shared/openmpi_64

One thing you should keep in mind, you are installing in a root / So, you should first make sure that the /shared has permission to write in it. Just do $ sudo chmod -R 0777 /shared/ before you start the configuration. Then, also make sure that openmpi_64 exists in it.

If the configuration works, then do:

$ make $ sudo make install

Try to follow this order.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-372816643, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbr_43ALzgBWz0gFS0iD6GEUtmyuguks5teDNxgaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

sbenkorichi commented 6 years ago

Ok thanks for reporting back. I will let Marcos to close this, or he might want to leave it open until he gets some feedback from INTEL.

MarjanLukezic commented 6 years ago

I hope the INTEL will fix his library.

Best regards,

Marjan Lukezic

On Fri, Mar 16, 2018 at 4:23 PM, Salah Benkorichi notifications@github.com wrote:

Ok thanks for reporting back. I will let Marcos to close it, or he might wants to leave it open until he gets some feedback from INTEL.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-373747442, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbr09nahP9CK6uEvZvKSl385Mp8rnCks5te9jegaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

MarjanLukezic commented 6 years ago

When I run glamt_v3.fds I still got an error: glmat_v3.fds input code:

&HEAD CHID='glmat', TITLE='Compart_Test'/ &TIME T_END=360.0/

&MESH IJK=21,10,10, XB=0.0,2.1,0.0,1.0,0.0,1.0 /

&REAC ID='React_Methane', FUEL='METHANE', CO_YIELD=0.1, SOOT_YIELD=0.01/ &DEVC ID='P_L', QUANTITY='PRESSURE', XYZ=1.05,0.5,0.5, SETPOINT=1.0E5/ &PRES SOLVER='GLMAT'/

&TAIL /

end of input code

In terminal: [mlukezic@localhost glmat_v3]$ ls glmat_01.s3d glmat_02.s3d glmat.binfo glmat_git.txt glmat.smv glmat_01.s3d.sz glmat_02.s3d.sz glmat.end glmat.sinfo glmat_v3.fds [mlukezic@localhost glmat_v3]$ fds_mpi_gnu_linux_64 glmat_v3.fds

Using GLMAT as pressure solver. List of H unknown numbers per proc: MYID= 0, NUNKH_LOCAL= 2100 Error: MKL Library compile flag was not defined for GLMAT as pressure solver.

DEVICE Activation Times 1 P_L No Activation STOP: FDS was improperly set-up (CHID: glmat) STOP: FDS was improperly set-up (CHID: glmat)

marcosvanella commented 6 years ago

Marjan, your compilation hasn't linked the code to the Intel MKL library.

Download and install Intel performance libs as explained in the wiki.
Add paths to the lib in your system, on your ~/.bashrc. In my case I put:

# MKL Environment Variables:
export INTEL_COMPILERS_AND_LIBS=/home/marcos/intel/compilers_and_libraries_2017/linux
source $INTEL_COMPILERS_AND_LIBS/mkl/bin/mklvars.sh intel64

If you downloaded intel 2018 -> use YOURPATH/intel/compilers_and_libraries_2018/linux in the definitions above. Make sure you see the -DWITH_MKL flag in compilation lines.

Intel support hasn't helped much so far. They are releasing an update for intel 2018 this week, which I was told contains several fixes. I'll see if the problem persists.

MarjanLukezic commented 6 years ago

Hi Marco,

i installed the latest Intel MKL library without any problems, but I did not know how to modify my .bushrc I'm going to try as you suggest. I complied fds-master with gfortran and now with Intel fortran, is it right or I made a mistake.

Best regards,

Marjan L.

On Mon, Mar 19, 2018 at 3:17 PM, marcosvanella notifications@github.com wrote:

Marjan, your compilation hasn't linked the code to the Intel MKL library.

Download and install Intel performance libs as explained in the wiki.

Add paths to the lib in your system, on your ~/.bashrc. In my case I put:

MKL Environment Variables:

export INTEL_COMPILERS_AND_LIBS=/home/marcos/intel/compilers_and_libraries_2017/linux

source $INTEL_COMPILERS_AND_LIBS/mkl/bin/mklvars.sh intel64

If you downloaded intel 2018 -> use YOURPATH/intel/compilers_and_libraries_2018/linux in the definitions above. Make sure you see the -DWITH_MKL flag in compilation lines.

Intel support hasn't helped much so far. They are releasing an update for intel 2018 this week, which I was told contains several fixes. I'll see if the problem persists.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-374227130, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbrwCaQsPdkaR2NDsEXLwWlM6aaOieks5tf73fgaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

sbenkorichi commented 6 years ago

your startup file (.bashrc ) is to be found in your home directory i.e. $ /home/userid/.bashrc By default it's hidden, so to see it, either open it through a terminal command, or just hit CTRL+H which will show hidden files. Then, at the bottom of the file, paste those lines in it.

marcosvanella commented 6 years ago

Marjan, note I took out the # from the lines in the ~/.bashrc on the previous post. Also compilers_and_libraries_XXXX/linux is where the mkl library is installed, you will not have an intel fortran compiler. You need to invoke the script $INTEL_COMPILERS_AND_LIBS/mkl/bin/mklvars.sh, if not the FDS compilation will not see the MKL library, check notes in the Wiki.

MarjanLukezic commented 6 years ago

I know how to change my bushrc file, but I did not know about mkl variables & commands . Right now I had .bashrc file these commands / lines:

User specific aliases and functions

============= FDS environment =======

export PATH=/opt/rh/devtoolset-7/root/bin:/home/mlukezic/fds-master/Build/mpi_gnu_linux_64/:$PATH export LD_LIBRARY_PATH=/opt/rh/devtoolset-7/root/lib/gcc/x86_64-redhat-linux/7:$LD_LIBRARY_PATH

export MPIDIST=/shared/openmpi_64 export PATH=$MPIDIST/bin:$PATH export LD_LIBRARY_PATH=$MPIDIST/lib:$LD_LIBRARY_PATH export OMP_NUM_THREADS=8

fds run from sub directory fds_mpi_gnu_linux_64

alias fds=/home/mlukezic/fds-master/Build/mpi_gnu_linux_64/fds_mpi_gnu_linux_64

alias smv=/home/mlukezic/fds-master/Build/mpi_gnu_linux_64/bin/smokeview

To remove the stack size limit on a non-Windows system (i.e. Linux, Unix or OSX)

ulimit -s unlimited

I_MPI_SHM_LMT=shm

============= FDS environment =======

as Marco wrote above I must add commands:

MKL Environment Variables:

export INTEL_COMPILERS_AND_LIBS=/home/mlukezic/intel/compilers_and_libraries_2018/linux source $INTEL_COMPILERS_AND_LIBS/mkl/bin/mklvars.sh intel64

I'm runing one fds case right now and it is better do this task later.

Best regards,

Marjan Lukezic

saschagottfried commented 6 years ago

TLDR;

The offical bundle is supposed to work on a CentOS 7.4 machine.

I start with a recommendation for Marjan's machine. Open a shell. Adjust the dynamic linker environment to load shared libraries from LIB64 folder of your FDS installation.

export LD_LIBRARY_PATH=/home/mlukezic/FDS/FDS6/bin/LIB64

Afterwards run a case using GLMAT solver.

$ /home/mlukezic/FDS/FDS6/bin/fds case_glmat.fds

If that is not working pay close attention to the output of ldd as shown in the analysis section below. Please add similar outputs to your next post. Good luck.

Analysis

I started from scratch spinning up a CentOS Linux 7/x86_64 machine from public catalog for Vagrant boxes. I just extracted the official bundle FDS Linux bundle into a folder and kept only a small number of files needed for this demonstration. I did not run bash installer shipped with the bundle.

My environment

[vagrant@localhost ~]$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)
[vagrant@localhost ~]$ uname -a
Linux localhost.localdomain 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

My FDS installation

For repeatability please adjust any shown path to LIB64 folder to match your local installation.


[vagrant@localhost ~]$ tree /vagrant/Dist/fds_6.6.0_linux64/
/vagrant/Dist/fds_6.6.0_linux64/
├── bin
│   ├── fds
│   ├── mpiexec
│   └── pmi_proxy
└── LIB64
├── libiomp5.so
├── libmpifort.so.12
└── libmpi.so.12

2 directories, 6 files


I started running `duct_flow_glmat.fds`

[vagrant@localhost ~]$ /vagrant/Dist/fds_6.6.0_linux64/bin/fds /vagrant/Verification/Pressure_Solver/duct_flow_glmat.fds

Mesh 1 is assigned to MPI Process 0 Mesh 2 is assigned to MPI Process 0 Mesh 3 is assigned to MPI Process 0 Mesh 4 is assigned to MPI Process 0 Mesh 5 is assigned to MPI Process 0 Mesh 6 is assigned to MPI Process 0 Mesh 7 is assigned to MPI Process 0 Mesh 8 is assigned to MPI Process 0 OpenMP thread 0 of 3 assigned to MPI process 0 of 0 OpenMP thread 3 of 3 assigned to MPI process 0 of 0 OpenMP thread 1 of 3 assigned to MPI process 0 of 0 OpenMP thread 2 of 3 assigned to MPI process 0 of 0 Completed Initialization Step 1 Completed Initialization Step 2 Completed Initialization Step 3 Completed Initialization Step 4

Using GLMAT as pressure solver. List of H unknown numbers per proc: MYID= 0, NUNKH_LOCAL= 32743

Fire Dynamics Simulator

Current Date : March 23, 2018 11:49:08 Version : FDS 6.6.0 Revision : FDS6.6.0-131-g88ae75a-HEAD Revision Date : Wed Nov 1 16:03:29 2017 -0400 Compiler : Intel ifort 17.0.4 Compilation Date : Nov 01, 2017 21:03:34

MPI Enabled; Number of MPI Processes: 1 OpenMP Enabled; Number of OpenMP Threads: 4

MPI version: 3.1 MPI library version: Intel(R) MPI Library 2017 Update 3 for Linux* OS

Job TITLE : Flow through a duct, test of GLMAT_SOLVER Job ID string : duct_flow_glmat

Time Step: 1, Simulation Time: 0.12 s Time Step: 2, Simulation Time: 0.25 s Time Step: 3, Simulation Time: 0.38 s Time Step: 4, Simulation Time: 0.50 s Time Step: 5, Simulation Time: 0.62 s Time Step: 6, Simulation Time: 0.75 s Time Step: 7, Simulation Time: 0.88 s Time Step: 8, Simulation Time: 1.00 s Time Step: 9, Simulation Time: 1.12 s Time Step: 10, Simulation Time: 1.25 s Time Step: 20, Simulation Time: 2.47 s Time Step: 30, Simulation Time: 3.58 s Time Step: 40, Simulation Time: 4.58 s Time Step: 50, Simulation Time: 5.57 s Time Step: 60, Simulation Time: 6.56 s Time Step: 70, Simulation Time: 7.47 s Time Step: 80, Simulation Time: 8.36 s Time Step: 90, Simulation Time: 9.25 s Time Step: 100, Simulation Time: 10.14 s Time Step: 200, Simulation Time: 19.02 s Time Step: 300, Simulation Time: 27.90 s Time Step: 400, Simulation Time: 36.78 s Time Step: 500, Simulation Time: 45.67 s Time Step: 600, Simulation Time: 53.74 s Time Step: 679, Simulation Time: 60.00 s MPI process 0 has completed


It just works. This is different to the experiences of Marjan and Marcos. So what is different in my current runtime environment? Let's print the shared object dependencies of `fds` executable.

[vagrant@localhost ~]$ ldd /vagrant/Dist/fds_6.6.0_linux64/bin/fds linux-vdso.so.1 => (0x00007ffd47070000) libiomp5.so => not found libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f22af849000) libm.so.6 => /lib64/libm.so.6 (0x00007f22af547000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f22af343000) libmpifort.so.12 => not found libmpi.so.12 => not found librt.so.1 => /lib64/librt.so.1 (0x00007f22af13a000) libc.so.6 => /lib64/libc.so.6 (0x00007f22aed76000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f22aeb60000) /lib64/ld-linux-x86-64.so.2 (0x000055f07d1f6000)


By default most shared libraries are loaded from `/lib64` folder. Since I did not used the bash installer I need to set `LD_LIBRARY_PATH` to the `LIB64` directory of the bundle installation. 

After adjusting the dynamic linker environment let's print shared object dependencies again:

[vagrant@localhost ~]$ export LD_LIBRARY_PATH=/vagrant/Dist/fds_6.6.0_linux64/LIB64 [vagrant@localhost ~]$ ldd /vagrant/Dist/fds_6.6.0_linux64/bin/fds linux-vdso.so.1 => (0x00007fff733f8000) libiomp5.so => /vagrant/Dist/fds_6.6.0_linux64/LIB64/libiomp5.so (0x00007f559edcf000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f559ebad000) libm.so.6 => /lib64/libm.so.6 (0x00007f559e8ab000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f559e6a7000) libmpifort.so.12 => /vagrant/Dist/fds_6.6.0_linux64/LIB64/libmpifort.so.12 (0x00007f559e2fd000) libmpi.so.12 => /vagrant/Dist/fds_6.6.0_linux64/LIB64/libmpi.so.12 (0x00007f559d5d5000) librt.so.1 => /lib64/librt.so.1 (0x00007f559d3cd000) libc.so.6 => /lib64/libc.so.6 (0x00007f559d009000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f559cdf3000) /lib64/ld-linux-x86-64.so.2 (0x000055bb94655000)


All shared object dependencies can be resolved now. Now we display the current value of  `LD_LIBRARY_PATH`:

[vagrant@localhost ~]$ echo $LD_LIBRARY_PATH /vagrant/Dist/fds_6.6.0_linux64/LIB64


BUT according to Marjan's first post the environment variable `LD_LIBRARY_PATH` on his machine is different. This is a copied line from Marjan's environment.

`export LD_LIBRARY_PATH=/usr/lib64:/home/mlukezic/FDS/FDS6/bin/LIB64:$LD_LIBRARY_PATH
`

What will happen if I add `/usr/lib64` to `LD_LIBRARY_PATH` as well to create a dynamic linker environment similar to Marjan's machine ?

[vagrant@localhost ~]$ export LD_LIBRARY_PATH=/usr/lib64/:/vagrant/Dist/fds_6.6.0_linux64/LIB64

To see how this change affect dynamic linker environment, let's print the shared object dependencies again. Now most of shared libaries will be loaded from `/usr/lib64` folder.

[vagrant@localhost ~]$ ldd /home/vagrant/fds/bin/fds linux-vdso.so.1 => (0x00007ffed8f5a000) libiomp5.so => /vagrant/Dist/fds_6.6.0_linux64/LIB64/libiomp5.so (0x00007fb5e2ad8000) libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007fb5e28bb000) libm.so.6 => /usr/lib64/libm.so.6 (0x00007fb5e25b9000) libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007fb5e23b5000) libmpifort.so.12 => /vagrant/Dist/fds_6.6.0_linux64/LIB64/libmpifort.so.12 (0x00007fb5e200b000) libmpi.so.12 => /vagrant/Dist/fds_6.6.0_linux64/LIB64/libmpi.so.12 (0x00007fb5e12e3000) librt.so.1 => /usr/lib64/librt.so.1 (0x00007fb5e10db000) libc.so.6 => /usr/lib64/libc.so.6 (0x00007fb5e0d17000) libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fb5e0b01000) /lib64/ld-linux-x86-64.so.2 (0x000056156143b000)


Running FDS with modified dynamic linker environment

[vagrant@localhost ~]$ /vagrant/Dist/fds_6.6.0_linux64/bin/fds duct_flow_glmat.fds Mesh 1 is assigned to MPI Process 0 Mesh 2 is assigned to MPI Process 0 Mesh 3 is assigned to MPI Process 0 Mesh 4 is assigned to MPI Process 0 Mesh 5 is assigned to MPI Process 0 Mesh 6 is assigned to MPI Process 0 Mesh 7 is assigned to MPI Process 0 Mesh 8 is assigned to MPI Process 0 OpenMP thread 0 of 3 assigned to MPI process 0 of 0 OpenMP thread 2 of 3 assigned to MPI process 0 of 0 OpenMP thread 1 of 3 assigned to MPI process 0 of 0 OpenMP thread 3 of 3 assigned to MPI process 0 of 0 Completed Initialization Step 1 Completed Initialization Step 2 Completed Initialization Step 3 Completed Initialization Step 4

Using GLMAT as pressure solver. List of H unknown numbers per proc: MYID= 0, NUNKH_LOCAL= 32743 Bus error


Ok, this is exactly the error that Marjan reported in FDS Google Discussion Forum. 

Let's rewind a step. We restore the previous value of `LD_LIBRARY_PATH`  - the `LIB64` folder of FDS bundle installation.

[vagrant@localhost ~]$ export LD_LIBRARY_PATH=/vagrant/Dist/fds_6.6.0_linux64/LIB64 [vagrant@localhost ~]$ /vagrant/Dist/fds_6.6.0_linux64/bin/fds /vagrant/Verification/Pressure_Solver/duct_flow_glmat.fds

Mesh 1 is assigned to MPI Process 0 Mesh 2 is assigned to MPI Process 0 Mesh 3 is assigned to MPI Process 0 Mesh 4 is assigned to MPI Process 0 Mesh 5 is assigned to MPI Process 0 Mesh 6 is assigned to MPI Process 0 Mesh 7 is assigned to MPI Process 0 Mesh 8 is assigned to MPI Process 0 OpenMP thread 0 of 3 assigned to MPI process 0 of 0 OpenMP thread 2 of 3 assigned to MPI process 0 of 0 OpenMP thread 1 of 3 assigned to MPI process 0 of 0 OpenMP thread 3 of 3 assigned to MPI process 0 of 0 Completed Initialization Step 1 Completed Initialization Step 2 Completed Initialization Step 3 Completed Initialization Step 4

Using GLMAT as pressure solver. List of H unknown numbers per proc: MYID= 0, NUNKH_LOCAL= 32743

Fire Dynamics Simulator ...



Works again. 

# Conclusion

My conclusion is that the dynamic linker environment is not configured properly for this specific machine or Linux distribution. If Marcos could have reproduced the error in fresh VM after running the bundle installer, I assume that bash installer is setting up `LD_LIBRARY_PATH` in a way that is not working for the current bundle and CentOS 7.4.

marcosvanella commented 6 years ago

Hi Sascha, interesting..

Correct, I previously reproduced the error on a Centos 7 VM, just by following the instructions from the bundle installer on FDS 6.6.0. I'll test your change on Monday.

So, a question that comes to mind is, how are the libs in /usr/lib64 and /lib64 related? Aren't they supposed to be the same libs in a fresh OS instance? Nice work.

Marjan, try Saschas change on LD_LIBRARY_PATH whenever you can.

sbenkorichi commented 6 years ago

Uhm, I think Saschas got a point, I had run into a difference of libraries locations from different flavors before, and the way in Centos is different than Ubuntu, not sure also how Fedora is handled. /lib64 is more for a root (Admin) whereas for /usr/lib64 is for that specific user. Also, when compiling some codes, you may ended up finding them into /usr/local/lib/ this is for instance for MATLAB usually I see it.

MarjanLukezic commented 6 years ago

Hi Sascha,

I compiled fds from source:

[mlukezic@localhost ~]$ fds

Fire Dynamics Simulator

Current Date : April 9, 2018 12:40:33 Revision : - Revision Date : Compiler : Gnu gfortran 7.2.1 Compilation Date : Mar 16, 2018 14:30:00

MPI Enabled; Number of MPI Processes: 1 OpenMP Enabled; Number of OpenMP Threads: 8

MPI version: 3.1 MPI library version: Open MPI v3.0.0, package: Open MPI root@localhost.localdomain Distribution, ident: 3.0.0, repo rev: v3.0.0, Sep 12, 2017

Consult FDS Users Guide Chapter, Running FDS, for further instructions.

Hit Enter to Escape...

One of my simulation took a long period to finished. It finished yesterday, I read your post on issue tracker. But I can't implement your idea with dynamic library as you suggest.

Here is my .bushrc file:

BOF

.bashrc

Source global definitions

if [ -f /etc/bashrc ]; then . /etc/bashrc fi

Uncomment the following line if you don't like systemctl's auto-paging

feature:

export SYSTEMD_PAGER=

User specific aliases and functions

============= FDS environment =======

export PATH=/opt/rh/devtoolset-7/root/bin:/home/mlukezic/fds- master/Build/mpi_gnu_linux_64/:$PATH export LD_LIBRARYPATH=/opt/rh/devtoolset-7/root/lib/gcc/x86 64-redhat-linux/7:$LD_LIBRARY_PATH

export MPIDIST=/shared/openmpi_64 export PATH=$MPIDIST/bin:$PATH export LD_LIBRARY_PATH=$MPIDIST/lib:$LD_LIBRARY_PATH export OMP_NUM_THREADS=8

how to fix glmat solver error by saschagottfried - path is not right for

my compile fds from source

export LD_LIBRARY_PATH=/home/mlukezic/FDS/FDS6/bin/LIB64

MKL Environment Variables:

export INTEL_COMPILERS_ANDLIBS=/home/mlukezic/intel/compilers

and_libraries_2018/linux

source $INTEL_COMPILERS_AND_LIBS/mkl/bin/mklvars.sh intel64

fds run from sub directory fds_mpi_gnu_linux_64

alias fds=/home/mlukezic/fds-master/Build/mpi_gnu_linux64/fds mpi_gnu_linux_64

alias smv=/home/mlukezic/fds-master/Build/mpi_gnu_linux_64/bin/smokeview

To remove the stack size limit on a non-Windows system (i.e. Linux, Unix

or OSX) ulimit -s unlimited

I_MPI_SHM_LMT=shm

============= FDS environment =======

EOF

When I tried command:

[root@localhost mlukezic]# find / -type d -name "LIB64" find: ‘/run/user/1000/gvfs’: Permission denied

I did not write this on issue tracker, because I think path is not right for my environment.

Best regards,

Marjan Lukezic

On Fri, Mar 23, 2018 at 5:02 PM, Salah Benkorichi notifications@github.com wrote:

Uhm, I think Sacha got a point, I had run into a difference of libraries locations from different flavors before, and the way in Centos is different than Ubuntu, not sure also how Fedora is handled. /lib64 is more for a root (Admin) whereas for /usr/lib64 is for that specific user. Also, when compiling some codes, you may ended up finding them into /usr/local/lib/ this is for instance for MATLAB usually I see it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-375714461, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbr_54JlVUGwEJ3_sd_CLpNJSQ9ihWks5thRyEgaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

saschagottfried commented 6 years ago

Hi Marjan,

I recommend to revert your machine back towards a typical end-user machine that uses the official precompiled binaries installed by the bundle installer.

Given your first post your previous FDS environment was configured like this. Please revert the current bash/shell configuration related to FDS back to where we started.

#============= FDS environment =======
export PATH=/home/mlukezic/FDS/FDS6/bin:$PATH
export LD_LIBRARY_PATH=/usr/lib64:/home/mlukezic/FDS/FDS6/bin/LIB64:$LD_LIBRARY$
export OMP_NUM_THREADS=8

ulimit -s unlimited

Salah Benkorichi adviced this code for bash
I_MPI_SHM_LMT=shm
#============= FDS environment =======

The root cause for segmentation fault when using GLMAT solver is due to setup of dynamic linker environment variable LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=/usr/lib64:/home/mlukezic/FDS/FDS6/bin/LIB64:$LD_LIBRARY$

Please remove path /usr/lib64 from the list of directories where libraries should be searched for.

export LD_LIBRARY_PATH=/home/mlukezic/FDS/FDS6/bin/LIB64

That's it. A working FDS environment on your machine (CentOS 7.4) can be verified like this

$ ldd /home/mlukezic/FDS/FDS6/bin/fds

Expected output

all shared libraries are loaded from /lib64/
except Intel OpenMP/MPI related libraries loaded from /home/mlukezic/FDS/FDS6/bin/LIB64

linux-vdso.so.1 =>  (0x00007fff733f8000)
libiomp5.so => /home/mlukezic/FDS/FDS6/bin/LIB64/libiomp5.so (0x00007f559edcf000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f559ebad000)
libm.so.6 => /lib64/libm.so.6 (0x00007f559e8ab000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f559e6a7000)
libmpifort.so.12 => /home/mlukezic/FDS/FDS6/bin/LIB64/libmpifort.so.12 (0x00007f559e2fd000)
libmpi.so.12 => /home/mlukezic/FDS/FDS6/bin/LIB64/libmpi.so.12 (0x00007f559d5d5000)
librt.so.1 => /lib64/librt.so.1 (0x00007f559d3cd000)
libc.so.6 => /lib64/libc.so.6 (0x00007f559d009000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f559cdf3000)
/lib64/ld-linux-x86-64.so.2 (0x000055bb94655000)

If this does not work, try these commands in an interactive shell session and post every command and its output here. Otherwise you make it very hard for us to help you. Always use absolute paths for binaries like fds and mpiexec until this issue is fixed.

$ export LD_LIBRARY_PATH=/home/mlukezic/FDS/FDS6/bin/LIB64
$ echo $LD_LIBRARY_PATH
$ ldd /home/mlukezic/FDS/FDS6/bin/fds
$ /home/mlukezic/FDS/FDS6/bin/fds your_case_using_glmat.fds

Please let me know if that works for you.

MarjanLukezic commented 6 years ago

Hi Sascha Gottfried.

Part of my .bashrc file:

User specific aliases and functions

============= FDS environment =======

export PATH=/home/mlukezic/FDS/FDS6/bin:$PATH

export

LD_LIBRARY_PATH=/usr/lib64:/home/mlukezic/FDS/FDS6/bin/LIB64:$LD_LIBRAR$ export LD_LIBRARY_PATH=/home/mlukezic/FDS/FDS6/bin/LIB64 export OMP_NUM_THREADS=8

ulimit -s unlimited

I_MPI_SHM_LMT=shm

============= FDS environment =======

when I run a file that Sascha suggested:

https://raw.githubusercontent.com/firemodels/fds/master/Verification/Pressure_Solver/duct_flow_glmat.fds

it works fine without any problems on my CentOS 7.

Thank you very much for all your help: Sascha Gottfried, Marco Svanella and Salah Benkorichi.

Best regards,

Marjan Lukezic

On Wed, Apr 11, 2018 at 6:08 PM, Sascha Gottfried notifications@github.com wrote:

Hi Marjan,

I recommend to revert your machine back towards a typical end-user machine that uses the official bundle installed by the bundle installer.

Given your first post your previous FDS environment was configured like

============= FDS environment =======export PATH=/home/mlukezic/FDS/FDS6/bin:$PATHexport LD_LIBRARY_PATH=/usr/lib64:/home/mlukezic/FDS/FDS6/bin/LIB64:$LD_LIBRARY$export OMP_NUM_THREADS=8

ulimit -s unlimited

Salah Benkorichi adviced this code for bash I_MPI_SHM_LMT=shm#============= FDS environment =======

The root cause for segmentation fault when using GLMAT solver is due to setup of dynamic linker environment variable LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=/usr/lib64:/home/mlukezic/FDS/FDS6/bin/LIB64:$LD_LIBRARY$

Please remove /usr/lib64 and change this line

export LD_LIBRARY_PATH=/home/mlukezic/FDS/FDS6/bin/LIB64

That's it. A working FDS environment on your machine (CentOS 7.4) can be verified like this

$ ldd /home/mlukezic/FDS/FDS6/bin/fds

Expected output

all shared libraries are loaded from /lib64/

except Intel MPI related libraries loaded from /home/mlukezic/FDS/FDS6/bin/LIB64

linux-vdso.so.1 => (0x00007fff733f8000) libiomp5.so => /home/mlukezic/FDS/FDS6/bin/LIB64/libiomp5.so (0x00007f559edcf000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f559ebad000) libm.so.6 => /lib64/libm.so.6 (0x00007f559e8ab000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f559e6a7000) libmpifort.so.12 => /home/mlukezic/FDS/FDS6/bin/LIB64/libmpifort.so.12 (0x00007f559e2fd000) libmpi.so.12 => /home/mlukezic/FDS/FDS6/bin/LIB64/libmpi.so.12 (0x00007f559d5d5000) librt.so.1 => /lib64/librt.so.1 (0x00007f559d3cd000) libc.so.6 => /lib64/libc.so.6 (0x00007f559d009000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f559cdf3000) /lib64/ld-linux-x86-64.so.2 (0x000055bb94655000)

If this does not work, try these commands in an interactive shell session and post every command its output here. Otherwise you make it very hard for us to help you. Always use absolute paths for binaries like fds and mpiexec until this issue is fixed.

$ export LD_LIBRARY_PATH=/home/mlukezic/FDS/FDS6/bin/LIB64 $ echo $LD_LIBRARY_PATH $ ldd /home/mlukezic/FDS/FDS6/bin/fds $ /home/mlukezic/FDS/FDS6/bin/fds your_case_using_glmat.fds

Please let me know if that works for you.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-380507934, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbrzUjcNANNiiVuyq1It6ZB2M-B2Ijks5tnipmgaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

marcosvanella commented 6 years ago

Thank you Sascha and Salah. Marjan, I'll close this issue now. Next time you can close the issue when things are resolved.

MarjanLukezic commented 6 years ago

Thank you, I didn't know that I can close the issue. Best regards, Marjan L.

On Mon, Apr 16, 2018 at 7:03 PM, marcosvanella notifications@github.com wrote:

Thank you Sascha and Salah. Marjan, I'll close this issue now. Next time you can close the issue when things are resolved.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-381676152, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbr1rTqgYtPMQzuuKbmz7qEIk25uGvks5tpM75gaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

saschagottfried commented 6 years ago

Marcos,

I would not close the issue, since the root cause was not fixed yet. If you do not fix the root cause, this might happen again when users installing official bundle on distributions like RHEL, CentOS and Fedora.

This is an excerpt from FDS 6.6.0 Bundle installer (line 263 ff)

#--- create BASH startup file

if [ "INTEL" == "INTEL" ] ; then
cat << BASH > $BASHRCFDS
#/bin/bash
export PATH=$FDS_root/bin:\$PATH
BASH
else
cat << BASH >> $BASHRCFDS
export PATH=$FDS_root/bin:$FDS_root/bin/openmpi_64/bin:\$PATH
export OPAL_PREFIX=$FDS_root/bin/openmpi_64
BASH
fi

if [ "LINUX" == "LINUX" ] ; then
cat << BASH >> $BASHRCFDS
export LD_LIBRARY_PATH=/usr/lib64:$FDS_root/bin/LIB64:\$LD_LIBRARY_PATH
BASH
fi

Adding /usr/lib64 to LD_LIBRARY_PATH results in application crashes on CentOS 7.4. This was demonstrated in the course of the issue and is still reproducible behaviour. Same observation applies to code creating a FDS6 modules file.

The root cause has to be addressed in some way, either in this issue or another one referencing this issue.

saschagottfried commented 6 years ago

Marjan, I suggest you change the name of the issue to highlight the relation to your OS: GLMAT solver crashes on CentOS 7.4

marcosvanella commented 6 years ago

All right. Marjan, change the name of the issue and I'll add Glenn to the conversation regarding the bundle scripts. Sascha, do you have a suggestion on how to go about fixing this on our side? Thank you.

saschagottfried commented 6 years ago

Glenn might explain any (historical) reasons for adding /usr/lib64. From my point of view adding folder LIB64 to LD_LIBRARY_PATH is required to adjust dynamic linker.

export LD_LIBRARY_PATH=$FDS_root/bin/LIB64:$LD_LIBRARY_PATH

even more defensive

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$FDS_root/bin/LIB64

gforney commented 6 years ago

I added /usrr/lib64 to LD_LIBRARY_PATH because it didn't work on our systems without doing this . Marcos and I will look at this again on our end

gforney commented 6 years ago

also note, the installer is not updating the .bashrc ie not updating any environment variables in your startup files. the installer just provides a sample module and set up commands that the user needs to adapt to their system

saschagottfried commented 6 years ago

Glenn, thanks for clarifying.

Then I would suggest to change the sample module and set up commands accordingly. Currently the installer explicitly instructs users to add the configuration to their runtime environment. I could not find a hint emphasizing this is just example code that has to be adapted. That is why the user reporting this issue just did copy & paste.

Glenn, could you please share details about the shared library dependencies that made you add /usr/lib64 to the bundle installer suggestion. If this dependency turns out to be a very NIST-specific configuration that is not expected in a typical end-user setup, then we might document it in the WIKI and remove /usr/lib64 from the suggestion on dynamic linker configuration currently made in the bundle installer.

gforney commented 6 years ago

Thanks. We will look at these installation issues again before we put out our next release.

On Tue, Apr 17, 2018, 2:52 AM Sascha Gottfried notifications@github.com wrote:

Glenn, thanks for clarifying.

Then I would suggest to change the sample module and set up commands accordingly. Currently the installer explicitly instructs users to add the configuration to their runtime environment. I could not find a hint emphasizing this is just example code that has to be adapted. That is why the user reporting this issue just did copy & paste.

Glenn, could you please share details about the shared library dependencies that made you add /usr/lib64 to the bundle installer suggestion. If this dependency turns out to be a very NIST-specific configuration that is not expected in a typical end-user setup, then we might document it in the WIKI and remove /usr/lib64 from the suggestion on dynamic linker configuration currently made in the bundle installer.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-381868544, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRjqTm-TjGJhp_zmi4SBVW4X3Oa3Lks5tpZEjgaJpZM4SQfOC .

sbenkorichi commented 5 years ago

I have installed Centos 6.10, this Glmat case works ok.

MarjanLukezic commented 5 years ago

Thank you, Salah for information about Centos 6.10.

Best regards,

Marjan Lukezic

On Mon, Jul 16, 2018 at 1:08 AM Salah Benkorichi notifications@github.com wrote:

I have installed Centos 6.10, this Glmat case works ok.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-405125270, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbrxJjbHbJSlSehGqSR_Shi82pYXl7ks5uG8tqgaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

saschagottfried commented 5 years ago

Hi Salah,

thank you for reporting feedback from CentOS 6.10. From my point of view the issue was caused when CentOS users changed their environment based on suggestions from the bundle installer - as Glenn pointed out in a comment from April 16. Salah, are you telling us that you successfully ran GLMAT cases on CentOS 6.10 after running the bundle installer and applying the suggested changes to dynamic linker environment (LD_LIBRARY_PATH)?

the installer just provides a sample module and set up commands that the user needs to adapt to their system

Marjan applied this suggestion to CentOS 7.4 having caused the issue. The bundle installer (FDS6.7.0-SMV6.7.1_linux64.sh) still contains a suggestion to add /usr/lib64 to LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=/usr/lib64:\$FDSBINDIR/LIB64:\$LD_LIBRARY_PATH

Glenn said, he will look into this while putting out the 6.7.0 release. I do not know if he did.

sbenkorichi commented 5 years ago

It was tested on VM box, I tested 6.7 on Centos 6.3. I don't recall that I've added /usr/lib64 or not. But, will check it later to confirm. I might try to test Centos 7.4 just in case.

MarjanLukezic commented 5 years ago

Dear Sascha Gottfried and Salah Benkorichi,

thank you for latest information about The bundle installer ( FDS6.7.0-SMV6.7.1_linux64.sh) and modification about library path. I can't try the lastest bundle installer on my centos server right now.

Best regards,

Marjan Lukezic

On Thu, Sep 20, 2018 at 12:56 PM Salah Benkorichi notifications@github.com wrote:

It was tested on VM box, I tested 6.7 on Centos 6.3. I don't recall that I've added /usr/lib64 or not. But, will check it later to confirm. I might try to test Centos 7.4 just in case.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/6056#issuecomment-423139740, or mute the thread https://github.com/notifications/unsubscribe-auth/AMgbr2TyhnRtRAHmR_sLezxU04B5k_H0ks5uc3RNgaJpZM4SQfOC .

--

___ Marjan Lukežič

"Excellence is not a singular act, but a habit. You are what you repeatedly do." - Shaquille O'Neal

"The walls between art and engineering exist only in our minds" - Theo Jansen“Simplicity is the ultimate form of sophistication.” - Leonardo da Vinci

firemodels / fds

GLMAT solver crashes on CentOS 7.4 #6056

============= FDS environment =======

============= FDS environment =======

0 0x00002aaaacaee4fc in __GI___libc_free (mem=0x2aaaaaace730) at malloc.c:2949

1 0x00002aaaaaab0f22 in open_path (name=name@entry=0x7fffffff9f30 "libittnotify.so", namelen=namelen@entry=16,

2 0x00002aaaaaab3712 in _dl_map_object (loader=loader@entry=0x2aaaaacce150,

3 0x00002aaaaaabe964 in dl_open_worker (a=a@entry=0x7fffffff6ce8) at dl-open.c:232

4 0x00002aaaaaaba2d4 in _dl_catch_error (objname=objname@entry=0x7fffffff6cd8, errstring=errstring@entry=0x7fffffff6ce0,

5 0x00002aaaaaabe2cb in _dl_open (file=0x7fffffff9f30 "libittnotify.so", mode=-2147483391,

6 0x00002aaaab591fbb in dlopen_doit (a=a@entry=0x7fffffff6ef0) at dlopen.c:66

7 0x00002aaaaaaba2d4 in _dl_catch_error (objname=0x9c14040, errstring=0x9c14048, mallocedp=0x9c14038,

8 0x00002aaaab5925bd in _dlerror_run (operate=operate@entry=0x2aaaab591f60 , args=args@entry=0x7fffffff6ef0)

9 0x00002aaaab592051 in __dlopen (file=, mode=) at dlopen.c:87

10 0x00000000038bc357 in mkl_serv_load_inspector ()

11 0x00000000038bc66d in mkl_serv_lock ()

12 0x00000000039b0d1b in mkl_serv_get_mpi_wrappers ()

13 0x00000000038cc467 in mkl_pds_lp64_cpardiso_mpi_comm_rank ()

14 0x000000000387662f in mkl_pds_lp64_cluster_sparse_solver ()

15 0x000000000355d8a4 in globalmatrix_solver::get_h_matrix_ludcmp () at ../../Source/pres.f90:2394

16 0x0000000003540a4a in globalmatrix_solver::glmat_solver_setup_h (stage_flag=3) at ../../Source/pres.f90:1841

17 0x00000000037223bf in fds () at ../../Source/main.f90:246

18 0x000000000054a29e in main ()

19 0x00002aaaaca8fc05 in __libc_start_main (main=0x54a270 , argc=2, ubp_av=0x7fffffffd988, init=,

20 0x000000000054a129 in _start ()

0 0x00002aaaacaee4fc in __GI___libc_free (mem=0x2aaaaaace730) at malloc.c:2949

1 0x00002aaaaaab0f22 in open_path (name=name@entry=0x7fffffff9f30 "libittnotify.so", namelen=namelen@entry=16,

2 0x00002aaaaaab3712 in _dl_map_object (loader=loader@entry=0x2aaaaacce150,

3 0x00002aaaaaabe964 in dl_open_worker (a=a@entry=0x7fffffff6ce8) at dl-open.c:232

4 0x00002aaaaaaba2d4 in _dl_catch_error (objname=objname@entry=0x7fffffff6cd8, errstring=errstring@entry=0x7fffffff6ce0,

5 0x00002aaaaaabe2cb in _dl_open (file=0x7fffffff9f30 "libittnotify.so", mode=-2147483391,

6 0x00002aaaab591fbb in dlopen_doit (a=a@entry=0x7fffffff6ef0) at dlopen.c:66

7 0x00002aaaaaaba2d4 in _dl_catch_error (objname=0x9c14040, errstring=0x9c14048, mallocedp=0x9c14038,

8 0x00002aaaab5925bd in _dlerror_run (operate=operate@entry=0x2aaaab591f60 , args=args@entry=0x7fffffff6ef0)

9 0x00002aaaab592051 in __dlopen (file=, mode=) at dlopen.c:87

10 0x00000000038bc357 in mkl_serv_load_inspector ()

11 0x00000000038bc66d in mkl_serv_lock ()

12 0x00000000039b0d1b in mkl_serv_get_mpi_wrappers ()

13 0x00000000038cc467 in mkl_pds_lp64_cpardiso_mpi_comm_rank ()

14 0x000000000387662f in mkl_pds_lp64_cluster_sparse_solver ()

15 0x000000000355d8a4 in globalmatrix_solver::get_h_matrix_ludcmp () at ../../Source/pres.f90:2394

16 0x0000000003540a4a in globalmatrix_solver::glmat_solver_setup_h (stage_flag=3) at ../../Source/pres.f90:1841

17 0x00000000037223bf in fds () at ../../Source/main.f90:246

18 0x000000000054a29e in main ()

19 0x00002aaaaca8fc05 in __libc_start_main (main=0x54a270 , argc=2, ubp_av=0x7fffffffd988, init=,

20 0x000000000054a129 in _start ()

============================================================================ == Configuring Open MPI

end of input code

MKL Environment Variables:

export INTEL_COMPILERS_AND_LIBS=/home/marcos/intel/compilers_and_libraries_2017/linux

source $INTEL_COMPILERS_AND_LIBS/mkl/bin/mklvars.sh intel64

User specific aliases and functions

============= FDS environment =======

fds run from sub directory fds_mpi_gnu_linux_64

alias smv=/home/mlukezic/fds-master/Build/mpi_gnu_linux_64/bin/smokeview

To remove the stack size limit on a non-Windows system (i.e. Linux, Unix or OSX)

============= FDS environment =======

MKL Environment Variables:

TLDR;

Analysis

.bashrc

Source global definitions

Uncomment the following line if you don't like systemctl's auto-paging

export SYSTEMD_PAGER=

User specific aliases and functions

============= FDS environment =======

how to fix glmat solver error by saschagottfried - path is not right for

export LD_LIBRARY_PATH=/home/mlukezic/FDS/FDS6/bin/LIB64

MKL Environment Variables:

export INTEL_COMPILERS_ANDLIBS=/home/mlukezic/intel/compilers

source $INTEL_COMPILERS_AND_LIBS/mkl/bin/mklvars.sh intel64

fds run from sub directory fds_mpi_gnu_linux_64

alias smv=/home/mlukezic/fds-master/Build/mpi_gnu_linux_64/bin/smokeview

To remove the stack size limit on a non-Windows system (i.e. Linux, Unix

============= FDS environment =======

User specific aliases and functions

============= FDS environment =======

export

============= FDS environment =======

============= FDS environment =======export PATH=/home/mlukezic/FDS/FDS6/bin:$PATHexport LD_LIBRARY_PATH=/usr/lib64:/home/mlukezic/FDS/FDS6/bin/LIB64:$LD_LIBRARY$export OMP_NUM_THREADS=8

0 0x00002aaaacaee4fc in GI_libc_free (mem=0x2aaaaaace730) at malloc.c:2949

19 0x00002aaaaca8fc05 in __libc_start_main (main=0x54a270
, argc=2, ubp_av=0x7fffffffd988, init=,

0 0x00002aaaacaee4fc in GI_libc_free (mem=0x2aaaaaace730) at malloc.c:2949

19 0x00002aaaaca8fc05 in __libc_start_main (main=0x54a270
, argc=2, ubp_av=0x7fffffffd988, init=,