FDS 6.5.3 Upgrade Issue

[tgob]

I have FDS 6.5.2 with OpenMPI 1.8.4 running stably on a Linux Ubuntu 16.04 LTS cluster with Mellanox Infiniband. The installation was completed using the NISTpre-compiled FDS binaries. I recently attempted to upgrade to FDS 6.5.3 using the NIST precompiled binaries but there is a problem with the new installation:

Installing 64 bit Linux FDS 6.5.3 and Smokeview 6.4.4

Options: 1) Press to begin installation 2) Type "extract" to copy the installation files to FDS_6.5.3-SMV_6.4.4_linux64.tar.gz

[Enter]

FDS install options Press 1 to install in /home/ob1/FDS/FDS6 [default] Press 2 to install in /opt/FDS/FDS6 Press 3 to install in /usr/local/bin/FDS/FDS6 Enter a directory path to install elsewhere

[1][Enter]

OpenMPI options Press 1 to install OpenMPI manually [default] See /home/ob1/FDS/FDS6/bin/README.html for details Press 2 to use /shared/openmpi_64ib

[2][Enter]

Installation directory: /home/ob1/FDS/FDS6 OpenMPI directory: /shared/openmpi_64ib

Installation beginning The directory, /home/ob1/FDS/FDS6, already exists. The installation directory, /home/ob1/FDS/FDS6, has been created. Creating directory /home/ob1/FDS/FDS6/Uninstall The installation directory, /home/ob1/FDS/FDS6/Uninstall, has been created.

Copying FDS installation files to /home/ob1/FDS/FDS6 Copy complete.

Backing up /home/ob1/.bashrc_fds to /home/ob1/.bashrc_fds_20170401_102309 Updating .bashrc_fds

Backing up /home/ob1/.bashrc to /home/ob1/.bashrc_20170401_102309 Updating .bashrc

*** Log out and log back in so changes will take effect.

Installation complete.

No issues were reported during the install but when I execute fds from a terminal command prompt on the Master node I get the following output:

---

Sorry! You were supposed to get help about: ini file:file not found But I couldn't open the help file: /shared/openmpi_64/share/openmpi/help-mpi-btl-openib.txt: No such file or directory. Sorry!

libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_cmd_destroy_flow, version IBVERBS_1.0 not defined in file libibverbs.so.1 with link time reference

libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_cmd_destroy_flow, version IBVERBS_1.0 not defined in file libibverbs.so.1 with link time reference

libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

Sorry! You were supposed to get help about:

btl:no-nics

But I couldn't open the help file:

/shared/openmpi_64/share/openmpi/help-mpi-btl-base.txt: No such file or directory.  Sorry!

---

Fire Dynamics Simulator

Current Date : April 1, 2017 10:32:42

Version : FDS 6.5.3

Revision : FDS6.5.3-598-geb56ed1

Revision Date : Thu Jan 19 16:12:59 2017 -0500

Compilation Date : Jan 22, 2017 18:04:30

MPI Enabled; Number of MPI Processes: 1

OpenMP Enabled; Number of OpenMP Threads: 4

MPI version: 3.0

MPI library version: Open MPI v1.8.4, package: Open MPI gforney@burn Distribution, ident: 1.8.4, repo rev: v1.8.3-330-g0344f04, Dec 19, 2014

Consult FDS Users Guide Chapter, Running FDS, for further instructions.

Hit Enter to Escape...

For some reason FDS appears to be trying to access a non-infinband openmpi installation and associated help files at:

/shared/openmpi_64/share/openmpi

However openmpi resides in the default Infiniband installation directory (as for FDS 6.5.2):

/shared/openmpi_64ib/share/openmpi

.bashrc and .bashrc_fds are setting the environment variables appropriately, and in particular PATH, LD_LIBRARY_PATH and FDSNETWORK as follows:

PATH:

/shared/openmpi_64ib/bin:/home/ob1/FDS/FDS6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

LD_LIBRARY_PATH

/shared/openmpi_64ib/lib:/home/ob1/FDS/FDS6/bin/LIB64:/home/ob1/FDS/FDS6/bin/INTELLIBS16

FDSNETWORK

Infiniband

Past upgrades (for example from 6.5.0 to 6.5.2) have been completed successfully by simply downloading the FDS and SmokeView precompiled Linux bundle and running the script (.sh) file.

Infiniband is still working, openmpi is still working (via Infiniband) and SSH is still working (password-less access) between all nodes.

A Windows 7 upgrade to 6.5.3 worked just fine (albeit without openmpi and Infinband on my Windows workstation).

I also tried a Linux Ubuntu 16.04 LTS install without openmpi or Infiniband. This also worked fine.

Any suggestions on how I might complete the upgrade to FDS 6.5.3?

[gforney]

try installing using this option. ie use the openmpi library that came with the fds installer

OpenMPI options Press 1 to install OpenMPI manually [default]

On Fri, Mar 31, 2017 at 11:17 PM, tgob notifications@github.com wrote:

I have FDS 6.5.2 with OpenMPI 1.8.4 running stably on a Linux Ubuntu 16.04 LTS cluster with Mellanox Infiniband. The installation was completed using the NISTpre-compiled FDS binaries. I recently attempted to upgrade to FDS 6.5.3 using the NIST precompiled binaries but there is a problem with the new installation:

Installing 64 bit Linux FDS 6.5.3 and Smokeview 6.4.4

Options:

1. Press to begin installation
2. Type "extract" to copy the installation files to FDS_6.5.3-SMV_6.4.4_linux64.tar.gz

[Enter]

FDS install options Press 1 to install in /home/ob1/FDS/FDS6 [default] Press 2 to install in /opt/FDS/FDS6 Press 3 to install in /usr/local/bin/FDS/FDS6 Enter a directory path to install elsewhere

[1][Enter]

OpenMPI options Press 1 to install OpenMPI manually [default] See /home/ob1/FDS/FDS6/bin/README.html for details Press 2 to use /shared/openmpi_64ib

[2][Enter]

Installation directory: /home/ob1/FDS/FDS6 OpenMPI directory: /shared/openmpi_64ib

Installation beginning The directory, /home/ob1/FDS/FDS6, already exists. The installation directory, /home/ob1/FDS/FDS6, has been created. Creating directory /home/ob1/FDS/FDS6/Uninstall The installation directory, /home/ob1/FDS/FDS6/Uninstall, has been created.

Copying FDS installation files to /home/ob1/FDS/FDS6 Copy complete.

Backing up /home/ob1/.bashrc_fds to /home/ob1/.bashrc_fds_20170401_102309 Updating .bashrc_fds

Backing up /home/ob1/.bashrc to /home/ob1/.bashrc_20170401_102309 Updating .bashrc

*** Log out and log back in so changes will take effect.

Installation complete.

No issues were reported during the install but when I execute fds from a terminal command prompt on the Master node I get the following output:

Sorry! You were supposed to get help about: ini file:file not found But I couldn't open the help file: /shared/openmpi_64/share/openmpi/help-mpi-btl-openib.txt: No such file or directory. Sorry!

libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_cmd_destroy_flow, version IBVERBS_1.0 not defined in file libibverbs.so.1 with link time reference

libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_cmd_destroy_flow, version IBVERBS_1.0 not defined in file libibverbs.so.1 with link time reference libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

Sorry! You were supposed to get help about:

btl:no-nics

But I couldn't open the help file:

/shared/openmpi_64/share/openmpi/help-mpi-btl-base.txt: No such file or directory. Sorry!

---

Fire Dynamics Simulator

Current Date : April 1, 2017 10:32:42

Version : FDS 6.5.3

Revision : FDS6.5.3-598-geb56ed1

Revision Date : Thu Jan 19 16:12:59 2017 -0500

Compilation Date : Jan 22, 2017 18:04:30

MPI Enabled; Number of MPI Processes: 1

OpenMP Enabled; Number of OpenMP Threads: 4

MPI version: 3.0

MPI library version: Open MPI v1.8.4, package: Open MPI gforney@burn Distribution, ident: 1.8.4, repo rev: v1.8.3-330-g0344f04, Dec 19, 2014

Consult FDS Users Guide Chapter, Running FDS, for further instructions.

Hit Enter to Escape...

For some reason FDS appears to be trying to access a non-infinband openmpi installation and associated help files at:

/shared/openmpi_64/share/openmpi

However openmpi resides in the default Infiniband installation directory (as for FDS 6.5.2):

/shared/openmpi_64ib/share/openmpi

.bashrc and .bashrc_fds are setting the environment variables appropriately, and in particular PATH, LD_LIBRARY_PATH and FDSNETWORK as follows:

PATH:

/shared/openmpi_64ib/bin:/home/ob1/FDS/FDS6/bin:/usr/ local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ usr/games:/usr/local/games:/snap/bin

LD_LIBRARY_PATH

/shared/openmpi_64ib/lib:/home/ob1/FDS/FDS6/bin/LIB64:/ home/ob1/FDS/FDS6/bin/INTELLIBS16

FDSNETWORK

Infiniband

Past upgrades (for example from 6.5.0 to 6.5.2) have been completed successfully by simply downloading the FDS and SmokeView precompiled Linux bundle and running the script (.sh) file.

Infiniband is still working, openmpi is still working (via Infiniband) and SSH is still working (password-less access) between all nodes.

A Windows 7 upgrade to 6.5.3 worked just fine (albeit without openmpi and Infinband on my Windows workstation).

I also tried a Linux Ubuntu 16.04 LTS install without openmpi or Infiniband. This also worked fine.

Any suggestions on how I might complete the upgrade to FDS 6.5.3?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRiQtMDCi8wua-OlQbGaxCwiTbkGUks5rrcG9gaJpZM4MwWP5 .

-- Glenn Forney

[tgob]

Thank you for responding so quickly Glenn.

Yes, I have already tried the manual OpenMPI install option, followed by manually adjusting the .bashrc MPIDIST_IB variable to /shared/openmpi_64ib, and passing this to .bashrc_fds through the source line.

The result was the same. Running fds from a terminal command line produced the same error message as previously posted. I'm not sure why fds is looking for the directory:

            /shared/openmpi_64/share/openmpi/

The actual directory with the help files is /shared/openmpi-64ib/share/openmpi.

Have the default OpenMPI or Infiniband installation directories changed since FDS6.5.2 was built?

One thing I haven't done is to recompile OpenMPI or use the bundled OpenMPI. This was originally compiled with GNU for Infiniband using the FDS 6.5.0 default installation directories and is working just fine (as it was with FDS 6.5.0 and FDS 6.5.2).

t.

From: Glenn Forney [mailto:notifications@github.com] Sent: Saturday, 1 April 2017 4:43 p.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

try installing using this option. ie use the openmpi library that came with the fds installer

OpenMPI options Press 1 to install OpenMPI manually [default]

On Fri, Mar 31, 2017 at 11:17 PM, tgob notifications@github.com wrote:

I have FDS 6.5.2 with OpenMPI 1.8.4 running stably on a Linux Ubuntu 16.04 LTS cluster with Mellanox Infiniband. The installation was completed using the NISTpre-compiled FDS binaries. I recently attempted to upgrade to FDS 6.5.3 using the NIST precompiled binaries but there is a problem with the new installation:

Installing 64 bit Linux FDS 6.5.3 and Smokeview 6.4.4

Options:

1. Press to begin installation
2. Type "extract" to copy the installation files to FDS_6.5.3-SMV_6.4.4_linux64.tar.gz

[Enter]

FDS install options Press 1 to install in /home/ob1/FDS/FDS6 [default] Press 2 to install in /opt/FDS/FDS6 Press 3 to install in /usr/local/bin/FDS/FDS6 Enter a directory path to install elsewhere

[1][Enter]

OpenMPI options Press 1 to install OpenMPI manually [default] See /home/ob1/FDS/FDS6/bin/README.html for details Press 2 to use /shared/openmpi_64ib

[2][Enter]

Installation directory: /home/ob1/FDS/FDS6 OpenMPI directory: /shared/openmpi_64ib

Installation beginning The directory, /home/ob1/FDS/FDS6, already exists. The installation directory, /home/ob1/FDS/FDS6, has been created. Creating directory /home/ob1/FDS/FDS6/Uninstall The installation directory, /home/ob1/FDS/FDS6/Uninstall, has been created.

Copying FDS installation files to /home/ob1/FDS/FDS6 Copy complete.

Backing up /home/ob1/.bashrc_fds to /home/ob1/.bashrc_fds_20170401_102309 Updating .bashrc_fds

Backing up /home/ob1/.bashrc to /home/ob1/.bashrc_20170401_102309 Updating .bashrc

*** Log out and log back in so changes will take effect.

Installation complete.

No issues were reported during the install but when I execute fds from a terminal command prompt on the Master node I get the following output:

Sorry! You were supposed to get help about: ini file:file not found But I couldn't open the help file: /shared/openmpi_64/share/openmpi/help-mpi-btl-openib.txt: No such file or directory. Sorry!

libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_cmd_destroy_flow, version IBVERBS_1.0 not defined in file libibverbs.so.1 with link time reference

libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_cmd_destroy_flow, version IBVERBS_1.0 not defined in file libibverbs.so.1 with link time reference libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

Sorry! You were supposed to get help about:

btl:no-nics

But I couldn't open the help file:

/shared/openmpi_64/share/openmpi/help-mpi-btl-base.txt: No such file or directory. Sorry!

---

Fire Dynamics Simulator

Current Date : April 1, 2017 10:32:42

Version : FDS 6.5.3

Revision : FDS6.5.3-598-geb56ed1

Revision Date : Thu Jan 19 16:12:59 2017 -0500

Compilation Date : Jan 22, 2017 18:04:30

MPI Enabled; Number of MPI Processes: 1

OpenMP Enabled; Number of OpenMP Threads: 4

MPI version: 3.0

MPI library version: Open MPI v1.8.4, package: Open MPI gforney@burn Distribution, ident: 1.8.4, repo rev: v1.8.3-330-g0344f04, Dec 19, 2014

Consult FDS Users Guide Chapter, Running FDS, for further instructions.

Hit Enter to Escape...

For some reason FDS appears to be trying to access a non-infinband openmpi installation and associated help files at:

/shared/openmpi_64/share/openmpi

However openmpi resides in the default Infiniband installation directory (as for FDS 6.5.2):

/shared/openmpi_64ib/share/openmpi

.bashrc and .bashrc_fds are setting the environment variables appropriately, and in particular PATH, LD_LIBRARY_PATH and FDSNETWORK as follows:

PATH:

/shared/openmpi_64ib/bin:/home/ob1/FDS/FDS6/bin:/usr/ local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ usr/games:/usr/local/games:/snap/bin

LD_LIBRARY_PATH

/shared/openmpi_64ib/lib:/home/ob1/FDS/FDS6/bin/LIB64:/ home/ob1/FDS/FDS6/bin/INTELLIBS16

FDSNETWORK

Infiniband

Past upgrades (for example from 6.5.0 to 6.5.2) have been completed successfully by simply downloading the FDS and SmokeView precompiled Linux bundle and running the script (.sh) file.

Infiniband is still working, openmpi is still working (via Infiniband) and SSH is still working (password-less access) between all nodes.

A Windows 7 upgrade to 6.5.3 worked just fine (albeit without openmpi and Infinband on my Windows workstation).

I also tried a Linux Ubuntu 16.04 LTS install without openmpi or Infiniband. This also worked fine.

Any suggestions on how I might complete the upgrade to FDS 6.5.3?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRiQtMDCi8wua-OlQbGaxCwiTbkGUks5rrcG9gaJpZM4MwWP5 .

-- Glenn Forney

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290891899 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypG0BDP-PUMYWdmqZuRX3GdOyYFTIks5rrceogaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Can you try the following: 1.) Reinstall FDS 6.5.3 , choose FDS install options Press 1 to install in /home/ob1/FDS/FDS6 [default] 1 OpenMPI options Press 1 to install OpenMPI manually [default] 1 Continue yes, yes, since you're overwritten the previous installation.

2) cd to FDS/FDS6/bin and then type:

 gunzip openmpi_1.8.4_linux_64.tar.gz
 tar -xvf openmpi_1.8.4_linux_64.tar

Open your .bashrc Make sure these two lines are found:

export MPIDIST_FDS=/home/ob1/FDS/FDS6/bin/openmpi_64
source ~/.bashrc_fds $MPIDIST_FDS

Now, test if mpirun works or not with any simple example case.

[tgob]

Thank you for your reply Salah.

I can do this but I doubt that your precompiled version of OpenMPI will utilize my Mellanox Infiniband network.

It took quite a lot of effort to get OpenMPI 1.8.4 working over Infiniband on my cluster, however the current OpenMPI install runs over Infiniband with FDS 6.5.2 and FDS 6.5.0 using the FDS documented default installation directories.

You can read about the install at procedure in the attached document.

Would you please advise if the directory structure for OpenMPI has changed from the FDS 6.5.2 to 6.5.3 build.

With kindest regards,

Tim

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Saturday, 1 April 2017 7:53 p.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Can you try the following: 1.) Reinstall FDS 6.5.3 , choose FDS install options Press 1 to install in /home/ob1/FDS/FDS6 [default] 1 OpenMPI options Press 1 to install OpenMPI manually [default] 1 Continue yes, yes, since you're overwritten the previous installation.

2. cd to FDS/FDS6/bin and then type:

   gunzip openmpi_1.8.4_linux_64.tar.gz tar -xvf openmpi_1.8.4_linux_64.tar

Open your .bashrc Make sure these two lines are found:

export MPIDIST_FDS=/home/ob1/FDS/FDS6/bin/openmpi_64 source ~/.bashrc_fds $MPIDIST_FDS

Now, test if mpirun works or not with any simple example case.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290900978 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypAqkLw3yIIlfn3BP7f4nCTAIqdroks5rrfRmgaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Can you run these commands

[salah@jbk27s092]$ echo $MPIDIST_FDS
/home/salah/FDS/FDS6/bin/openmpi_64
[salah@jbk27s092]$  echo $MPIDIST_ETH
/shared/openmpi_64
[salah@jbk27s092]$ echo $MPIDIST_IB

[salah@jbk27s092]$ echo $MPIDIST
/shared/openmpi_64

[tgob]

Here are the variables from the original install as listed in my initial post:

FDS install options

Press 1 to install in /home/ob1/FDS/FDS6 [default]

OpenMPI options Press 2 to use /shared/openmpi_64ib

$MPIDIST_FDS /home/ob1/FDS/FDS6/bin/openmpi_64

$MPIDIST_ETH ("" or null)

$MPIDIST_IB /shared/openmpi_64ib

$MPIDIST /shared/openmpi_64ib

These are all set correctly in accordance with .bashrc and .bashrc_fds

The problem appears to be associated with the reference to shared/openmpi_64/share/openmpi/.

The actual directory that contains these files is shared/openmpi_64ib/share/openmpi/.

Which takes me back to my question, have the default Infiniband and OpenMPI installation directories changed between FDS 6.5.2 and 6.5.3?

With kindest regards,

Tim

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Saturday, 1 April 2017 11:05 p.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Can you run these commands

[salah@jbk27s092]$ echo $MPIDIST_FDS /home/salah/FDS/FDS6/bin/openmpi_64 [salah@jbk27s092]$ echo $MPIDIST_ETH /shared/openmpi_64 [salah@jbk27s092]$ echo $MPIDIST_IB

[salah@jbk27s092]$ echo $MPIDIST /shared/openmpi_64

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290910480 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypBLVbAZDhsozdUSYxdqFuhiYOYSZks5rriFlgaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Tim, As far as I know it didn't change. The colleagues at NIST are using the Latest version as well and it's working. Wait for Kevin or gforney to detail on this matter.

Regards, Salah

[gforney]

Fds doesn't care where the openmpi library is located. But you need to tell it correctly. The installer script "asks" for openmpi location so it can set environment varuables. ( PATH, etc). So edit your .bashrc to have source ~/.bashrc path_to_open_mpi_library w

On Apr 1, 2017 8:08 AM, "Salah Benkorichi" notifications@github.com wrote:

Tim, As far as I know it didn't change. The colleagues at NIST are using the Latest version as well and it's working. Wait for Kevin or gforney to detail on this matter.

Regards, Salah

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290916197, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRvSzz_KOypZVLD2UGwvJHDTNUgrcks5rrj5QgaJpZM4MwWP5 .

[tgob]

The previously posted error message indicates that there is an issue with how FDS 6.5.3 is interpreting the variables passed to it. This is not an issue with either .bashrc or .bashrc_fds. FDS 6.5.3 is looking for error message files in a non-existent directory. I figure that this is specific to Infiniband installs.

I'll start looking at the code in detail over the next few days to try and sort out what is going on, but I reiterate that FDS 6.5.2 and 6.5.0 worked perfectly from the bundled .sh install files, and my install of openmpi 1.8.4 is rock solid using Infiniband. The only thing that has changed here is FDS 6.5.3.

With kindest regards,

Tim

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 1:09 a.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Tim, As far as I know it didn't change. The colleagues at NIST are using the Latest version as well and it's working. Wait for Kevin or gforney to detail on this matter.

Regards, Salah

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290916197 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypChLe_zhvynRtefRkJ1rSAIeCr_Aks5rrj5QgaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Tim, Could you please share the fds environment in your .bashrc file. FDS 6.5.3 changes your .bashrc after installing it, Check if this library path is set for fds_bashrc source ~/.bashrc_fds $MPIDIST_IB

[tgob]

Yes, that is exactly what is in .bashrc.

The file is attached.

I really do appreciate NIST's responsiveness on this. Thank you for this.

t.

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 1:58 a.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Tim, Could you please share the fds environment in your .bashrc file. FDS 6.5.3 changes your .bashrc after installing it, Check if this library path is set for fds_bashrc source ~/.bashrc_fds $MPIDIST_IB

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290918596 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypEES43J1a3AK4gXw_8QB_ZlHgzspks5rrknUgaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Tim, There is no attached file. You mentioned earlier about the installation document, you didn't attach it. Attach it as .txt file or zip it. Some formats are not supported.

[tgob]

Thank you for your email Glenn.

It's nice to know that FDS shouldn't care where openmpi is installed (equivalent to earlier version behaviour) but this doesn't seem to hold for Infiniband.

My understanding is that the FDS install script detects openmpi (and openmpi with ib) if it is installed in the default directory locations. This worked perfectly with FDS6.5.2 and 6.5.0. FDS 6.5.3 detects /shared/openmpi_64ib correctly and offers this as the installation parameter. That's good but why is it looking for an error message in /shared/openmpi_64/share/openmpi? Look at these directories again please. /shared/openmpi-64ib/share/openmpi exists. . /shared/openmpi-64/share/openmpi does not exist.

I'll be looking at the FDS code over the next few days to see if I can figure out what has changed. But it's way too late here in God's Own (NZ) for my tired grey cells to solve this.

With kindest regards,

t.

From: Glenn Forney [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 1:25 a.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Fds doesn't care where the openmpi library is located. But you need to tell it correctly. The installer script "asks" for openmpi location so it can set environment varuables. ( PATH, etc). So edit your .bashrc to have source ~/.bashrc path_to_open_mpi_library w

On Apr 1, 2017 8:08 AM, "Salah Benkorichi" notifications@github.com wrote:

Tim, As far as I know it didn't change. The colleagues at NIST are using the Latest version as well and it's working. Wait for Kevin or gforney to detail on this matter.

Regards, Salah

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290916197, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRvSzz_KOypZVLD2UGwvJHDTNUgrcks5rrj5QgaJpZM4MwWP5 .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290917050 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypJU_bWI2mHt7ZVM1NM2LLnv9bCJPks5rrkINgaJpZM4MwWP5 .Image removed by sender.

[tgob]

Here they are in .zip format.

t.

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 2:13 a.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Tim, There is no attached file. You mentioned earlier about the installation document, you didn't attach it. Attach it as .txt file or zip it. Some formats are not supported.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290919296 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypDakCTPWYHNX8IvFmLJLQ6pOrQ4aks5rrk1bgaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Tim I still don't see any file. First click on view in github and reply from there than just replying back to emails. The just drag and drop it.

[tgob]

Try this.., Install.zip

[sbenkorichi]

I've checked your .bashrc file, the parameters are set in accordance. This version was compiled with intel 16, while your Openmpi was done previously with GNU on 6.5.0. Can you try to test this plz : modify your .bashrc file by adding IFORT as this line below : source ~/.bashrc_fds $MPIDIST_IB $IFORT_COMPILER_LIB on your terminal update your .bashrc: source ~/.bashrc Then check if fds now is working. Otherwise, I would recommend you use the latest version of OpenMPI https://www.open-mpi.org/software/ompi/v2.1/ , and if you can get intel 17 compiler it would be better as well. https://software.intel.com/en-us/intel-parallel-studio-xe They provide a free version for students, that you can use on 3 different machines. And, maybe @gforney, might have other suggestions that might fix it within these setup.

Regards, Salah

[tgob]

No joy Salah - same error message.

The variable $IFORT_COMPILER_LIB is not initialized in .bashrc so the proposed change is simply passing a null as the second parameter to .bashrc_fds.

Similarly, $INTEL_SHARED_LIB is initialized to /intel64 in .bashrc. There is no intel64 directory at the root or in ~/FDS/FDS6/bin/INTELLIB16.

Would you please confirm that I should be changing to OpenMPI version 2.1. In the past anything other than 1.8.4 (from FDS 6.3.0) caused problems, as identified in the FDS Users manual.

Unfortunately I am not a student and I use fds for commercial fire engineering design. The Intel compiler suit is around US$3,000 (plus GST) for a version that supports mpi .

With kindest regards,

t.

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 11:22 a.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

I've checked your .bashrc file, the parameters are set in accordance. This version was compiled with intel 16, while your Openmpi was done previously with GNU on 6.5.0. Can you try to test this plz : modify your .bashrc file by adding IFORT as this line below : source ~/.bashrc_fds $MPIDIST_IB $IFORT_COMPILER_LIB on your terminal update your .bashrc: source ~/.bashrc Then check if fds now is working. Otherwise, I would recommend you use the latest version of OpenMPI https://www.open-mpi.org/software/ompi/v2.1/ , and if you can get intel 17 compiler it would be better as well. https://software.intel.com/en-us/intel-parallel-studio-xe They provide a free version for students, that you can use on 3 different machines. And, maybe @gforney https://github.com/gforney , might have other suggestions that might fix it within these setup.

Regards, Salah

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290953768 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypHRIJ3V9wGqyZhevHITM9rc7zJrxks5rrtw1gaJpZM4MwWP5 .Image removed by sender.

[gforney]

Where exactly is your infiniband openmpi library located? What version is it? What compiler was used to build it?

On Apr 1, 2017 10:12 PM, "tgob" notifications@github.com wrote:

No joy Salah - same error message.

The variable $IFORT_COMPILER_LIB is not initialized in .bashrc so the proposed change is simply passing a null as the second parameter to .bashrc_fds.

Similarly, $INTEL_SHARED_LIB is initialized to /intel64 in .bashrc. There is no intel64 directory at the root or in ~/FDS/FDS6/bin/INTELLIB16.

Would you please confirm that I should be changing to OpenMPI version 2.1. In the past anything other than 1.8.4 (from FDS 6.3.0) caused problems, as identified in the FDS Users manual.

Unfortunately I am not a student and I use fds for commercial fire engineering design. The Intel compiler suit is around US$3,000 (plus GST) for a version that supports mpi .

With kindest regards,

t.

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 11:22 a.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

I've checked your .bashrc file, the parameters are set in accordance. This version was compiled with intel 16, while your Openmpi was done previously with GNU on 6.5.0. Can you try to test this plz : modify your .bashrc file by adding IFORT as this line below : source ~/.bashrc_fds $MPIDIST_IB $IFORT_COMPILER_LIB on your terminal update your .bashrc: source ~/.bashrc Then check if fds now is working. Otherwise, I would recommend you use the latest version of OpenMPI https://www.open-mpi.org/software/ompi/v2.1/ , and if you can get intel 17 compiler it would be better as well. https://software.intel.com/en-us/intel-parallel-studio-xe They provide a free version for students, that you can use on 3 different machines. And, maybe @gforney https://github.com/gforney , might have other suggestions that might fix it within these setup.

Regards, Salah

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ firemodels/fds/issues/4904#issuecomment-290953768 , or mute the thread < https://github.com/notifications/unsubscribe-auth/ ATPypHRIJ3V9wGqyZhevHITM9rc7zJrxks5rrtw1gaJpZM4MwWP5> .Image removed by sender.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290959895, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRuZTFrrlsDv_zjWSoJ3lGZz7RGrYks5rrwP7gaJpZM4MwWP5 .

[tgob]

OpenMPI is installed at: /shared/openmpi_64/bin

The OpenMPI libraries are at: /shared/openmpi_64/lib

OpenMPI is version 1.8.4

It was compiled using GNU g++ 4.8.2 in accordance with the OpenMPI Installation Instructions.

OpenMPI works fine with both FDS 6.5.2 and FDS 6.5.0 over Infinband using the bundled NIST binaries. It also works fine with the OpenMPI test programs connectivity_c, hello_c and ring_c.

t.

From: Glenn Forney [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 3:29 p.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Where exactly is your infiniband openmpi library located? What version is it? What compiler was used to build it?

On Apr 1, 2017 10:12 PM, "tgob" notifications@github.com wrote:

No joy Salah - same error message.

The variable $IFORT_COMPILER_LIB is not initialized in .bashrc so the proposed change is simply passing a null as the second parameter to .bashrc_fds.

Similarly, $INTEL_SHARED_LIB is initialized to /intel64 in .bashrc. There is no intel64 directory at the root or in ~/FDS/FDS6/bin/INTELLIB16.

Would you please confirm that I should be changing to OpenMPI version 2.1. In the past anything other than 1.8.4 (from FDS 6.3.0) caused problems, as identified in the FDS Users manual.

Unfortunately I am not a student and I use fds for commercial fire engineering design. The Intel compiler suit is around US$3,000 (plus GST) for a version that supports mpi .

With kindest regards,

t.

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 11:22 a.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

I've checked your .bashrc file, the parameters are set in accordance. This version was compiled with intel 16, while your Openmpi was done previously with GNU on 6.5.0. Can you try to test this plz : modify your .bashrc file by adding IFORT as this line below : source ~/.bashrc_fds $MPIDIST_IB $IFORT_COMPILER_LIB on your terminal update your .bashrc: source ~/.bashrc Then check if fds now is working. Otherwise, I would recommend you use the latest version of OpenMPI https://www.open-mpi.org/software/ompi/v2.1/ , and if you can get intel 17 compiler it would be better as well. https://software.intel.com/en-us/intel-parallel-studio-xe They provide a free version for students, that you can use on 3 different machines. And, maybe @gforney https://github.com/gforney , might have other suggestions that might fix it within these setup.

Regards, Salah

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ firemodels/fds/issues/4904#issuecomment-290953768 , or mute the thread < https://github.com/notifications/unsubscribe-auth/ ATPypHRIJ3V9wGqyZhevHITM9rc7zJrxks5rrtw1gaJpZM4MwWP5> .Image removed by sender.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290959895, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRuZTFrrlsDv_zjWSoJ3lGZz7RGrYks5rrwP7gaJpZM4MwWP5 .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290962240 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypFDe9__mwcs_b5HEglgXNgh_sDWAks5rrxXqgaJpZM4MwWP5 .Image removed by sender.

[gforney]

Based on what you said I would have said you openmpi library/distribution is located at /shared/openmpi_64. Do you have this line in your .bashrc file?

source ~/.bashrc_fds /shared/openmpi_64

When you type fds at a command line what does it say?

On Apr 2, 2017 4:57 AM, "tgob" notifications@github.com wrote:

OpenMPI is installed at: /shared/openmpi_64/bin

The OpenMPI libraries are at: /shared/openmpi_64/lib

OpenMPI is version 1.8.4

It was compiled using GNU g++ 4.8.2 in accordance with the OpenMPI Installation Instructions.

OpenMPI works fine with both FDS 6.5.2 and FDS 6.5.0 over Infinband using the bundled NIST binaries. It also works fine with the OpenMPI test programs connectivity_c, hello_c and ring_c.

t.

From: Glenn Forney [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 3:29 p.m.

To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Where exactly is your infiniband openmpi library located? What version is it? What compiler was used to build it?

On Apr 1, 2017 10:12 PM, "tgob" notifications@github.com wrote:

No joy Salah - same error message.

The variable $IFORT_COMPILER_LIB is not initialized in .bashrc so the proposed change is simply passing a null as the second parameter to .bashrc_fds.

Similarly, $INTEL_SHARED_LIB is initialized to /intel64 in .bashrc. There is no intel64 directory at the root or in ~/FDS/FDS6/bin/INTELLIB16.

Would you please confirm that I should be changing to OpenMPI version 2.1. In the past anything other than 1.8.4 (from FDS 6.3.0) caused problems, as identified in the FDS Users manual.

Unfortunately I am not a student and I use fds for commercial fire engineering design. The Intel compiler suit is around US$3,000 (plus GST) for a version that supports mpi .

With kindest regards,

t.

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 11:22 a.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

I've checked your .bashrc file, the parameters are set in accordance. This version was compiled with intel 16, while your Openmpi was done previously with GNU on 6.5.0. Can you try to test this plz : modify your .bashrc file by adding IFORT as this line below : source ~/.bashrc_fds $MPIDIST_IB $IFORT_COMPILER_LIB on your terminal update your .bashrc: source ~/.bashrc Then check if fds now is working. Otherwise, I would recommend you use the latest version of OpenMPI https://www.open-mpi.org/software/ompi/v2.1/ , and if you can get intel 17 compiler it would be better as well. https://software.intel.com/en-us/intel-parallel-studio-xe They provide a free version for students, that you can use on 3 different machines. And, maybe @gforney https://github.com/gforney , might have other suggestions that might fix it within these setup.

Regards, Salah

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/

firemodels/fds/issues/4904#issuecomment-290953768> , or mute the thread < https://github.com/notifications/unsubscribe-auth/ ATPypHRIJ3V9wGqyZhevHITM9rc7zJrxks5rrtw1gaJpZM4MwWP5> .Image removed by sender.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290959895, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRuZTFrrlsDv_ zjWSoJ3lGZz7RGrYks5rrwP7gaJpZM4MwWP5 .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ firemodels/fds/issues/4904#issuecomment-290962240 , or mute the thread < https://github.com/notifications/unsubscribe-auth/ATPypFDe9__mwcs_ b5HEglgXNgh_sDWAks5rrxXqgaJpZM4MwWP5> .Image removed by sender.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290973767, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRiNHZ3spuqCzjfn38BxA6y_4vwvCks5rr2LwgaJpZM4MwWP5 .

[sbenkorichi]

Yes, that's what he has in his bashrc, bashrc.txt

FDS -----------------------------------

export MPIDIST_FDS=/home/ob1/FDS/FDS6/bin/openmpi_64 export MPIDIST_ETH= export MPIDIST_IB=/shared/openmpi_64ib INTEL_SHARED_LIB=$IFORT_COMPILER_LIB/intel64 source ~/.bashrc_fds $MPIDIST_IB

FDS -----------------------------------

[sbenkorichi]

As he said, once he tries to execute it in the terminal, it throws a warnings as he shared above, this might be an issue of libraries. No issues were reported during the install but when I execute fds from a terminal command prompt on the Master node I get the following output:

Sorry! You were supposed to get help about: ini file:file not found But I couldn't open the help file: /shared/openmpi_64/share/openmpi/help-mpi-btl-openib.txt: No such file or directory. Sorry!

libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_cmd_destroy_flow, version IBVERBS_1.0 not defined in file libibverbs.so.1 with link time reference

libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_cmd_destroy_flow, version IBVERBS_1.0 not defined in file libibverbs.so.1 with link time reference

libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

Sorry! You were supposed to get help about:

btl:no-nics But I couldn't open the help file:

/shared/openmpi_64/share/openmpi/help-mpi-btl-base.txt: No such file or directory. Sorry! Fire Dynamics Simulator

Current Date : April 1, 2017 10:32:42

Version : FDS 6.5.3

Revision : FDS6.5.3-598-geb56ed1

Revision Date : Thu Jan 19 16:12:59 2017 -0500

Compilation Date : Jan 22, 2017 18:04:30

MPI Enabled; Number of MPI Processes: 1

OpenMP Enabled; Number of OpenMP Threads: 4

MPI version: 3.0

MPI library version: Open MPI v1.8.4, package: Open MPI gforney@burn Distribution, ident: 1.8.4, repo rev: v1.8.3-330-g0344f04, Dec 19, 2014

Consult FDS Users Guide Chapter, Running FDS, for further instructions.

Hit Enter to Escape...

[tgob]

My error entirely Glenn ( I am transcribing stuff from the Linux cluster [which does almost nothing except FDS] to my regular Windows workstation via my hand-written notebooks which I keep as a record of what I have done).

OpenMPI is installed at: /shared/openmpi_64ib/bin

The OpenMPI libraries are at: /shared/openmpi_64ib/lib

Yes, these are in .bashrc and appear in $PATH and $LD_LIBRARY_PATH on interactive and non-interactive login.

The fds error test has been posted previously (and I see that Salah has posted this again).

With kindest regards,

Tim

From: Glenn Forney [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 10:51 p.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Based on what you said I would have said you openmpi library/distribution is located at /shared/openmpi_64. Do you have this line in your .bashrc file?

source ~/.bashrc_fds /shared/openmpi_64

When you type fds at a command line what does it say?

On Apr 2, 2017 4:57 AM, "tgob" notifications@github.com wrote:

OpenMPI is installed at: /shared/openmpi_64/bin

The OpenMPI libraries are at: /shared/openmpi_64/lib

OpenMPI is version 1.8.4

It was compiled using GNU g++ 4.8.2 in accordance with the OpenMPI Installation Instructions.

OpenMPI works fine with both FDS 6.5.2 and FDS 6.5.0 over Infinband using the bundled NIST binaries. It also works fine with the OpenMPI test programs connectivity_c, hello_c and ring_c.

t.

From: Glenn Forney [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 3:29 p.m.

To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Where exactly is your infiniband openmpi library located? What version is it? What compiler was used to build it?

On Apr 1, 2017 10:12 PM, "tgob" notifications@github.com wrote:

No joy Salah - same error message.

The variable $IFORT_COMPILER_LIB is not initialized in .bashrc so the proposed change is simply passing a null as the second parameter to .bashrc_fds.

Similarly, $INTEL_SHARED_LIB is initialized to /intel64 in .bashrc. There is no intel64 directory at the root or in ~/FDS/FDS6/bin/INTELLIB16.

Would you please confirm that I should be changing to OpenMPI version 2.1. In the past anything other than 1.8.4 (from FDS 6.3.0) caused problems, as identified in the FDS Users manual.

Unfortunately I am not a student and I use fds for commercial fire engineering design. The Intel compiler suit is around US$3,000 (plus GST) for a version that supports mpi .

With kindest regards,

t.

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Sunday, 2 April 2017 11:22 a.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

I've checked your .bashrc file, the parameters are set in accordance. This version was compiled with intel 16, while your Openmpi was done previously with GNU on 6.5.0. Can you try to test this plz : modify your .bashrc file by adding IFORT as this line below : source ~/.bashrc_fds $MPIDIST_IB $IFORT_COMPILER_LIB on your terminal update your .bashrc: source ~/.bashrc Then check if fds now is working. Otherwise, I would recommend you use the latest version of OpenMPI https://www.open-mpi.org/software/ompi/v2.1/ , and if you can get intel 17 compiler it would be better as well. https://software.intel.com/en-us/intel-parallel-studio-xe They provide a free version for students, that you can use on 3 different machines. And, maybe @gforney https://github.com/gforney , might have other suggestions that might fix it within these setup.

Regards, Salah

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/

firemodels/fds/issues/4904#issuecomment-290953768> , or mute the thread < https://github.com/notifications/unsubscribe-auth/ ATPypHRIJ3V9wGqyZhevHITM9rc7zJrxks5rrtw1gaJpZM4MwWP5> .Image removed by sender.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290959895, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRuZTFrrlsDv_ zjWSoJ3lGZz7RGrYks5rrwP7gaJpZM4MwWP5 .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ firemodels/fds/issues/4904#issuecomment-290962240 , or mute the thread < https://github.com/notifications/unsubscribe-auth/ATPypFDe9__mwcs_ b5HEglgXNgh_sDWAks5rrxXqgaJpZM4MwWP5> .Image removed by sender.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290973767, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRiNHZ3spuqCzjfn38BxA6y_4vwvCks5rr2LwgaJpZM4MwWP5 .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290978619 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypK8K7YINqswCblwpGli62aYn-22iks5rr314gaJpZM4MwWP5 .Image removed by sender.

[gforney]

Thought he said his library was at /shared/openmpi_64 not /shared/openmpi_64ib

On Apr 2, 2017 6:58 AM, "Salah Benkorichi" notifications@github.com wrote:

Yes, that's what he has in his bashrc, bashrc.txt https://github.com/firemodels/fds/files/888314/bashrc.txt

FDS -----------------------------------

export MPIDIST_FDS=/home/ob1/FDS/FDS6/bin/openmpi_64 export MPIDIST_ETH= export MPIDIST_IB=/shared/openmpi_64ib INTEL_SHARED_LIB=$IFORT_COMPILER_LIB/intel64 source ~/.bashrc_fds $MPIDIST_IB

FDS -----------------------------------

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-290979033, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRv_lef4wg9NpNlOm5yTNCYim_yKeks5rr39RgaJpZM4MwWP5 .

[gforney]

what version of openmpi do you have?

here are a couple of experiments to try

The following link contains an openmpi 1.8.4 infiniband distribution and an fds linux executable built against this openmpi library. The source for this fds is identical to the source used to build the latest "official" fds, ie githash eb56ed1 .
https://drive.google.com/drive/folders/0B-W-dkXwdHWNSVJhWXJBMXlSMDQ?usp=sharing

Experiment 1

1. create a directory named ~/test
2. cd ~/test
3. assuming you downloaded the openmpi...tar.gz file to your home directory, type: tar xvf ~/openmpi_1.8.4_linux_64ib.tar.gz
4. edit your .bashrc file replacing the source ~/.bashrc_fds xxx to the following source ~/test/openmpi_64ib
5. log out and log back in then type fds what does it say?

Experiment 2

1. do steps 1 -> 4 in experiment 1
2. assuming you downloaded fds_mpi_intel_linux_64ib from the above google drive link to the current directory, type ./fds_mpi_intel_linux_64ib what does it say?

Experiment 3 reinstall the "official" FDS but select the openmpi library we distribute

with all these experiments, after any edits to .bashrc you have to log out and log back in. (in theory you can just source .bashrc - but logging out and logging back in is "safer" )

Note all previous fds' that we have distributed were built against an ethernet (not infiniband) version of Openmpi.

[20:54:34 gforney@blaze:~ ] $ ls /shared/openmpi_64ib bin etc include lib share

You need to invoke .bashrc_fds using source ~/.bashrc_fds /shared/openmpi_64ib

[tgob]

Thanks for the suggestions Glenn.

OpenMPI is 1.8.4 from https://www.open-mpi.org/software/ompi/v1.8/

First up I'll try recompiling OpenMPI with Infiniband support into /shared/openmpi_64. If nothing else this should correct the fds error messages.

You have previously indicated that FDS6.5.3 doesn't actually care where OpenMPI is located so long as its location (and the associated dynamically linked libraries) are correctly passed to .bashrc_fds. For whatever reason FDS is loosing the ib in the current directory path to the error messages (it is looking for /shared/openmpi_64/share/openmpi/help-mpi-btl-openib.txt when it should be looking for for /shared/openmpi_64ib/share/openmpi/help-mpi-btl-openib.txt). I figure that if fds is looking in the wrong place for the error messages then it is probably looking in the wrong place for init files which is the cause of the error.

I shall then proceed with your suggested experiments.

This may take a few days but I'll report back shortly...

From: Glenn Forney [mailto:notifications@github.com] Sent: Monday, 3 April 2017 1:40 p.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

what version of openmpi do you have?

here are a couple of experiments to try

The following link contains an openmpi 1.8.4 infiniband distribution and an fds linux executable built against this openmpi library. The source for this fds is identical to the source used to build the latest "official" fds, ie githash https://github.com/firemodels/fds/commit/eb56ed1a8a2205333c5b98d636226159ba063d20 eb56ed1 . https://drive.google.com/drive/folders/0B-W-dkXwdHWNSVJhWXJBMXlSMDQ?usp=sharing

Experiment 1

1. create a directory named ~/test
2. cd ~/test
3. assuming you downloaded the openmpi...tar.gz file to your home directory, type: tar xvf ~/openmpi_1.8.4_linux_64ib.tar.gz
4. edit your .bashrc file replacing the source ~/.bashrc_fds xxx to the following source ~/test/openmpi_64ib
5. log out and log back in then type fds what does it say?

Experiment 2

1. do steps 1 -> 4 in experiment 1
2. assuming you downloaded fds_mpi_intel_linux_64ib from the above google drive link to the current directory, type ./fds_mpi_intel_linux_64ib what does it say?

Experiment 3 reinstall the "official" FDS but select the openmpi library we distribute

with all these experiments, after any edits to .bashrc you have to log out and log back in. (in theory you can just source .bashrc - but logging out and logging back in is "safer" )

Note all previous fds' that we have distributed were built against an ethernet (not infiniband) version of Openmpi.

[20:54:34 gforney@blaze:~ ] $ ls /shared/openmpi_64ib bin etc include lib share

You need to invoke .bashrc_fds using source ~/.bashrc_fds /shared/openmpi_64ib

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-291031445 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypCG4JyRY_5GckYDZqOvSRoqYYHhuks5rsE3WgaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Tim, Try what gforney has suggested. As for the type of the error your receiving might be due to something broken in your ompi. There is miscommunication and setup between the libraries. It's sort of generic issue. I've seen other people receiving it after they non properly install and set ompi with whatever they want to run it with. Let us know how it went.

[tgob]

Thank you for your further response Salah.

In trying to sort out the source of this error my focus is on what was working and what has changed with the installation.

My OpenMPI installation over IB has worked perfectly with FDS 6.5.2 for over 6 months clocking up 1,000s of core hours of processing on numerous fds models. It also runs OpenMPI diagnostics without error, has been subjected to multi-mesh verification, and strong and weak scaling tests. My OpenMPI installation also worked faultlessly with FDS 6.5.0.

Each of my experiments is preceded by a clean mirror install. The OS kernel is locked and other Linux software updates are highly managed. OpenMPI over Infiniband continues to perform faultlessly across all nodes in my cluster.

The only thing that has changed in recent times is fds (through the upgrade to FDS 6.5.3). The fds error that I have reported is from running the command fds at a terminal prompt on a single node. So OpenMPI should not even be instigated and Infiniband should not be exercised. In my mind this suggests that the problem is not with my OpenMPI or Infiniband installation, but with FDS 6.5.3.

From my efforts to date it appears that FDS 6.5.3 is looking in the wrong directory for OpenMPI components and error messages. This appears to be something to do with OpenMPI installed in the /shared/openmpi-64ib directory as the error messages indicate that fds is searching for OpenMPI components in /shared/openmpi_64. The 'ib' in the path changes the $FDSNETWORK environment variable to 'Infiniband' in .bashrc_fds, but I have yet to establish what this does in the compiled code.

Please elaborate if you can see a flaw in what I think is a logical diagnostic approach.

In a preceding post you recommended updating to OpenMPI Version 2.1 and I have asked you to confirm this as the FDS documentation still recommends Version 1.8.4. I agree that there are good reasons for updating to a more recent version of OpenMPI (primarily because OpenMPI Version 1.8.4 is now listed as 'retired' by the OpenMPI development team). However the more changes that I implement to my system without discovering the cause of the issue actually makes solving the problem potentially more difficult.

I appreciate that I seem to be the only dude having a problem with FDS 6.5.3 running with OpenMPI implemented over Infiniband on a Linux cluster. If anyone else in the community has had similar problems I would love to hear from you and so would NIST.

With kindest regards,

Tim

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Monday, 3 April 2017 5:51 p.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Tim, Try what gforney has suggested. As for the type of the error your receiving might be due to something broken in your ompi. There is miscommunication and setup between the libraries. It's sort of generic issue. I've seen other people receiving it after they non properly install and set ompi with whatever they want to run it with. Let us know how it went.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-291054180 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypMOtgwb_ncxIgQq3W-k64txEFVeRks5rsIjWgaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Tim, I myself using the latest version of MPI, it worked fine for me. The latest version ought work fine too with you since you're going to compile it. As for the latest release of FDS, there are somethings didn't work directly with it as it did with the previous releases. However, these bugs have been fixed. It could be an issue of sensitivities between the libraries, I can't comment on it. However, I recommend at first you dont touch the /shared/openmpi_64 directory as it is. And follow the suggestion of Gleen, to install it under another directory to see what's the reason behind this bug.

[tgob]

Thank you for your reply Salah

This GitHub thread is getting to be a really messy post. Perhaps we can work off line to resolve this issue?

I tried OpenMPI 2.1.0 today (compiled with GNU). Worked fine with other stuff but crashed and burned with the NIST FDS 6.5.2 and 6.5.3 Intel bundles.

I have also started on Glenn's experiments but I need some clarification.

For experiment 1 which version of FDS do you want me to use? I assume that you want me to try the latest NIST FDS6.5.3 bundle compiled with Intel. I have commented on Glenn's instructions below:

Experiment 1

Environment: NIST FDS 6.5.3 installed with manual OpenMPI. Previous FDS and OpenMPI installs uninstalled. Ubuntu 16.04.

1. create a directory named ~/test Done
2. cd ~/test Done
3. assuming you downloaded the openmpi...tar.gz file to your home directory (Yes), type: tar xvf ~/openmpi_1.8.4_linux_64ib.tar.gz Done
4. edit your .bashrc file replacing the source ~/.bashrc_fds xxx to the following source ~/bashrc_fds ~/test/openmpi_64ib Done
5. log out and log back in then type fds Done what does it say? What appears to be the same error as previously posted. I'll capture this and send it shortly.

t.

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Monday, 3 April 2017 11:19 p.m. To: firemodels/fds Cc: tgob; Author Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Tim, I myself using the latest version of MPI, it worked fine for me. The latest version ought work fine too with you since you're going to compile it. As for the latest release of FDS, there are somethings didn't work directly with it as it did with the previous releases. However, these bugs have been fixed. It could be an issue of sensitivities between the libraries, I can't comment on it. However, I recommend at first you dont touch the /shared/openmpi_64 directory as it is. And follow the suggestion of Gleen, to install it under another directory to see what's the reason behind this bug.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-291115040 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypFwHZitT7_J9iyZ2YHBGIAlfxeMTks5rsNW_gaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Tim, no worries, you are doing fine. We need to keep the conversation here, so this might be referred to in case other users might face such issue. For the FDS version, we would like you use the latest version, in case there will be an issue, it will be fixed in the coming release. ( there wont be any point to fix previous releases). Let us know how the first test went on.

[tgob]

Dear Glenn,

I have tried your experiments today. The output from each is contained in the attached .txt files.

The experiments were completed on just the Master Node.

Notes:

Experiment 1 Step 4. There is an error in the edited source line. I have added in .bashrc_fds.

Experiment 2. The downloaded source file does not have executable permissions. The file was modified with chmod 755 fds_mpi_intel_linux_64ib.

I have printed out the PATH, LD_LIBRARY_PATH and FDSNETWORK variables in each experiment output file.

I understand that previous fds distribution were built against Ethernet OpenMPI. This leads me to ask why fds incorporates the MPIDIST_IB variable in .bashrc, and examines this to set FDSNETWORK as infiniband if “ib” is in MPIDIST_IB?

On my previous fds installs I have compiled OpenMPI to use IB for networking (primarily for the significant reduction in network overhead). This should be largely transparent to fds as SSH (over Ethernet) is used to establish and close the OpenMPI environment across the cluster.

Unless anything is immediately apparent to you from these experiments I figure my next step will be to compile fds 6.5.3 using a GNU compiler and see what happens.

With kindest regards,

Tim

exp1.txt exp3.txt exp2.txt

[tgob]

Oops, I seem to have accidentally closed this issue during my last post. I have reopened it. Silly me. t,

[sbenkorichi]

Tim, You might clicked on "comment and close" button, that's why it got closed. Just hit comment button next time. I think I understand your issue now, Gleen was right, it's about setting your OMPI correctly.

Check what I've done with exp1 1) Downloaded two files. ( put them in home directory)

[salah@jbk27s092 ~]$ mkdir test
[salah@jbk27s092 ~]$ cd test
[salah@jbk27s092 test]$ tar xvf ~/openmpi_1.8.4_linux_64ib.tar.gz

Then it just got extracted.

2) opened my .bashrc file, then modified the bashrc fds line to: source ~/.bashrc_fds ~/test/openmpi_64ib

Here is what I've put in my .bashrc source ~/.bashrc_fds ~/test/openmpi_64ib save it and close it.

[salah@jbk27s092 ~]$ source ~/.bashrc
[salah@jbk27s092 ~]$ mpirun
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    orterun:nothing-to-do
But I couldn't open the help file:
    /shared/openmpi_64ib/share/openmpi/help-orterun.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
[salah@jbk27s092 ~]$ which mpirun
~/test/openmpi_64ib/bin/mpirun

The reason you got that error is because the library are being exported, however it's not linked properly with bashr fds. So, in order to use mpirun, you need to specify it's directory. Next, create any random file, and copy any inputfile to it ( I call it example, and copy box_burn_away, you find it in the example/fires directory. Then, I had to activate the fds.

[salah@jbk27s092 ~]$ sudo chmod +x fds_mpi_intel_linux_64ib 
[sudo] password for salah: 
[salah@jbk27s092 ~]$ cd example/
[salah@jbk27s092 example]$ ls
box_burn_away1.fds
[salah@jbk27s092 example]$ which mpirun
~/test/openmpi_64ib/bin/mpirun
[salah@jbk27s092 example]$ ~/test/openmpi_64ib/bin/mpirun -np 1 /home/salah/fds_mpi_intel_linux_64ib box_burn_away1.fds 
 Mesh      1 is assigned to MPI Process      0
 OpenMP thread   2 of   3 assigned to MPI process      0 of      0
 OpenMP thread   1 of   3 assigned to MPI process      0 of      0
 OpenMP thread   3 of   3 assigned to MPI process      0 of      0
 OpenMP thread   0 of   3 assigned to MPI process      0 of      0
 Completed Initialization Step  1
 Completed Initialization Step  2
 Completed Initialization Step  3
 Completed Initialization Step  4

 Fire Dynamics Simulator

 Current Date     : April  5, 2017  12:48:04
 Version          : FDS 6.5.3
 Revision         : FDS6.5.3-598-geb56ed1
 Revision Date    : Thu Jan 19 16:12:59 2017 -0500
 Compilation Date : Apr 02, 2017  21:18:46

 MPI Enabled; Number of MPI Processes:          1
 OpenMP Enabled; Number of OpenMP Threads:   4

 MPI version: 3.0
 MPI library version: Open MPI v1.8.4, package: Open MPI gforney@burn Distribution, ident: 1.8.4, repo rev: v1.8.3-330-g0344f04, Dec 19, 2014

 Job TITLE        : Test BURN_AWAY feature
 Job ID string    : box_burn_away1

 Time Step:      1, Simulation Time:      0.01 s
 Time Step:      2, Simulation Time:      0.02 s
 Time Step:      3, Simulation Time:      0.03 s
 Time Step:      4, Simulation Time:      0.04 s
 Time Step:      5, Simulation Time:      0.05 s

As you can see, it's working for my case. Please, try to follow the steps I shared here and see if it's working

[tgob]

I am becoming confused.

In Glenn's Experiment 1 he asked for fds to be run using the default NIST FDS_6.5.3 Linux install with the installed openmpi_1.8.4_linux_64ib tar. I have done these experiments and sent you the results. What are these results showing you?

You have run mpirun (your step 2), and then you go on to run fds_mpi_intel_linux_64ib which is Glenn's Experiment 2.

You state that ' The reason you got that error is because the library are being exported, however it's not linked properly with bashr fds. So, in order to use mpirun, you need to specify it's directory.' You appear to correct this 'lack of linking' by specifying full directories for the mpirun command. This makes no sense to me because both mpirun and fds and their associated libraries are in the path variables and are correctly set to the associated environment variables.

I do appreciate your time on this, but we seem to be taking 'shots in the dark' at solutions and introducing more potential problems instead of isolating issues. While I am happy to try stuff (and your suggestions are most welcome) it would be most helpful if I could understand what we expect these experiments to show. Please note that I am not a Linux guru (I do write and maintain extensive program code, but largely in assembler, machine code and a bunch of antiquated languages).

When I am instructed to download and run a file (such as fds_mpi_intel_linux_64ib) and it doesn't work, it takes me some time to figure out that the file needs to have executable permissions assigned. It would be most helpful if your instructions were explicit and targeted at the lowest common denominator (me).

Please take a moment to reflect on what was working for my cluster, what was changed, and the nature of the originally reported issue.

1. My cluster nodes are running Ubuntu 16.04 LTS on Intel multi-core processors with heaps of memory.

2. I have a stable OpenMPI 1.8.4 installation complied using GNU that utilizes Infiniband. It runs with the NIST default Linux FDS 6.5.2 and previous versions. It runs other programs.

3. SSH between nodes and from a node to itself is password-less over 1 GB Ethernet.

4. There is a common 'Projects' directory shared between nodes using NFS over Infiniband.

5. When I install the NIST default Linux 6.5.3, fds fails with reference to directory paths that do not exist, are not part of the OpenMPI or Infiniband installations, and are not referenced in either .bashrc or .bashrc_fds.

6. The error occurs with just a single node powered up.

FDS 6.5.3 must be doing something different from FDS 6.5.2 (and earlier releases) with its initialization and calls to OpenMPI 1.8.4. While this may be a Fortran source code change it may also be due to NIST's Intel compiler.

I await your response.

With kindest regards,

Tim

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Wednesday, 5 April 2017 11:53 p.m. To: firemodels/fds Cc: tgob; State change Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Tim, You might clicked on "comment and close" button, that's why it got closed. Just hit comment button next time. I think I understand your issue now, Gleen was right, it's about setting your OMPI correctly.

Check what I've done with exp1

1. Downloaded two files. ( put them in home directory)

[salah@jbk27s092 ~]$ mkdir test [salah@jbk27s092 ~]$ cd test [salah@jbk27s092 test]$ tar xvf ~/openmpi_1.8.4_linux_64ib.tar.gz

Then it just got extracted.

2. opened my .bashrc file, then modified the bashrc fds line to: source ~/.bashrc_fds ~/test/openmpi_64ib

Here is what I've put in my .bashrc source ~/.bashrc_fds ~/test/openmpi_64ib save it and close it.

[salah@jbk27s092 ~]$ source ~/.bashrc [salah@jbk27s092 ~]$ mpirun

Sorry! You were supposed to get help about: orterun:nothing-to-do But I couldn't open the help file: /shared/openmpi_64ib/share/openmpi/help-orterun.txt: No such file or directory. Sorry!

[salah@jbk27s092 ~]$ which mpirun ~/test/openmpi_64ib/bin/mpirun

The reason you got that error is because the library are being exported, however it's not linked properly with bashr fds. So, in order to use mpirun, you need to specify it's directory. Next, create any random file, and copy any inputfile to it ( I call it example, and copy box_burn_away, you find it in the example/fires directory. Then, I had to activate the fds.

[salah@jbk27s092 ~]$ sudo chmod +x fds_mpi_intel_linux_64ib [sudo] password for salah: [salah@jbk27s092 ~]$ cd example/ [salah@jbk27s092 example]$ ls box_burn_away1.fds [salah@jbk27s092 example]$ which mpirun ~/test/openmpi_64ib/bin/mpirun [salah@jbk27s092 example]$ ~/test/openmpi_64ib/bin/mpirun -np 1 /home/salah/fds_mpi_intel_linux_64ib box_burn_away1.fds Mesh 1 is assigned to MPI Process 0 OpenMP thread 2 of 3 assigned to MPI process 0 of 0 OpenMP thread 1 of 3 assigned to MPI process 0 of 0 OpenMP thread 3 of 3 assigned to MPI process 0 of 0 OpenMP thread 0 of 3 assigned to MPI process 0 of 0 Completed Initialization Step 1 Completed Initialization Step 2 Completed Initialization Step 3 Completed Initialization Step 4

Fire Dynamics Simulator

Current Date : April 5, 2017 12:48:04 Version : FDS 6.5.3 Revision : FDS6.5.3-598-geb56ed1 Revision Date : Thu Jan 19 16:12:59 2017 -0500 Compilation Date : Apr 02, 2017 21:18:46

MPI Enabled; Number of MPI Processes: 1 OpenMP Enabled; Number of OpenMP Threads: 4

MPI version: 3.0 MPI library version: Open MPI v1.8.4, package: Open MPI gforney@burn Distribution, ident: 1.8.4, repo rev: v1.8.3-330-g0344f04, Dec 19, 2014

Job TITLE : Test BURN_AWAY feature Job ID string : box_burn_away1

Time Step: 1, Simulation Time: 0.01 s Time Step: 2, Simulation Time: 0.02 s Time Step: 3, Simulation Time: 0.03 s Time Step: 4, Simulation Time: 0.04 s Time Step: 5, Simulation Time: 0.05 s

As you can see, it's working for my case. Please, try to follow the steps I shared here and see if it's working

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-291837927 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypGRg3bYzekLAY2CmGPlHuEeZQBYrks5rs4CtgaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Tim, That's an error I tested exp2. As for making it excitable , its done even with fds bundle. I will test the other exp (1&3) step by step and share them here. As for mpirun what I meant is when you what's after bashrc fds , and you still exporting its old directory , you end up with such error. So, that's why its better to specify the full path to it. I hope this help now.

[sbenkorichi]

exp1: It received no error. exp3: 1) Downloaded and make the bundle executable. 2) fds installation with openmpi that comes with it.

[salah@jbk27s092 Downloads]$ ./FDS_6.5.3-SMV_6.4.4_linux64.sh 

Installing 64 bit Linux FDS 6.5.3 and Smokeview 6.4.4

Options:
  1) Press <Enter> to begin installation
  2) Type "extract" to copy the installation files to FDS_6.5.3-SMV_6.4.4_linux64.tar.gz
1

FDS install options
  Press 1 to install in /home/salah/FDS/FDS6 [default]
  Press 2 to install in /opt/FDS/FDS6
  Press 3 to install in /usr/local/bin/FDS/FDS6
  Enter a directory path to install elsewhere
1

OpenMPI options
  Press 1 to install OpenMPI manually [default]
     See /home/salah/FDS/FDS6/bin/README.html for details
  Press 2 to use /shared/openmpi_64
1

Installation directory: /home/salah/FDS/FDS6
     OpenMPI directory: to be specified later
Do you wish to proceed with the installation? (yes/no) yes

Installation beginning
Creating directory /home/salah/FDS/FDS6
The installation directory, /home/salah/FDS/FDS6, has been created.
Creating directory /home/salah/FDS/FDS6/Uninstall
The installation directory, /home/salah/FDS/FDS6/Uninstall, has been created.

Copying FDS installation files to /home/salah/FDS/FDS6
Copy complete.

Backing up /home/salah/.bashrc_fds to /home/salah/.bashrc_fds_20170406_143525
Updating .bashrc_fds

Backing up /home/salah/.bashrc to /home/salah/.bashrc_20170406_143525
Updating .bashrc

*** Log out and log back in so changes will take effect.

Installation complete.
[salah@jbk27s092 Downloads]$ cd ~/FDS/FDS6/bin
[salah@jbk27s092 bin]$ gunzip openmpi_1.8.4_linux_64.tar.gz
[salah@jbk27s092 bin]$ tar -xvf openmpi_1.8.4_linux_64.tar

Open .bashrc file and change source ~/.bashrc_fds to source ~/.bashrc_fds $MPIDIST_FDS log out and log in again, or do source ~/.bashrc

Here, I tested it on a simple case.

[salah@jbk27s092 ~]$ cd example
[salah@jbk27s092 example]$ which mpirun
~/FDS/FDS6/bin/openmpi_64/bin/mpirun
[salah@jbk27s092 example]$ ~/FDS/FDS6/bin/openmpi_64/bin/mpirun -np 1 ~/FDS/FDS6/bin/fds box_burn_away1.fds
 Mesh      1 is assigned to MPI Process      0
 OpenMP thread   1 of   3 assigned to MPI process      0 of      0
 OpenMP thread   2 of   3 assigned to MPI process      0 of      0
 OpenMP thread   0 of   3 assigned to MPI process      0 of      0
 OpenMP thread   3 of   3 assigned to MPI process      0 of      0
 Completed Initialization Step  1
 Completed Initialization Step  2
 Completed Initialization Step  3
 Completed Initialization Step  4

 Fire Dynamics Simulator

 Current Date     : April  6, 2017  14:47:29
 Version          : FDS 6.5.3
 Revision         : FDS6.5.3-598-geb56ed1
 Revision Date    : Thu Jan 19 16:12:59 2017 -0500
 Compilation Date : Jan 22, 2017  18:04:30

 MPI Enabled; Number of MPI Processes:          1
 OpenMP Enabled; Number of OpenMP Threads:   4

 MPI version: 3.0
 MPI library version: Open MPI v1.8.4, package: Open MPI gforney@burn Distribution, ident: 1.8.4, repo rev: v1.8.3-330-g0344f04, Dec 19, 2014

 Job TITLE        : Test BURN_AWAY feature
 Job ID string    : box_burn_away1

 Time Step:      1, Simulation Time:      0.01 s
 Time Step:      2, Simulation Time:      0.02 s
 Time Step:      3, Simulation Time:      0.03 s
 Time Step:      4, Simulation Time:      0.04 s
 Time Step:      5, Simulation Time:      0.05 s

I can't trace what exactly has changed in this version that made it not working with openmpi that you've already installed. In your exp3 file,

]0;ob1@Master: ~ob1@Master:~$ mpirun
mpirun: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory

I would request you try to follow the steps as I did above, by specifying the full path to it.

[tgob]

Salah,

I have attempted to duplicate your results.

Exp 1.

Remove all previous OpenMPI and FDS installs.

Install openmpi_1.8.4_linux_64ib into ~/test.

Adjust PATH and LD_LIBRARY_PATH variables to incorporate ~/test/openpmi_64ib/bin and ~/test/openmpi_64ib/lib.

Log out and log in.

Before proceeding to the fds installation I decided to test the OpenMPI install.

:/~ mpirun

mpirun: error while loading shared libraries: libtorque.so.2: cannot open shared object file: no such file or directory

For completeness I executed the command with the full Openmpi path with the same error message.

libtorque.so.2 is associated with the Torque batch resource manager. This isn't distributed with Ubuntu but it is distributed with CentOS and RHEL.

Exp 2

There is no point in proceeding with Experiment 2 as openmpi_1.8.4_linux_64ib will not work with Ubuntu.

Exp 3.

Install the official FDS_6.5.3-SMV_6.4.4_linux64.sh and the bundled OpenMPI (per your instructions below).

Adjust the path variables to point to the OpenMPI installation (in ~/FDS/FDS6/bin/openmpi_64/bin)

Again, test the OpenMPI installation before fds:

:/~ mpirun

We are back to an error message concerning libcrypto.so.10 - no such file or directory.

libcrypto.so.10 is associated with OpenSSL - a general purpose cryptography library distributed with CentOS and RHEL, but not with Ubuntu.

So the problems that I am having with OpenMPI appear to relate to the OS differences during compilation (and this possibly extends to the NIST bundled FDS as well).

Interestingly, the OpenMPI distribution for Ubuntu referred to in the NIST Wiki, https://github.com/firemodels/fds/wiki/Installing-Open-MPI-on-a-Linux-Cluster, works just fine, but I doubt that it will utilize my clusters Infiniband resources.

I look forward to your further consideration of this matter.

Tim

[sbenkorichi]

Tim, The above testing, It was under Centos 7. And I believe also NIST are using it Centos 6. I've tested it now on my laptop (with linux mint, i.e ubuntu). I've received similar error as yours .

salah@salah ~/example $ which mpirun
/home/salah/test/openmpi_64ib/bin/mpirun
salah@salah ~/example $ /home/salah/test/openmpi_64ib/bin/mpirun -np 1 /home/salah/fds_mpi_intel_linux_64ib box_burn_away1.fds 
/home/salah/test/openmpi_64ib/bin/mpirun: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory

I tried also to use the openmpi that comes with the bundle, it throws the same error. I would let @gforney if he has other suggestion.

[sbenkorichi]

Tim, This is how I fixed that error.

sudo apt-get update
sudo apt-get install libssl1.0.0 libssl-dev

then create a link to fix the naming of the file.

cd /lib/x86_64-linux-gnu
sudo ln -s libssl.so.1.0.0 libssl.so.10
sudo ln -s libcrypto.so.1.0.0 libcrypto.so.10

test it now

salah@salah ~/example $ /home/salah/test/openmpi_64ib/bin/mpirun -np 1 /home/salah/fds_mpi_intel_linux_64ib box_burn_away1.fds 
 Mesh      1 is assigned to MPI Process      0
 OpenMP thread   3 of   3 assigned to MPI process      0 of      0
 OpenMP thread   2 of   3 assigned to MPI process      0 of      0
 OpenMP thread   0 of   3 assigned to MPI process      0 of      0
 OpenMP thread   1 of   3 assigned to MPI process      0 of      0
 Completed Initialization Step  1
 Completed Initialization Step  2
 Completed Initialization Step  3
 Completed Initialization Step  4

 Fire Dynamics Simulator

 Current Date     : April  7, 2017  10:22:50
 Version          : FDS 6.5.3
 Revision         : FDS6.5.3-598-geb56ed1
 Revision Date    : Thu Jan 19 16:12:59 2017 -0500
 Compilation Date : Apr 02, 2017  21:18:46

 MPI Enabled; Number of MPI Processes:          1
 OpenMP Enabled; Number of OpenMP Threads:   4

 MPI version: 3.0
 MPI library version: Open MPI v1.8.4, package: Open MPI gforney@burn Distribution, ident: 1.8.4, repo rev: v1.8.3-330-g0344f04, Dec 19, 2014

 Job TITLE        : Test BURN_AWAY feature
 Job ID string    : box_burn_away1

 Time Step:      1, Simulation Time:      0.01 s
 Time Step:      2, Simulation Time:      0.02 s
 Time Step:      3, Simulation Time:      0.03 s
 Time Step:      4, Simulation Time:      0.04 s
 Time Step:      5, Simulation Time:      0.05 s
 Time Step:      6, Simulation Time:      0.06 s

All the 3 experiments work now. Try to test it on your cluster to see if this solves it or not.

[tgob]

Thanks Salah,

you appear to be loading the Open SSL libraries into Ubuntu.

I'm tied up with other stuff for a few days. I'll get back to you shortly.

t.

[tgob]

Progress...

The key to this issue appears to be in Glenn's last post: Note all previous fds' that we have distributed were built against an ethernet (not infiniband) version of Openmpi.

This implies that the current distribution was built against an Infiniband (IB) version of OpenMPI: an implication which is supported by the contents of the default FDS6.5.3 installed files. I missed the importance of this statement in my earlier reads but, just like the FDS manuals, every sentence is relevant and may be critical.

The OpenMPI installation on my Linux cluster implements Ethernet over IB. The IB communication is therefore effectively transparent to the application so FDS 6.5.2 just sees Ethernet (albeit running at IB speed). However the default FDS6.5.3 bundle (and the alternative FDS executable in Glenn's post) incorporates Intel IB components which appear to be the cause of the problems I have been experiencing.

In order to overcome both the IB conflicts, the OS issues (Ubuntu as opposed to CentOS Linux) and FDS compiler issues (GNU as opposed to Intel) I compiled the mpi_gnu_linux_64 FDS source code using GNU compilers this evening.

Whoot!

            FDS now runs on an individual node without error.

            My original OpenMPI 1.8.4 install (continues to) runs without error.

            FDS runs under OpenMPI.

            FDS test models run under OpenMPI  using multiple nodes using Infiniband.

            There are no OpenSSL library errors.

FDS terminal output reports an IEEE underflow flag for the models that I have tested to date. This error is readily suppressed through compiler options but first I need to I need to verify the FDS 6.5.3 install. SmokeView visualization indicates that this error is not affecting simulated fire dynamics. If this is actually a problem (which I doubt) then I shall consider a Ubuntu kernel upgrade as a first step although this will be time-consuming.

At the moment I have been working on just two nodes of my cluster. I'll report back once I have completed verification across all cluster nodes.

t.

PS. Salah's post of 7 April 2017 does fix the OpenSSL issue associated with OS issues affecting OpenMPI installations. However it does not resolve the underlying default FDS installation IB conflicts.

[tgob]

Further to my last post...

Verification was completed yesterday. No problems with this however there are a few problems with the install.

With fully assigned processes on the Master Node I am getting SIGSEGV errors associated with memory. Note that stack and memlock limits are unlimited for all logins and each node has 16 GB of RAM. I'm guessing that this is something to do with FDS or the GNU compiler as OpenMPI 1.8.4 has been running stable with FDS 6.5.2 for thousands of core hours on numerous models.

Run time has increased appreciably over the previous Nist bundled Intel FDS 6.5.2. Using standard models I'm getting run a run time increase of around 20% for a given number of processes. I suspect this is due to compiler differences as the NIST Intel FDS 6.5.3 bundle runs slightly faster (about 5%) than NIST Intel 6.5.2 bundle on a Windows platform.

I have a request for you. Can you make available an Intel compiled version of FDS 6.5.3 for Linux with OpenMPI support and without Infiniband? This would be essentially the same build as the NIST bundled FDS 6.5.2 for Linux with OpenMPI.

With kindest regards,

Tim

[sbenkorichi]

Tim, Some users faced this issue of memory under linux with this release. This would be fixed in the next release. However, you one way you solve this issue, is by setting the stack memory to ulimted put this command in your terminal before you run fds

ulimit -s unlimited
fds case.fds

Another alternative way, is to edit the /etc/bash.bashrc/ Here is a link for it http://disbauxes.upc.es/code/applied-physics-department/changing-the-stacksize-ulimit-s-limit-whenever-running-mpirun/

Of course INTEL is going to be more faster than GNU. As for your request, maybe Gleen could reply to you on that.

Regards, Salah

[gforney]

We did not intend to distribute an "infiniband" fds in the last bundle. If so then this is an error. I'll take a look. We are looking at other options. Using openmpi 2.1.0 and using Intel mpi.

On Apr 14, 2017 5:10 AM, "Salah Benkorichi" notifications@github.com wrote:

Tim, Some users faced this issue of memory under linux with this release. This would be fixed in the next release. However, you one way you solve this issue, is by setting the stack memory to ulimted put this command in your terminal before you run fds

ulimit -s unlimited fds case.fds

Another alternative way, is to edit the /etc/bash.bashrc/ Here is a link for it http://disbauxes.upc.es/code/applied-physics-department/ changing-the-stacksize-ulimit-s-limit-whenever-running-mpirun/

Of course INTEL is going to be more faster than GNU. As for your request, maybe Gleen could reply to you on that.

Regards, Salah

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-294121763, or mute the thread https://github.com/notifications/unsubscribe-auth/AL1BRhwXqEkCeV9_-MryKOiWbEQeIO3vks5rvzf6gaJpZM4MwWP5 .

[tgob]

Dear Salah,

The issue is not with stack or memory lock bounds (I did explain this). I have set memlock hard and soft to unlimited in /etc/security/limits.conf. Further .bashrc includes ulimit -s unlimited. The command ulimit -a from terminal reports unlimited memory and stack allocation.

On speed I am not convinced that Intel compilers will necessarily produce significant decreases in processing time. There are many benchmark tests available on the www that show that actual processing speed is highly dependant on the nature of the test. While Intel market their compilers as faster than gnu this is application and optimization dependant.

Dear Glenn,

I have inferred that the latest NIST bundle incorporated IB from your earlier statement where your referred to previous releases not having IB components. Maybe I have read too much into your statement?

There is another aspect of FDS that you may be able to explain to me. The compiled code seems to be looking at a number of environment variables including MPIDIST, FDSNETWORK. Is there a definitive list of these complete with default and possible values? I can see from .bashrc and and .bashrc_fds that FDSNETWORK is set to either "" or infiniband depending on the whether or not MPIDIST terminates in 'ib'.

I'll be trying some more stuff over the next few days including a GNU upgrade, a Ubuntu kernel upgrade, and OpenMPI 2.1.0.

With kindest regards,

Tim

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Friday, 14 April 2017 9:10 p.m. To: firemodels/fds Cc: tgob; State change Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Tim, Some users faced this issue of memory under linux with this release. This would be fixed in the next release. However, you one way you solve this issue, is by setting the stack memory to ulimted put this command in your terminal before you run fds

ulimit -s unlimited fds case.fds

Another alternative way, is to edit the /etc/bash.bashrc/ Here is a link for it http://disbauxes.upc.es/code/applied-physics-department/changing-the-stacksize-ulimit-s-limit-whenever-running-mpirun/

Of course INTEL is going to be more faster than GNU. As for your request, maybe Gleen could reply to you on that.

Regards, Salah

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-294121763 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypO3BrLkKVAWMk2_zhdtdONnhkqxoks5rvzf6gaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Tim, Can you run any simple example with what I've provided above, and share the file if it's different than the examples provided with the bundle, with the outcome result. I'm eager to see the outcome of it.

[tgob]

Salah, I don't understand what you're asking for. Let me clarify my current installation:

OS: Ubuntu 16.04 LTS.

OFED: Mellanox compiled under GNU to produce Infiniband over Ethernet on dedicated IP addresses.

OpenMPI: 1.8.4 compiled under GNU.

NFS: ~/Project directory file share from Master Node (this is where fds models are processed).

SSH: password-less between all nodes with common user name and admin privileges.

FDS: 6.5.3 compiled using GNU from GitHub clone. mpi_gnu_linux_64 source with default compiler options.

I have run the entire fds verification suite (the models are contained in the Examples directory and listed in Appendix B of the FDS Technical Reference Guide, Volume 2, Verification) and these all run with outputs within NIST tolerances.

I have run my own test models (a simple multi-mesh compartment fire) to examine processing speed and performance with different OpenMP and OpenMPI allocations. FDS fails intermittently with SIGSEGV errors when processes (OMP and/or OpenMPI) use all available cores. The failures occur at the beginning of a simulation after the four initialization steps.

Would you please clarify what you want me to run, under what environment, and what you expect this to demonstrate. When a model crashes there is no useful fire dynamics or error output other that the cryptographic terminal SIGSEGV error report. When a model runs it runs to completion.

With kindest regards,

Tim

From: Salah Benkorichi [mailto:notifications@github.com] Sent: Friday, 14 April 2017 10:26 p.m. To: firemodels/fds Cc: tgob; State change Subject: Re: [firemodels/fds] FDS 6.5.3 Upgrade Issue (#4904)

Tim, Can you run any simple example with what I've provided above, and share the file if it's different than the examples provided with the bundle, with the outcome result. I'm eager to see the outcome of it.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/4904#issuecomment-294132243 , or mute the thread https://github.com/notifications/unsubscribe-auth/ATPypMqnjUNp3-o8oxrvja10zq7dyq1Jks5rv0m0gaJpZM4MwWP5 .Image removed by sender.

[sbenkorichi]

Tim, We had previously another user who faced similar issue. He could run the case under the local machine, however, if he would start the calculation from one machine and assign the meshes to be processed under another node, it would still result in the same error of segmentation fault. It was solved as I shared with you above by adding

ulimit -s unlimited
ulimit -v unlimited

to/etc/bash.bashrc for the each node. However, if this issue still persists, then you might be running a very large mesh, that needs more than 16 GB that you have. In this case, you might think of dividing your mesh.