OpenMP implementation

[gforney]

Hi, as Lukas mentioned I will be working full time on OpenMP in FDS for the next six
months at the Jülich Supercomputing Centre.

I am trying to get the verifications to run as I write this. I had noticed that, as
you mentioned, there are some odd errors for certain parts of the OpenMP version. Therefore
my plan was to initially focus on a single solver and see that I get it running, verified
and get some figures regarding performance and scalability on the various HPC system
we have access to.

If we could then set up some sort of SVN-access, so that I can participate in the continuous
integration testing, that would be wonderful.
I'd then try to proceed through the various solvers, so that hopefully by the end of
my six months we have verified OpenMP parallelism in most critical parts of FDS.

I'll keep you posted on my progress!

Cheers,
Daniel

Original issue reported on code.google.com by randy.mcdermott on 2013-11-15 14:45:49

[gforney]

Daniel,

The first step that Kevin suggested was to run the verification suite and compile the
FDS Verification Guide yourself.  This means running the Matlab scripts to generate
the figures and running LaTeX to compile the guide.  Instructions are given here:

https://code.google.com/p/fds-smv/wiki/FDS_Verification_Process

Do not worry about OpenMP just yet.  Just get all these cases to run on your cluster.
 Present the compiled guide to us and voila! we will grant you commit privileges.

As you start merging code in, we will want to work closely with Kris Overholt to make
sure Firebot does all the proper testing of the OpenMP version and gets identical results
to the serial version.

Cheers,
Randy

Original issue reported on code.google.com by randy.mcdermott on 2013-11-15 14:54:18

[gforney]

If any of the scripts used to run the verification cases doesn't work, let us know.
We are trying to get these automated scripts to run on any linux cluster, not just
ours.

Original issue reported on code.google.com by mcgratta on 2013-11-15 15:12:51

[gforney]

Issue 2031 has been merged into this issue.

Original issue reported on code.google.com by randy.mcdermott on 2013-11-15 15:16:24

[gforney]

I should also mention that, for the automatic Firebot build script, we currently have
the Intel Inspector for performing OpenMP thread checking to ensure that things are
running as expected. This is where I left off OpenMP development a few months ago,
because I could not get even a simple parallelized loop to work properly with the Intel
Inspector tool (i.e., it would give dozens of thread errors).

I have all of the basic procedures in place for Firebot to build the OpenMP version,
in fact Firebot currently compiles the release version of OpenMP on Linux, but the
remaining test procedures are commented out for now.

So, after you get things working and get commit access, I want to make sure that we
add in the OpenMP version of FDS to the nightly testing process so that you can work
incrementally on the OpenMP version.

Original issue reported on code.google.com by koverholt on 2013-11-15 18:04:01

[gforney]

Ok, I have been able to run all the serial and MPI verification services both on my
workstation and our cluster.

The Smokeview rendering also appear to be working but the matlab script runs into trouble:

Printing plot 273...
dataplot completed successfully!
Error: Size of predicted metric is zero for scatterplot Relative Error. Skipping scatterplot.
Error using  .* 
Matrix dimensions must agree.

Error in scatplot (line 249)
        M_bar = sum(log(Predicted_Values).*weight)/sum(weight);

Error in FDS_verification_script (line 61)
scatplot(saved_data, drange, ...

Can you maybe give me some pointers as to what the issue here might be?
The rest of the script executes fine and the PDF is generated correctly (but of course
is missing figures).

Original issue reported on code.google.com by daniel.haarhoff on 2013-11-19 12:11:33

[gforney]

OK, I will take a look at this. Perhaps there is some small difference in our versions
of Matlab. 

Original issue reported on code.google.com by mcgratta on 2013-11-19 13:58:29

[gforney]

Can you post any errors that occurred during the Matlab plotting? The error you posted
indicates that there might have been some trouble earlier in the dataplot script for
certain cases/rows. You might see error messages such as "Error: File [filename] does
not exist. Skipping case." or "Error: Problem with dataplot row [...]".

Original issue reported on code.google.com by koverholt on 2013-11-19 14:36:54

[gforney]

Sorry, forgot to include that line. Right at the start I get:

Error: File ../../Verification/Radiation/radiation_box_100___50_devc.csv does not exist.
Skipping case.

I logged the output of the Run_FDS_Cases.sh to file. There the only reference to the
above case is:

Running fds_intel_linux_64 radiation_box_100___50.fds

The radiation_box_100___50.out tells me that the case ran successfully.

The strange thing is that radiation_box_100___50_devc.csv exists right where all ther
other csv files are and I can open and read it like all the others.

Ideas?

Original issue reported on code.google.com by daniel.haarhoff on 2013-11-19 15:14:38

[gforney]

These issues are difficult to troubleshoot over the Issue Tracker because there are
many variables that could be affecting the cases and scripts. We run in to these issues
on our own cluster sometimes, so it would be helpful to see if there is something we
can do better in the scripts.

Is it possible for me to get an account on your cluster so we can have a closer look?

Original issue reported on code.google.com by koverholt on 2013-11-19 15:21:11

[gforney]

Check the file properties:

-rw-r--r-- 1 mcgratta div866   939 Jun  7 11:53 radiation_box_100___50_devc.csv

Check ownership of directory. Check if this file is different than the others. 

You can also try just running the script called 

FDS-SMV/Utilities/Matlab/scripts/radiation_box.m

The Error statement comes from here. Perhaps it is a problem with the relative location
of the directories.

Original issue reported on code.google.com by mcgratta on 2013-11-19 15:26:19

[gforney]

Thanks. Got it to run after I changed the file properties.

Next problem was that a set of the csv files had Null characters in certain rows. Fixed
that with sed, but any pointers as to how to avoid this problem in the future would
be appreciated.

Next problem is this:

Error: Problem with dataplot row 40 (energy_budget_adiabatic_walls); check syntax of
FDS/model results (d2) columns. Skipping case.
Warning: Ignoring extra legend entries. 
> In legend at 286
  In dataplot at 391
  In FDS_verification_script at 60

I removed the trycatch and got this error:
Subscript indices must either be real positive integers or logicals.

Error in dataplot (line 301)
                    Save_Predicted_Metric(i,j,1) = M(indices(end),d2_Dep_Col)-d2_Initial_Value;

Error in FDS_verification_script (line 60)
[saved_data,drange] = dataplot(Dataplot_Inputs_File, Working_Dir, Manuals_Dir);

I checked the energy_budget_adiabatic_walls_hrr.csv but couldn't make out any irregularities.
Am I looking in the wrong place?

I'll ask about access to the cluster tomorrow. I've started extending the runfds scripts
to take our moab queue manager into account. Is that something you could see yourself
adding to the trunk at some stage? Then I'd make sure to expand my brutish hackery
into something pretty.

Original issue reported on code.google.com by daniel.haarhoff on 2013-11-20 13:26:43

[gforney]

Thanks for this. I know it's a bit painful, but our hope is that we can make this entire
process more robust. There are so many subtle differences in the OS, versions of Matlab,
and so on.

First, in what way did you have to change the file properties?

Can you post one of the files with Null characters? We use NaN in the .csv files and
Matlab recognizes it.

As for the d2 columns -- d2 refers to the dependent variable of the FDS results. d1
refers to the experiment. Check the .csv file headers and compare to the character
strings in the row 40 of the dataplot input file.

Original issue reported on code.google.com by mcgratta on 2013-11-20 13:43:56

[gforney]

No worries. Thanks for your help trudging through this.

I finally got everything to work (see attachment).

The energy_budget_adiabatic_walls_hrr.csv didn't have any header. So I copied the header
of energy_budget_adiabatic_two_fuels_hrr.csv over by hand and that fixed it.

Regarding the nullchars: I don't have any of those files anymore. In the three cases
which I investigated the null chars had replaced the leading space character in a row
of data. I found literal NaN in other csv files.
This leads me to believe that the problems lies somewhere else.

PS: Do you have a quick method for preventing the matlab script from popping up all
the figure windows? I prevents me from getting any work done while it runs.

Original issue reported on code.google.com by daniel.haarhoff on 2013-11-20 14:17:15

---

• _Attachment: FDS_Verification_Guide.pdf_

[gforney]

Wow -- that was fast. Great work. 

I cannot understand why the file called energy_budget_adiabatic_walls_hrr.csv does
not have a header. This file is created by FDS and should have the same format as all
other _hrr.csv files. 

As for figures, there is a file called FDS-SMV/Utilities/Matlab/scripts/plot_style.m.
This script is called by all other scripts and contains the basic style parameters
for the plots. There is a flag:

Figure_Visibility = 'on';

Try turning this to 'off'. When we run these scripts at night on our linux cluster,
the linux version of Matlab does not allow the figures to pop up because we run the
script via remote login. 

I have added you to the list of Project Committers. You may need to check out the repository
again using your username rather than anonymous. I don't know how to convert an existing
check out from anonymous to one with commit privilege. Once you get this all set up,
we can talk more about how to gradually build in the OpenMP functionality.

Original issue reported on code.google.com by mcgratta on 2013-11-20 15:10:39

[gforney]

Can you tell me which compiler versions you use? I'd like to make doubly sure that this
doesn't cause a problem once I submit.

Original issue reported on code.google.com by daniel.haarhoff on 2013-12-16 10:32:56

[gforney]

We have a very recent version of the Intel Fortran compiler, maybe 14, but I will cc
Glenn to be sure. In any case, if you have only "touched" OpenMP lines in the code,
and you do an Update before committing, you will not cause any trouble with our daily
builds. I do not believe that we have included the OpenMP version in our daily compiling
checks.

Original issue reported on code.google.com by mcgratta on 2013-12-16 10:37:52

[gforney]

Kevin is correct, we are using version 14, I'll give you the exact version when I get
into work.

Original issue reported on code.google.com by gforney on 2013-12-16 12:26:48

[gforney]

on our system at NIST, ifort -v reports
14.0.0

Original issue reported on code.google.com by gforney on 2013-12-16 14:07:00

[gforney]

The nightly build script compiles the debug and release versions of the OpenMP target,
but the more thorough testing is currently disabled until we need it. So, no FDS cases
are run, but it would catch any compiler errors or warnings during the build stage.

Original issue reported on code.google.com by koverholt on 2013-12-16 14:32:22

[gforney]

Hi Daniel,

I just wanted to check in to see how things were progressing with the OpenMP version.
In FDS, we like to work with and commit small, incremental changes in accordance with
the philosophy of continuous integration (http://en.wikipedia.org/wiki/Continuous_integration#Everyone_commits_to_the_baseline_every_day).
Therefore, it would be great if you could submit a small change that parallelizes only
a few loops in the code. That way, we could make sure that 1) all of the verification
cases work with a simple implementation of OpenMP, and 2) the Intel Inspector runs
successfully with thread checking and doesn't report any errors.

When you have a chance, and as you are working on implementing OpenMP, you can go ahead
and commit a small change to the above effect. Then, we can enable the OpenMP version
in the nightly Firebot testing process to make sure things are proceeding smoothly.
Then, as you continue to implement OpenMP, we can be sure that things are working correctly
and slowly build in the changes.

Let us know if you run into any issues or if we can help in any way.

Original issue reported on code.google.com by koverholt on 2014-01-07 12:47:50

[gforney]

Hi, I was having trouble with some numerical instabilities in one of
the verification cases. But I think I've got it located and should
have a commit to velo.f90 for you tomorrow. One for divg.f90 should be
close to follow.

Should I also submit a commit to the makefile to reflect the inclusion
of the relevant OpenMP parts?

Original issue reported on code.google.com by daniel.haarhoff on 2014-01-08 20:43:11

[gforney]

Sounds great, thanks. Yes, commit any corresponding changes to the makefile as well.

Original issue reported on code.google.com by koverholt on 2014-01-08 20:57:00

[gforney]

Quick update. While I have fixed the numerical instabilities I am having trouble verifying
my changes.  A handfull of cases fail to meet the tolerances specified.

Once I have verified code, I will commit. In the mean time I have begun work on the
radiation and wall routines.

Original issue reported on code.google.com by daniel.haarhoff on 2014-01-14 14:26:34

[gforney]

Dear Daniel,

I worry about these instabilities and also that when you speak of "your changes" to
velo, say, that your first commit may be changes to every loop in velo (or a good number
of them).  Please, just commit a change to a single loop in velo first.  Then let our
firebot run on it.  We need to proceed very slowly here.

Thank you.

Randy

Original issue reported on code.google.com by randy.mcdermott on 2014-01-14 14:50:13

[gforney]

No Problem, I just committed the parallelisation of a single loop in divg.
The other commit from me activates OpenMP in the makefile for divg and func(requireded
to determine available threads IIRC).

This change verifies without any complaints for me.

Please let me know when and if I should continue submitting the other loops.

Original issue reported on code.google.com by daniel.haarhoff on 2014-01-15 08:28:30

[gforney]

Thanks, Daniel.

Our previous collaborator who worked with the OpenMP version of FDS used Intel Inspector
to do thread checking on the executable as a way to verify the operation of OpenMP.

I tried using the Intel Inspector to thread check the new OpenMP changes that you committed.
I ran the Intel Inspector using the fds_openmp_intel_linux_64_db target on the case
FDS-SMV/Verification/Timing_Benchmarks/bench2.fds. I get the following errors from
the Intel Inspector tool (see the attached text file for more info).

[kjo@blaze Scripts]$ ./inspect_openmp.sh 
  8 threads available
Warning: One or more threads in the application accessed the stack of another thread.
This may indicate one or more bugs in your application. Setting the Inspector to detect
data races on stack accesses and running another analysis may help you locate these
and other bugs.

 Fire Dynamics Simulator

 Compilation Date : Wed, 15 Jan 2014
 Current Date     : January 15, 2014  15:09:37

 Version: FDS 6.0.2; MPI Disabled; OpenMP Enabled
 Number of available threads:   8

 SVN Revision No. : 18052

 Job TITLE        : plume fire
 Job ID string    : bench2

 Time Step:      1,    Simulation Time:      0.04 s
 Time Step:      2,    Simulation Time:      0.09 s
 Time Step:      3,    Simulation Time:      0.13 s
 Time Step:      4,    Simulation Time:      0.18 s
 Time Step:      5,    Simulation Time:      0.22 s

STOP: FDS completed successfully

12 new problem(s) found 
    12 Data race problem(s) detected 

Do you have any insight about these errors and why the data race conditions are being
reported for such a simple loop? This is the point where I had trouble with OpenMP
development about a year ago.

Original issue reported on code.google.com by koverholt on 2014-01-15 20:55:59

---

• Attachment: out.txt

[gforney]

Hi, can you maybe send me your script so that I can reproduce the report?

I'm also getting data-races, but a very different report (no less cryptic, though).

Original issue reported on code.google.com by daniel.haarhoff on 2014-01-16 13:32:10

[gforney]

Daniel:

The scripts are located in the repository at FDS-SMV/Utilities/Scripts/inspect_openmp.sh
and FDS-SMV/Utilities/Scripts/inspect_report.sh.

Original issue reported on code.google.com by koverholt on 2014-01-16 14:12:10

[gforney]

Ok, looks like the data races were false positives due to the compiler flags set in
_db.

I've created a new target named 'inspect' and modified the inspection script a little.
With the new binary and script no data races are detected.

I have just committed the changes so that you can also check for yourself.

The following warning persists. A ti3 collect with scope=extreme crashes during the
run. I am looking into this problem. But since the warning is outside the main loop
I would assume that the issue lies outside of my changes to divg.

Would you like me to continue committing further OpenMP instrumentation? I'll run both
verification and inspection prior to my commits.

Original issue reported on code.google.com by daniel.haarhoff on 2014-01-21 09:22:18

[gforney]

Daniel:

This is great, thanks for the update.

I am out of the office until next week. Now that the inspector is running cleanly,
I would say that you keep slowly adding/committing OpenMP directives and running the
Inspector and verification suite. When I get back to the office, I will enable the
proper checking in the Firebot testing process and catch up with you then. Nice work!

Original issue reported on code.google.com by koverholt on 2014-01-21 15:46:30

[gforney]

Daniel:

We got some odd errors from your latest commit for the MPI version of FDS (SVN 18118),
see the following and let us know what you think:

Errors from Stage 4b - Compile FDS release:
Building mpi_intel_linux_64ib
mpif77 -c -m64 -O2 -vec_report0 -ipo ../../FDS_Source/prec.f90
mpif77 -c -m64 -O2 -vec_report0 -ipo ../../FDS_Source/mpip.f90
mpif77 -c -m64 -O2 -vec_report0 -ipo ../../FDS_Source/cons.f90
mpif77 -c -m64 -O2 -vec_report0 -ipo ../../FDS_Source/devc.f90
mpif77 -c -m64 -O2 -vec_report0 -ipo ../../FDS_Source/type.f90
mpif77 -c -m64 -O2 -vec_report0 -ipo ../../FDS_Source/pois.f90

fortcom: Severe: **Internal compiler error: internal abort** Please report this error
along with the circumstances in which it occurred in a Software Problem Report.  Note:
File and line given may not be explicit cause of this error.

ifort: error #10014: problem during multi-file optimization compilation (code 3)
make: *** [mpi_intel_linux_64ib] Error 3

Stage 4a warnings:
Fatal compilation error: Out of memory asking for 140733826068480.
ipo-4: error #11005: multi-object compilation 2 returned error status 1
Fatal compilation error: Out of memory asking for 5827059712.

Stage 4a warnings:
Fatal compilation error: Out of memory asking for 140733826068480.
ipo-4: error #11005: multi-object compilation 2 returned error status 1
Fatal compilation error: Out of memory asking for 5827059712.
ipo-4: error #11005: multi-object compilation 4 returned error status 1
ifort: error #10014: problem during multi-file optimization compilation (code 1)

Original issue reported on code.google.com by koverholt on 2014-01-22 09:15:52

[gforney]

I'm sorry but I can't reproduce that error.
Could you maybe attempt a manual compile, maybe on a different machine? The error suggests
that the compiler ran out of RAM.

Do the other mpi targets compile?

Original issue reported on code.google.com by daniel.haarhoff on 2014-01-22 10:25:12

[gforney]

Thanks for checking. We'll do some more testing on our end. 

Original issue reported on code.google.com by koverholt on 2014-01-22 10:27:10

[gforney]

Different question: Are the indicies II and IIG etc in the wall loops unique to each
WC?

Most wall loops have the following construct:
WALL_LOOP: DO IW=1,N_EXTERNAL_WALL_CELLS+N_INTERNAL_WALL_CELLS
   WC=>WALL(IW)
   II  = WC%ONE_D%II
   JJ  = WC%ONE_D%JJ
   KK  = WC%ONE_D%KK
   IOR = WC%ONE_D%IOR
   IIG = WC%ONE_D%IIG
   JJG = WC%ONE_D%JJG
   KKG = WC%ONE_D%KKG

Ideally I would like to parallelise the wall loop. I could easily do this if I can
be certain that the WC%ONE_D%II etc. are unique to that cell. Otherwise the assignments
like MU(II,JJ, KK) = FOO(BAR) could lead to a data race.

Do you know how these indices work, or can you tell me what they signify?

Original issue reported on code.google.com by daniel.haarhoff on 2014-01-22 11:34:44

[gforney]

II, JJ, and KK are the cell indices on the solid side of the wall interface. IIG, JJG,
and KKG are the cell indices on the "gas" side of the interface. IOR is the orientation
of the interface, pointing towards the gas side. These values are unique to each WC.
For a 3D mesh, for example one that is 64 by 32 by 16, the first gas phase cell at
the lower left side of the mesh is (1,1,1). If there is a wall on the left side of
the mesh, the II,JJ,KK of the first wall cell would be (0,1,1). The IIG,JJG,KKG of
this same wall cell is (1,1,1). IOR would be 1. If there is a wall on the right side,
the first wall cell would have II,JJ,KK of (65,1,1) and IIG,JJG,KKG of (64,1,1). IOR
would be -1.

Original issue reported on code.google.com by mcgratta on 2014-01-22 13:39:51

[gforney]

Thanks! That clears stuff up.

Now the thing is that the intel inspector deems some of the operations on arrays that
use the above mentioned indices as data races.
With the information you gave me I would deem them to be safe.

Should I create a gold standard or some sort of ignore list and add it to the script?
Or is it better to keep receiving warnings and let us humans decide whether they are
actual data races each time?

Original issue reported on code.google.com by daniel.haarhoff on 2014-01-24 09:58:03

[gforney]

In my experience, I would say it is always a good idea to take compiler warnings seriously.
We can run FDS on the major computing platforms because we adhere strictly to the Fortran
2003 standard. Our compilers have not yet become fully compliant with Fortran 2008,
at least the last time I checked. 

Anyway, I do not recommend that we ignore these warnings -- there might be more to
it than we understand. Pointers in Fortran can be tricky, and I do not completely understand
the hazards they present when optimizing compilation. I suggest that we not try to
parallelize the wall cell loops now. Let's focus on the 3D loops over the entire domain
in divg.f90, mass.f90, velo.f90, and whatever else is taking the most CPU time. The
wall cell loops are still less than 10% of the total CPU time. As we get more experienced
with OpenMP, we might better understand what the warnings mean.

Original issue reported on code.google.com by mcgratta on 2014-01-24 13:11:40

[gforney]

I think the cause of the error 

140733826068480

is a bug in the compiler. Why in the world is the compiler trying to allocate 140 GB
of RAM? It is quite possible for the error not to be reproducible if the machine you
are compiling it on is able to allocate the asked amount (which is not always the same
as being able to use it due to overcommit, it may just hope that you allocate but never
use, so it may not actually reserve the block until you touch it, and if it cannot,
it will core-dump the process). Try watching the virtual memory foot print (VIRT column
in top) and see what happens when the compilation succeeds.

One suspect could be if it decided to perform some optimization on a large array and
had some insane O(exp(n)) algorithm that showed its ugly side once n got sufficiently
large.

Original issue reported on code.google.com by spachev on 2014-01-26 00:06:02

[gforney]

spachev:

The error shown in Comment 31 has already been addressed in Issue 2076 and was occurring
because of a different part of the code unrelated to OpenMP.

Original issue reported on code.google.com by koverholt on 2014-01-26 01:30:37

[gforney]

Daniel:

I added you to the nightly Firebot email list, so you should start receiving build
status emails from Firebot.

I also enabled the OpenMP version in the Firebot nightly testing process. So, Firebot
will now build the debug and release versions of OpenMP FDS, check for errors in Intel
Inspector, and run the verification suite (in debug and release modes), and check for
any runtime errors.

Original issue reported on code.google.com by koverholt on 2014-01-27 16:58:48

[gforney]

Daniel:

Is this "normal" output from the Intel Inspector (esp. the warning about being multithreaded)?
I tried running the Inspector with the latest version of the openmp_intel_linux_64_inspect
target, but this is the output that I got. I just want to make sure that I am grepping
for the right things in the Firebot testing process.

[kjo@blaze Scripts]$ ./inspect_openmp.sh 
rm -f *.o *.mod *.obj 
Building openmp_intel_linux_64_inspect
ifort -c -g -O0 ../../FDS_Source/prec.f90
ifort -c -g -O0 ../../FDS_Source/mpis.f90
ifort -c -g -O0 ../../FDS_Source/cons.f90
ifort -c -g -O0 ../../FDS_Source/devc.f90
ifort -c -g -O0 -openmp ../../FDS_Source/type.f90
ifort -c -g -O0 ../../FDS_Source/pois.f90
ifort -c -g -O0 -openmp ../../FDS_Source/mesh.f90
ifort -c -g -O0 -openmp ../../FDS_Source/func.f90
ifort -c -g -O0 ../../FDS_Source/samr.f90
ifort -c -g -O0 ../../FDS_Source/data.f90
ifort -c -g -O0 ../../FDS_Source/smvv.f90
ifort -c -g -O0 ../../FDS_Source/irad.f90
ifort -c -g -O0 ../../FDS_Source/ieva.f90
ifort -c -g -O0 ../../FDS_Source/scrc.f90
ifort -c -g -O0 ../../FDS_Source/geom.f90
ifort -c -g -O0 ../../FDS_Source/ctrl.f90
ifort -c -g -O0 ../../FDS_Source/turb.f90
ifort -c -g -O0 ../../FDS_Source/hvac.f90
ifort -c -g -O0 ../../FDS_Source/fire.f90
ifort -c -g -O0 ../../FDS_Source/radi.f90
ifort -c -g -O0 ../../FDS_Source/evac.f90
ifort -c -g -O0 ../../FDS_Source/part.f90
ifort -c -g -O0 ../../FDS_Source/mass.f90
ifort -c -g -O0 -openmp ../../FDS_Source/velo.f90
ifort -c -g -O0 ../../FDS_Source/vege.f90
ifort -c -g -O0 ../../FDS_Source/wall.f90
ifort -c -g -O0 ../../FDS_Source/pres.f90
ifort -c -g -O0 ../../FDS_Source/dump.f90
ifort -c -g -O0 ../../FDS_Source/read.f90
ifort -c -g -O0 -openmp ../../FDS_Source/divg.f90
ifort -c -g -O0 ../../FDS_Source/init.f90
ifort -c -g -O0 ../../FDS_Source/main.f90
ifort -g -O0 -shared-intel -openmp -o fds_openmp_intel_linux_64_inspect prec.o mpis.o
cons.o devc.o data.o type.o mesh.o func.o smvv.o irad.o ieva.o pois.o scrc.o radi.o
evac.o geom.o part.o vege.o ctrl.o turb.o samr.o dump.o hvac.o mass.o read.o wall.o
fire.o divg.o init.o velo.o pres.o main.o
/home/kjo/FDS-SMV/FDS_Compilation/openmp_intel_linux_64_inspect/fds_openmp_intel_linux_64_inspect:
symbol lookup error: /home/kjo/FDS-SMV/FDS_Compilation/openmp_intel_linux_64_inspect/fds_openmp_intel_linux_64_inspect:
undefined symbol: __intel_new_feature_proc_init
Warning: Failed to find more than one thread. Ensure that your program is multithreaded.

0 new problem(s) found 

Original issue reported on code.google.com by koverholt on 2014-01-27 18:29:04

[gforney]

Actually, it seems to not be performing the inspection due to the error that I showed
before. Are you able to run the FDS-SMV/Utilities/Scripts/inspect_openmp.sh script
without errors?

Original issue reported on code.google.com by koverholt on 2014-01-27 18:33:49

[gforney]

I can use the script without any issues. I've been running prior to
all of my commits.

From the error messages it looks to me as if the compilation fails
during linking. My first guess would be that something went wrong when
cleaning or when sourcing the compilervars / inspxe_vars.

I added the compilation to the inspection script. It wasn't in there
previously, maybe this caused the problem.

Original issue reported on code.google.com by daniel.haarhoff on 2014-01-27 18:43:54

[gforney]

You were right, it was a local install of FDS on my account and the sourcing of files
that was messing things up. Fixed now, and it works on firebot's account. Thanks!

Original issue reported on code.google.com by koverholt on 2014-01-27 18:52:35

[gforney]

Do you get this warning when you run the inspect_openmp.sh script? Even though there
are no errors at the end of the inspection? If so, I will ignore this warning in Firebot.

Warning: One or more threads in the application accessed the stack of another thread.
This may indicate one or more bugs in your application. Setting the Inspector to detect
data races on stack accesses and running another analysis may help you locate these
and other bugs.

Original issue reported on code.google.com by koverholt on 2014-01-27 19:11:31

[gforney]

Yes, I also get that warning. I have looked into it:

 - it occurs even when the only OpenMP enabled in the makefile is func
(so only the threadcount check function)
 - setting the scope for inspection to extreme fails to produce that
warning but crashes the program

I have an appointment with the support here are JSC this week to have
a closer look. But right now I am guessing that this is safe and has
something to do with getting the threadcount.

Original issue reported on code.google.com by daniel.haarhoff on 2014-01-27 21:38:40

[gforney]

Thanks for the information. I've put in a statement to ignore that warning for now.
We'll see how Firebot likes the new changes/checks tonight.

Original issue reported on code.google.com by koverholt on 2014-01-27 21:49:35

[gforney]

Daniel:

I'm seeing some errors in today's OpenMP version that I did not see yesterday in the
debug version of FDS OpenMP. I'm getting SIGSEGV crashes on random verification cases
or the case will quit without printing time step information and without an error (see
below). The errors seem to be related to commits r18216 and r18217 because it works
if I comment out the radi FOPENMPFLAGS in the makefile. I'm going to revert these changes
so I can make sure Firebot is working okay without errors.

[kjo@blaze Controls]$ fds_mp_db hrr_freeze.fds
  8 threads available

 Fire Dynamics Simulator

 Compilation Date : Tue, 28 Jan 2014
 Current Date     : January 28, 2014  17:28:43

 Version: FDS 6.0.2; MPI Disabled; OpenMP Enabled
 Number of available threads:   8

 SVN Revision No. : 18217

 Job TITLE        : Test of NO_UPDATE_DEVC_ID
 Job ID string    : hrr_freeze

[kjo@blaze Controls]$

Original issue reported on code.google.com by koverholt on 2014-01-28 22:33:52

[gforney]

Short update:

I think I have submitted all the loops contributing significantly to the runtime that
can be parallelised using only OpenMP pragmas.

I have identified three issues that need to be solved to continue further:

 1. Loops utilise ZZ_GET break upon parallelisation
 2. The cross thread stack access warning persists
 3. wall cell indices cause data races

Regarding 1. ZZ_GET:
  These loops constitute significant runtime percentages in divg and wall.
  I have found a solution which involves turning ZZ_GET into an allocatable.
  I am still trying to understand what is going wrong. It might be a compiler bug.
  If it isn't a bug I would like to submit changes to the actual code. Is this oK?

Regarding 2. CTSA
  Using custom inspection profiles in the intel inspector I have located the relevant
line. [1]
  The inspector doesn't detect a data race upon closer inspection.
  So it seems to only be a warning. But I would prefer to understand it.
  As of now I haven't been able to figure it out, but I am working on it.
  Do you have any ideas?

Regarding 3. Wall Loops
  It would be nice to safely parallelise these.
  Do you have any idea if any of the II, IIG etc. indices could be identical for multiple
wall cells?
  For example: what happens in corners where a cell might have two or three walls next
to it?

Cheers
Daniel

Appendix:
=== [1] =================================================
SUBROUTINE GET_SENSIBLE_ENTHALPY_DIFF(N,TMPG,DELTA_H_S)

INTEGER :: ITMP
REAL(EB), INTENT(IN) :: TMPG
INTEGER, INTENT(IN) :: N
REAL(EB), INTENT(OUT) :: DELTA_H_S

ITMP = MIN(5000,NINT(TMPG))

DELTA_H_S = CP_AVG_DIFF(ITMP,N)*TMPG <<< this is the one

END SUBROUTINE GET_SENSIBLE_ENTHALPY_DIFF
========================================================

Original issue reported on code.google.com by daniel.haarhoff on 2014-02-05 15:58:52

[gforney]

Hi Daniel,

I would say 1. is OK to do.  I can't imagine that firebot will not pick up any problems
if ZZ_GET gets broken.

I will let Kris comment on 2.

Regarding 3., yes, IIG, can be the same for two different wall cells, as you say, in
a corner (IIG, gas phase cell, is the same) or edge of a block (II, the ghost cell,
will be the same).

R

Original issue reported on code.google.com by randy.mcdermott on 2014-02-05 17:30:21