Running DLC in parallel has poor performance due to improper usage of cores

[ewquon]

Description

Unexpected behaviors have been observed using mpiexec -n $nC --bind-to core python driver.py. E.g., with 23 wind speeds and 6 turbulence seeds (nC=N+1=139, with N=23*6)

• Running current code with 4 nodes results in [N+36, 36, 36, 31] processes per node (instead of [36,36,36,31])
• With fix, running without --bind-to core option: [36,36,36,36]
• With fix, with --bind-to core option: [36,36,36,31]

The fix, courtesy of @nikhar-abbas :

diff --git a/weis/aeroelasticse/openmdao_openfast.py b/weis/aeroelasticse/openmdao_openfast.py
index 7de0f78..a417071 100644
--- a/weis/aeroelasticse/openmdao_openfast.py
+++ b/weis/aeroelasticse/openmdao_openfast.py
@@ -1013,7 +1013,7 @@ class FASTLoadCases(ExplicitComponent):
         fastBatch.overwrite_outfiles = True  #<--- Debugging only, set to False to prevent OpenFAST from running if the .outb already exists

         # Run FAST
-        if self.mpi_run and self.options['opt_options']['driver']['optimization']['flag']:
+        if self.mpi_run: #and self.options['opt_options']['driver']['optimization']['flag']:
             summary_stats, extreme_table, DELs, chan_time = fastBatch.run_mpi(self.mpi_comm_map_down)
         else:
             if self.cores == 1:
@@ -1123,7 +1123,7 @@ class FASTLoadCases(ExplicitComponent):
         iec.overwrite       = False # TODO: elevate these options to analysis input file
         iec.run_dir         = self.FAST_runDirectory

-        if self.mpi_run and self.options['opt_options']['driver']['optimization']['flag']:
+        if self.mpi_run: #and self.options['opt_options']['driver']['optimization']['flag']:
             iec.parallel_windfile_gen = True
             iec.mpi_run               = self.FASTpref['analysis_settings']['mpi_run']
             iec.comm_map_down         = self.FASTpref['analysis_settings']['mpi_comm_map_down']
diff --git a/weis/glue_code/runWEIS.py b/weis/glue_code/runWEIS.py
index 2f9d334..c565f5a 100644
--- a/weis/glue_code/runWEIS.py
+++ b/weis/glue_code/runWEIS.py
@@ -183,7 +183,7 @@ def run_weis(fname_wt_input, fname_modeling_options, fname_opt_options, overridd
             # Save data to numpy and matlab arrays
             fileIO.save_data(froot_out, wt_opt)

-    if MPI and modeling_options['Level3']['flag'] and opt_options['driver']['optimization']['flag']:
+    if MPI and modeling_options['Level3']['flag']: #and opt_options['driver']['optimization']['flag']:
         # subprocessor ranks spin, waiting for FAST simulations to run
         sys.stdout.flush()
         if rank in comm_map_up.keys():

Steps to reproduce issue

Using https://github.com/ewquon/WEIS/tree/powercurve_points, which tracks https://github.com/WISDEM/WEIS/tree/powercurve_points, run with modeling options:

• openfast.analysis_settings.Analysis_level = 2
• openfast.dlc_settings.run_IEC = True
• openfast.dlc_settings.IEC:

     - DLC: 1.1
       U: [3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
       Seeds: [13,17,19,23,29,37]

[nikhar-abbas]

@ewquon thanks for communicating this

I am about to open a PR with a few other minor fixes. Given that your git diff all includes bugs that I probably introduced, I can fix things in a way that should be a bit more robust.

[ewquon]

Resolved by https://github.com/WISDEM/WEIS/pull/124