WISDEM / WEIS

Wind Energy with Integrated Servo-controls Toolset
https://weis.readthedocs.io/en/latest/
Apache License 2.0
53 stars 40 forks source link

Running DLC in parallel has poor performance due to improper usage of cores #113

Closed ewquon closed 3 years ago

ewquon commented 3 years ago

Description

Unexpected behaviors have been observed using mpiexec -n $nC --bind-to core python driver.py. E.g., with 23 wind speeds and 6 turbulence seeds (nC=N+1=139, with N=23*6)

The fix, courtesy of @nikhar-abbas :

diff --git a/weis/aeroelasticse/openmdao_openfast.py b/weis/aeroelasticse/openmdao_openfast.py
index 7de0f78..a417071 100644
--- a/weis/aeroelasticse/openmdao_openfast.py
+++ b/weis/aeroelasticse/openmdao_openfast.py
@@ -1013,7 +1013,7 @@ class FASTLoadCases(ExplicitComponent):
         fastBatch.overwrite_outfiles = True  #<--- Debugging only, set to False to prevent OpenFAST from running if the .outb already exists

         # Run FAST
-        if self.mpi_run and self.options['opt_options']['driver']['optimization']['flag']:
+        if self.mpi_run: #and self.options['opt_options']['driver']['optimization']['flag']:
             summary_stats, extreme_table, DELs, chan_time = fastBatch.run_mpi(self.mpi_comm_map_down)
         else:
             if self.cores == 1:
@@ -1123,7 +1123,7 @@ class FASTLoadCases(ExplicitComponent):
         iec.overwrite       = False # TODO: elevate these options to analysis input file
         iec.run_dir         = self.FAST_runDirectory

-        if self.mpi_run and self.options['opt_options']['driver']['optimization']['flag']:
+        if self.mpi_run: #and self.options['opt_options']['driver']['optimization']['flag']:
             iec.parallel_windfile_gen = True
             iec.mpi_run               = self.FASTpref['analysis_settings']['mpi_run']
             iec.comm_map_down         = self.FASTpref['analysis_settings']['mpi_comm_map_down']
diff --git a/weis/glue_code/runWEIS.py b/weis/glue_code/runWEIS.py
index 2f9d334..c565f5a 100644
--- a/weis/glue_code/runWEIS.py
+++ b/weis/glue_code/runWEIS.py
@@ -183,7 +183,7 @@ def run_weis(fname_wt_input, fname_modeling_options, fname_opt_options, overridd
             # Save data to numpy and matlab arrays
             fileIO.save_data(froot_out, wt_opt)

-    if MPI and modeling_options['Level3']['flag'] and opt_options['driver']['optimization']['flag']:
+    if MPI and modeling_options['Level3']['flag']: #and opt_options['driver']['optimization']['flag']:
         # subprocessor ranks spin, waiting for FAST simulations to run
         sys.stdout.flush()
         if rank in comm_map_up.keys():

Steps to reproduce issue

Using https://github.com/ewquon/WEIS/tree/powercurve_points, which tracks https://github.com/WISDEM/WEIS/tree/powercurve_points, run with modeling options:

nikhar-abbas commented 3 years ago

@ewquon thanks for communicating this

I am about to open a PR with a few other minor fixes. Given that your git diff all includes bugs that I probably introduced, I can fix things in a way that should be a bit more robust.

ewquon commented 3 years ago

Resolved by https://github.com/WISDEM/WEIS/pull/124