Example 2 fails with RotorSE: True

[mlidGWT]

Description

Hi again,

I am trying to run WEIS in the format of example 02, where an openfast_file and openfast_dir are specified with the OpenFAST input files. I am able to both run example 02 and my own simulations with this format, until enabling RotorSE, at which point I get the errors below in the Current Behavior section (the errors are different for example 02 than for my own runs).

In my own directory, I am using the IEA 15MW RWT model, though it seems to be a slightly different 'vintage', with 60 blade stations and other numerical differences vs. those in example 02.

I am not trying to run an optimization, simply to use WEIS's DLC-generation ability paired with more control over the OpenFAST input files, so if my error is around some attempted optimization task, please advise how to turn that off. I have set driver: optimization: flag: False and recorder: flag: False in the modeling_options.yaml and am still getting the below error.

If enabling RotorSE means something in the WEIS files needs to be configured differently, it would be great if documentation around those changes could be added - perhaps to the README file in that example's folder - as they are not self-evident.

Thanks very much for your help - this issue is keeping me from making progress in my day job! Best, Meg

Steps to reproduce issue

In example 02:

1. Set RotorSE: flag: True in `modeling_options.yaml'
2. Run WEIS using python weis_driver.py

Current behavior

Example 02 error

Using weis.aeroelasticse in ROSCO_toolbox... Loading wind turbine data for NREL's ROSCO tuning and simulation processeses Loading FAST model: IEA-15-240-RWT-UMaineSemi.fst Loading rotor performace data from text file: /Users/meglidrbauch/Software/WEIS/examples/01_aeroelasticse/OpenFAST_models/IEA-15-240-RWT/IEA-15-240-RWT/Cp_Ct_Cq.IEA15MW.txt Traceback (most recent call last): File "/Users/meglidrbauch/Software/WEIS/examples/02_control_opt/weis_driver.py", line 15, in wt_opt, modeling_options, opt_options = run_weis(fname_wt_input, fname_modeling_options, fname_analysis_options) File "/Users/meglidrbauch/Software/WEIS/weis/glue_code/runWEIS.py", line 154, in run_weis wt_opt = yaml2openmdao(wt_opt, modeling_options, wt_init, opt_options) File "/Users/meglidrbauch/Software/WEIS/WISDEM/wisdem/glue_code/gc_WT_InitModel.py", line 51, in yaml2openmdao hub = wt_init["components"]["hub"] KeyError: 'hub'

My directory error

Note: this error comes after updating the n_pitch_perf_surfaces parameter in modeling_options.yaml to 104 to match the pitch vector dimension in the Cp_Ct_Cq.txt file, and the n_tsr_perf_surfaces parameter to 72 for the same reason. I am unable to follow the proverbial thread in the error message to determine where the max_allowable_td_ratio should be found.

Using weis.aeroelasticse in ROSCO_toolbox... Loading wind turbine data for NREL's ROSCO tuning and simulation processeses Loading FAST model: IEA-15-240-RWT-GWTBaseline.fst Loading rotor performace data from text file: /Users/meglidrbauch/Software/LoadsRuns/ModelLibrary/IEA15_GWTBaseline/IEA15_GWTB_Cp_Ct_Cq.txt Traceback (most recent call last): File "/Users/meglidrbauch/Software/LoadsRuns/SimulationLibrary/RunDir/weis_driver.py", line 15, in wt_opt, modeling_options, opt_options = run_weis(fname_wt_input, fname_modeling_options, fname_analysis_options) File "/Users/meglidrbauch/Software/WEIS/weis/glue_code/runWEIS.py", line 159, in run_weis wt_opt = myopt.set_initial(wt_opt, wt_init) File "/Users/meglidrbauch/Software/WEIS/WISDEM/wisdem/glue_code/gc_PoseOptimization.py", line 1685, in set_initial wt_opt["tcons.max_allowable_td_ratio"] = blade_constr["tip_deflection"]["margin"] File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/problem.py", line 513, in setitem self.set_val(name, value) File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/problem.py", line 544, in set_val self.model.set_val(name, val, units=units, indices=indices) File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/system.py", line 5368, in set_val raise KeyError(f'{model.msginfo}: Variable "{name}" not found.') KeyError: ' : Variable "tcons.max_allowable_td_ratio" not found.'

Expected behavior

Simulations run as expected and I am able to run DLCs through WEIS with OpenFAST input files defined.

Code versions

• Python 3.10
• WEIS develop

[dzalkind]

Hi,

These 02 examples are designed to be run without most parts of WISDEM. They will not update an OpenFAST model except to run different load cases. Enabling RotorSE will result in unexpected behavior.

We are planning to release a series of software and documentation improvements over the next year with new ARPA-E funding, and we would be happy to discuss specific solutions for you and your team.

Best, Dan

[dzalkind]

Let's leave this issue open until we can throw an appropriate error for the situation posed.

[mlidGWT]

Hi @dzalkind, thanks for the explanation - I didn't understand the relationship between the modules and was going off of the settings for previous WEIS runs.

I'm now encountering an issue where WEIS seems to run the first Case without any issue, but then does not proceed to the next one and seems to hang after printing the runtime for the completed Case, almost like it is expecting user input, but without any prompts about what to input. Do you know what might be causing this?

Thank you! Meg

[dzalkind]

Hi Meg,

Can you please share the output log? It may also just be that WEIS finished its runs. There should be two case_matrix files (one .txt and one .yaml) where you run the OpenFAST simulations. There, you can confirm which cases have been set up.

Dan

[mlidGWT]

Hi Dan,

I don't seem to have an output log that indicates what is happening in the backend, but the case matrix looks like all the cases are properly defined: case_matrix.txt

Please advise how to enable the output log you are referring to - I don't see one in the WEIS yamls or the .fst file. (verbosity in the modeling_options seems to be for printing to screen, Echo in the .fst file seems to echo the inputs, and I take SumPrint to be a summary of the results, not the simulation state)

Meg

[dzalkind]

Hi Meg,

By the output log, I mean the terminal messages from the WEIS run. You can python <weis_driver.py> > output.log to save it to a file.

Dan

[mlidGWT]

I'm not having success with that command. I've tried some variations, but as written I get:

zsh: parse error near `>'

If I try python weis_driver.py > output.log:

RuntimeWarning: /Users/meglidrbauch/Software/WEIS/weis/control/dac.py:734 invalid value encountered in scalar divide

which then hangs and I have to interrupt, leading to a long Traceback message.

[dzalkind]

Instead of writing to your terminal, it's writing to output.log. Is there anything in that file before you interrupted it?

[mlidGWT]

Ah, I was expecting the file to be in the temp/ folder, but it was at the cd level. Now attached:

output.log

It was hanging after

OpenFAST terminated normally.

and required two CTRL+C interrupts before adding the final 4 lines of the file.

[dzalkind]

Are you still running with the same modeling options that we defined here?

Can you let it go for a while and try to figure out where it's hanging? Is there an error message that appears when you CTRL+C?

[mlidGWT]

I am still running it with the OpenFAST executable, I now have verbosity: True, but I do not have it saving timeseries or iterations. Currently all of the *SE options in the WISDEM section are set to False, as well as BOS.

I have had it running since 5 ET, and so far the log only contains up until

OpenFAST terminated normally.

(I have refreshed it a couple times). I can try a single keyboard interrupt to see what else comes in, since it seems to take 2 interrupts to fully kill the run.

[mlidGWT]

This time a single interrupt killed it, at which point the final 4 lines of the log file were printed, as in the version I shared.

The terminal error message after the interrupt:

RuntimeWarning: /Users/meglidrbauch/Software/WEIS/weis/control/dac.py:734 invalid value encountered in scalar divide^CTraceback (most recent call last): File "/Users/meglidrbauch/Software/LoadsRuns/SimulationLibrary/6011_SBIR/weis_driver.py", line 15, in wt_opt, modeling_options, opt_options = run_weis(fname_wt_input, fname_modeling_options, fname_analysis_options) File "/Users/meglidrbauch/Software/WEIS/weis/glue_code/runWEIS.py", line 199, in run_weis wt_opt.run_model() File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/problem.py", line 629, in run_model self.model.run_solve_nonlinear() File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/system.py", line 4684, in run_solve_nonlinear self._solve_nonlinear() File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/group.py", line 2532, in _solve_nonlinear self._nonlinear_solver._solve_with_cache_check() File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/solvers/nonlinear/nonlinear_runonce.py", line 26, in _solve_with_cache_check self.solve() # don't use caching File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/solvers/nonlinear/nonlinear_runonce.py", line 45, in solve self._gs_iter() File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/solvers/solver.py", line 800, in _gs_iter subsys._solve_nonlinear() File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/explicitcomponent.py", line 312, in _solve_nonlinear self._compute_wrapper() File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/explicitcomponent.py", line 283, in _compute_wrapper self.compute(self._inputs, self._outputs, File "/Users/meglidrbauch/Software/WEIS/weis/aeroelasticse/openmdao_openfast.py", line 611, in compute summary_stats, extreme_table, DELs, Damage, case_list, case_name, chan_time, dlc_generator = self.run_FAST(inputs, discrete_inputs, fst_vt) File "/Users/meglidrbauch/Software/WEIS/weis/aeroelasticse/openmdao_openfast.py", line 2107, in run_FAST summary_stats, extreme_table, DELs, Damage, chan_time = fastBatch.run_serial() File "/Users/meglidrbauch/Software/WEIS/weis/aeroelasticse/runFAST_pywrapper.py", line 361, in run_serial _name, _ss, _et, _dl, _dam, _ct = evaluate(c) File "/Users/meglidrbauch/Software/WEIS/weis/aeroelasticse/runFAST_pywrapper.py", line 480, in evaluate return fast.execute() File "/Users/meglidrbauch/Software/WEIS/weis/aeroelasticse/runFAST_pywrapper.py", line 265, in execute case_name, sum_stats, extremes, dels, damage = self.la._process_output(output, File "/Users/meglidrbauch/Software/WEIS/pCrunch/pCrunch/analysis.py", line 178, in _process_output extremes = output.extremes(output.channels) File "/Users/meglidrbauch/Software/WEIS/pCrunch/pCrunch/io/openfast.py", line 132, in extremes idx_max = self.idxmaxs[i] File "/Users/meglidrbauch/Software/WEIS/pCrunch/pCrunch/io/openfast.py", line 19, in wrapper return f(self, args, kwargs) File "/Users/meglidrbauch/Software/WEIS/pCrunch/pCrunch/io/openfast.py", line 167, in idxmaxs return np.argmax(self.data, axis=0) File "<__array_function__ internals>", line 200, in argmax File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 1242, in argmax return _wrapfunc(a, 'argmax', axis=axis, out=out, kwds) File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc return bound(args, **kwds) KeyboardInterrupt

[mlidGWT]

I let it run one more time before logging off (~5:30 PM ET) and just checked back on it (8:40 PM ET). It seems it wasn't hung, just taking an incredibly long time in between Cases. The first Case finished running in OpenFAST at 5:37 (~4 minute runtime on a 320s simulation), and OpenFAST was not called again until 6:18 PM for Case 2, finishing up about 4 minutes later as well. The 3rd Case started at 7:03 PM, the 4th at 7:48 PM, and the 5th at 8:30, which has completed but the 6th not yet begun. I will share the updated log file tomorrow once it is completed, but I don't see any messages in it that indicate why the delay between cases is so long. As far as I'm aware, all optimizations should be off and it should simply be reading and running the specified OpenFAST input files, though it's possible I've missed something somewhere.

[ptrbortolotti]

Hello Meg, thank you, that helps the debugging... Can you please check to make sure that your output files have a reasonable size (10-100 MB)? Is the number of output channel "ordinary" (100 or so), is your DT_Out in OpenFAST normal (100 Hz or so)? The alternative is that something went wrong with the compiling of pCrunch, but that would be a first. Also, is there a way for us to reproduce the issue locally? If not, feel free to send me an invite and we can do a screenshare Pietro

[mlidGWT]

Hi Pietro and Dan,

Thank you both for your patience and support, I'm a bit embarrassed to say that everything seems to be working, I just overscoped the outputs for what my computer is capable of. I had nodal outputs turned on for both AeroDyn and ElastoDyn, for all 3 blades with more than 20 nodes in each case. Each output file was 1.13GB! I'm trying again with some different settings. I didn't realize how much of an impact the nodal settings would have on run time, but it makes sense in hindsight. I might suggest these would be good warnings/notes to include in documentation around nodal outputs. :)

Thank you again for your support, and apologies if I wasted your time! Meg