Nalu, Nalu-Nalu, Nalu-AMR comparison

[kevmoor]

May need to be broken up into several tasks

• Need to answer solver convergence within one iteration and overall picard iterations convergence.
• Maybe use ML tasks to help inform this. Could go to steady state, and then restart with different solver parameters (inner and picard) and observe forces.
• End of March FSI milestone is Go-No-Go decision. Having Nalu comparison should be very valuable for March milestone and negotiation with WETO.
• If Nalu only is already set up then should try that as it should be close
• In parallel do the benchmark problem. To verify beamdyn setup – pass in loads and verify deflections are the same – (pull in Neil).

[kevmoor]

Kevin to create script that produces 6x6 matrices matching benchmark beam and send to structural folks to review. Nate to update aero side of openfast. We get it running well in standalone openfast, and then move to HFM.

[pscrozi]

This isn't critical to the completion of the milestone, but could be valuable.

[pscrozi]

Phil sent Nate his prior work on this, so we have a good starting point.

Probably need a meeting to discuss how best to approach this, particularly with ABL matching. Potential invitees: Nate, Phil, Lawrence.

[pscrozi]

Nate was doing some work on meshing for this right now. No further updates today. We just have to get everything set up to run these cases. Already met to discuss verification tests they wanted to run, but this is a backburner issue until milestone is complete.

[pscrozi]

Phil and Nate met to review meshing strategies for this. Nate is going to start working on this and how to generate nested refinements in Nalu. Lawrence and Myra will help with this.

[kevmoor]

Nate got scripts and proof of concept for conformal refined mesh in similar way to AMR refinements, but conformal with 3:1 mapping on hexes. Will test on simple solar panel case adjacent, then apply to turbine. Need to script up a few of the manual steps to get exo file at the end.

[pscrozi]

Nate updated Phil's script with sidesets and now we're getting better results. We still have to automate some of sizings and things, but it works as currently hard-coded.

[pscrozi]

Nate, Phil, and Lawrence have been discussing this. Nate will send out a coordination e-mail this afternoon with building cases with growing complexity to test out the Nalu-Nalu coupling. May start with actuator line and make sure we're ok and not breaking anything.

Identifying a pathway to build up from simple cases up to the turbine itself.

Trying to run a refined case that is failing in the equation set somewhere that is going all NaNs.

We may need a mesh specification from Ganesh.

[pscrozi]

Nate still needs a new mesh specification for Nalu-Nalu runs, as of 2/28.

Nate got the flat plate case running. Has a lot of diffusion between the levels, probably because the background is too coarse right now. Wanted to get it running before throwing more cells at it. Seems to be working with multi-realm.

These are nested conformal boxes. Nate will work on this for turbines.

Trying to get Nalu-Nalu overset working.

Will we be able to run an FSI simulation soon? Can we do this with IEA-15MW and accessible to everyone on Eagle?

(Note: we're currently out of allocation on Eagle.)

[pscrozi]

Nearly have final mesh for IEA-15MW.

Could run on Kestrel since Nalu-Wind seems to do fine on Kestrel.

Nate will start these Nalu-Nalu runs with IEA-15MW soon. Need to get build on Kestrel first, or an FSI build on one of the Sandia machines. Probably best on Kestrel since we have higher priority there. Could test on attaway first. Maybe early next week could get running on Kestrel.

Could put mesh somewhere so someone else could pick it up.

[kevmoor]

Nate: Starting build today on kestrel, posted quick one-liner for moving/syncing things to kestrel. Once build is done, will start setting up nalu and nalu^2 (exawind driver) case for IEA15MW.

[pscrozi]

Nate has a build on Kestrel, and now is working on the input files. Worked with Phil on module issues.

[pscrozi]

Nate needs to connect with Phil since there's weirdness in the build. Bad yaml file error, even though yaml verified. Someday might want a pre-process check. Showing Trilinos runtime solver support not enabled. Something weird going on.

Nate will switch to hypre. Kevin has not been able to swap over to hypre.

Kevin will post as a bug the issue he's seeing. Add Jon, Alan W, Paul M, Phil. STK's changes to overset might not be tested with hypre.

[pscrozi]

Still trying to run on Kestrel. Nate posted latest run issues to Jon Rood, and waiting to hear back.

May be an issue with the mesh decomposition in Nalu-Wind to cause an MPI issue.

Alternatively, may try to run elsewhere.

Ganesh has FSI simulations running on Kestrel, so perhaps Nate can copy build process that Ganesh used?

Ganesh has been able to get build system working on Kestrel and Frontier.

Nate could try the old version of spack-manager to get it to build on Kestrel.

[pscrozi]

Nate discovered why it wasn't running: failing on rebalance. Something about the build on Kestrel with parmetis and STK-balance. When removing the rebalance, can get the case to run.

Nate now has a Nalu-Nalu multi-realm running. Still not with FSI and not moving. Taking about 2 minutes to run a step or so. So very slow. Nate will look at ideas to reduce the timestep time. How many elements per core? 50k. We should parking lot this.

Nate has a stand-alone STK-balance going too to see if it fails.

[pscrozi]

Nate foiled by Kestrel a bit, but back on it now. Building ExaWind on "Flight" HPC at Sandia now, which is empty now. Trying to build this week and get it running this week. Decide with Phil on matrix of cases we want to run.

[pscrozi]

Nate: have all files moved over, and doing setup. Should start running tomorrow (4/25). Need a short meeting with Phil to discuss test matrix.

[pscrozi]

Nate has put up info on the slack channel about this. There's an issue with the most recent FSI stack and the entire ExaWind stack, but there's an issue when Nalu-Wind tries to start with the precursor, it fails on an OpenFast library. We've tried moving back and bisecting, but can't find a version that doesn't do that, so the issue may be elsewhere. Looks specific to the restart with Nalu-Wind. Phil and Nate can hopefully discuss offline.

Phil suspects that it is a failure of the management of our software, specifically Tioga and Nalu-Wind. Complicated when it crosses multiple codes with many dependencies. The recent update with Trilinos could be a culprit here. Versioning could be an issue.

[pscrozi]

Nate: Flight has been down this week. Waiting for that to come back online.

[pscrozi]

Nate: back up running on Flight. Have AMR/Nalu running. Now fixed an input file error, and running the precursor again. That should show if there's an issue with the Nalu/Nalu.

[pscrozi]

Nate has been working on different corner cases on the Nalu/Nalu case, nested refinement, etc. Nate has been running cases, but trying to get the full turbine working. Found and worked through an OpenFAST issue, and a segfault on restart, which Nate fixed.

[pscrozi]

Should we continue this task, or call it off. Been on the board for 4 months. How much time is Nate spending on this? (Wait for Nate to discuss.)

[pscrozi]

Why are we doing this? Phil: we want to evaluate the numerical convergence of our FSI algorithm. We need to evaluate the solver convergence. With Nalu-Nalu, we can do outer iterations and evaluate the convergence of our algorithm.

Where are we on this right now? Ilker is working on AMR-Wind outer iteration capability. Would our resources be better spent there?

There would still be questions about using different discretizations between Nalu-Wind and AMR-Wind.

Nate: we dropped this temporarily due to getting the milestone work done, and this task wasn't on the critical path. But the case is still important, meshed, and ready to go.

Mike: If it can be done in short-order, we should proceed. We want this in a journal article.

Nate: does anyone have a script that can be used to extract iterations from log files? (No answer.)

[pscrozi]

No updates

[pscrozi]

Nate: have a Nalu-Nalu running slowly with 15 MW. Need to sit down with some folks to decide what we want to see with a paper on this. We want to try high wind speeds. Might want to track the convergence on lower wind speed cases as well. Explore parameter space: solution iterations, wind speeds, etc.

Phil: run 2 or 3 cases to start. (1) reproduce lower wind speed coupled Nalu-AMR; (2) redo the case that's failing with AMR-Wind that we couldn't do with the high wind speed with ExaWind driver. Nate will run these.

[pscrozi]

Nate: have 3 cases running (1, 2, and 4 nonlinear iterations). Setting up Nalu-AMR to mirror, but limited to 1 nonlinear iteration. Should have a basic comparison to see if extra solves change the solution significantly. If there are big differences, we need more nonlinear iterations. Then we can move on to one of the higher wind speed cases to see if this helps.

Mike: Ilker is pushing on this to make AMR-Wind be able to do outer iterations.

[pscrozi]

Nate: 3 cases didn't get very far. Post-processing now. Ran with elastodyne, but resubmitting with beamdyne.

[pscrozi]

Nate: I did get a case running after some weird troubleshooting. Run out to about 20 s or so, at about 1 s per hour. Expect 80 or 90 seconds on a full run. Going to copy and resubmit with more non-linear iterations to get out to the same amount of time. Need to decide how many rotations we need and windspeeds we want. Now running with beamdyne.

[pscrozi]

Nate's update as of 8/2/24:

It seems like a never-ending series of challenges to complete this.

• The beamdyn single outer iteration nalu-nalu case ran for around 13000 steps and then ended with a crash and message about negative control volumes. • I posted on slack showing the mesh deformation...it looks like the volume mesh near the tip is folding in on itself...which is bad, and likely the source of the crash message • I also noticed that with this case, openfast stopped writing data output after the precursor...we need to investigate this one...

In general, I think we're still uncovering places where the code isn't robust. I understand the concern about the pace, and can dedicate time this quarter to making sure it gets out the door, but I also want to make sure we all understand that it is a very winding path to get there.

[kevmoor]

Mike: hoping we are just putting Nalu-Nalu to the side, Nate: correct, shifting to Pazy wing.

Problems recently found in this will crop with when we attempt with AMR (folding with mesh at tip, and mesh deformation). Main driver is to do more nonlinear iterations. Mike - should just invest in Nalu-AMR wind. Should move into backlog.