search

Exawind / hfm-fsi

Input files and data for HFM FSI Work
0 stars 0 forks source link

Nalu, Nalu-Nalu, Nalu-AMR comparison #79

Open kevmoor opened 4 months ago

kevmoor commented 4 months ago

May need to be broken up into several tasks

kevmoor commented 4 months ago

Kevin to create script that produces 6x6 matrices matching benchmark beam and send to structural folks to review. Nate to update aero side of openfast. We get it running well in standalone openfast, and then move to HFM.

pscrozi commented 4 months ago

This isn't critical to the completion of the milestone, but could be valuable.

pscrozi commented 4 months ago

Phil sent Nate his prior work on this, so we have a good starting point.

Probably need a meeting to discuss how best to approach this, particularly with ABL matching. Potential invitees: Nate, Phil, Lawrence.

pscrozi commented 4 months ago

Nate was doing some work on meshing for this right now. No further updates today. We just have to get everything set up to run these cases. Already met to discuss verification tests they wanted to run, but this is a backburner issue until milestone is complete.

pscrozi commented 4 months ago

Phil and Nate met to review meshing strategies for this. Nate is going to start working on this and how to generate nested refinements in Nalu. Lawrence and Myra will help with this.

kevmoor commented 4 months ago

Nate got scripts and proof of concept for conformal refined mesh in similar way to AMR refinements, but conformal with 3:1 mapping on hexes. Will test on simple solar panel case adjacent, then apply to turbine. Need to script up a few of the manual steps to get exo file at the end.

pscrozi commented 4 months ago

Nate updated Phil's script with sidesets and now we're getting better results. We still have to automate some of sizings and things, but it works as currently hard-coded.

pscrozi commented 4 months ago

Nate, Phil, and Lawrence have been discussing this. Nate will send out a coordination e-mail this afternoon with building cases with growing complexity to test out the Nalu-Nalu coupling. May start with actuator line and make sure we're ok and not breaking anything.

Identifying a pathway to build up from simple cases up to the turbine itself.

Trying to run a refined case that is failing in the equation set somewhere that is going all NaNs.

We may need a mesh specification from Ganesh.

pscrozi commented 4 months ago

Nate still needs a new mesh specification for Nalu-Nalu runs, as of 2/28.

Nate got the flat plate case running. Has a lot of diffusion between the levels, probably because the background is too coarse right now. Wanted to get it running before throwing more cells at it. Seems to be working with multi-realm.

These are nested conformal boxes. Nate will work on this for turbines.

Trying to get Nalu-Nalu overset working.

Will we be able to run an FSI simulation soon? Can we do this with IEA-15MW and accessible to everyone on Eagle?

(Note: we're currently out of allocation on Eagle.)

pscrozi commented 4 months ago

Nearly have final mesh for IEA-15MW.

Could run on Kestrel since Nalu-Wind seems to do fine on Kestrel.

Nate will start these Nalu-Nalu runs with IEA-15MW soon. Need to get build on Kestrel first, or an FSI build on one of the Sandia machines. Probably best on Kestrel since we have higher priority there. Could test on attaway first. Maybe early next week could get running on Kestrel.

Could put mesh somewhere so someone else could pick it up.

kevmoor commented 4 months ago

Nate: Starting build today on kestrel, posted quick one-liner for moving/syncing things to kestrel. Once build is done, will start setting up nalu and nalu^2 (exawind driver) case for IEA15MW.

pscrozi commented 3 months ago

Nate has a build on Kestrel, and now is working on the input files. Worked with Phil on module issues.

pscrozi commented 3 months ago

Nate needs to connect with Phil since there's weirdness in the build. Bad yaml file error, even though yaml verified. Someday might want a pre-process check. Showing Trilinos runtime solver support not enabled. Something weird going on.

Nate will switch to hypre. Kevin has not been able to swap over to hypre.

Kevin will post as a bug the issue he's seeing. Add Jon, Alan W, Paul M, Phil. STK's changes to overset might not be tested with hypre.

pscrozi commented 3 months ago

Still trying to run on Kestrel. Nate posted latest run issues to Jon Rood, and waiting to hear back.

May be an issue with the mesh decomposition in Nalu-Wind to cause an MPI issue.

Alternatively, may try to run elsewhere.

Ganesh has FSI simulations running on Kestrel, so perhaps Nate can copy build process that Ganesh used?

Ganesh has been able to get build system working on Kestrel and Frontier.

Nate could try the old version of spack-manager to get it to build on Kestrel.

pscrozi commented 3 months ago

Nate discovered why it wasn't running: failing on rebalance. Something about the build on Kestrel with parmetis and STK-balance. When removing the rebalance, can get the case to run.

Nate now has a Nalu-Nalu multi-realm running. Still not with FSI and not moving. Taking about 2 minutes to run a step or so. So very slow. Nate will look at ideas to reduce the timestep time. How many elements per core? 50k. We should parking lot this.

Nate has a stand-alone STK-balance going too to see if it fails.

pscrozi commented 2 months ago

Nate foiled by Kestrel a bit, but back on it now. Building ExaWind on "Flight" HPC at Sandia now, which is empty now. Trying to build this week and get it running this week. Decide with Phil on matrix of cases we want to run.

pscrozi commented 2 months ago

Nate: have all files moved over, and doing setup. Should start running tomorrow (4/25). Need a short meeting with Phil to discuss test matrix.

pscrozi commented 2 months ago

Nate has put up info on the slack channel about this. There's an issue with the most recent FSI stack and the entire ExaWind stack, but there's an issue when Nalu-Wind tries to start with the precursor, it fails on an OpenFast library. We've tried moving back and bisecting, but can't find a version that doesn't do that, so the issue may be elsewhere. Looks specific to the restart with Nalu-Wind. Phil and Nate can hopefully discuss offline.

Phil suspects that it is a failure of the management of our software, specifically Tioga and Nalu-Wind. Complicated when it crosses multiple codes with many dependencies. The recent update with Trilinos could be a culprit here. Versioning could be an issue.

pscrozi commented 1 month ago

Nate: Flight has been down this week. Waiting for that to come back online.

pscrozi commented 1 month ago

Nate: back up running on Flight. Have AMR/Nalu running. Now fixed an input file error, and running the precursor again. That should show if there's an issue with the Nalu/Nalu.

pscrozi commented 1 month ago

Nate has been working on different corner cases on the Nalu/Nalu case, nested refinement, etc. Nate has been running cases, but trying to get the full turbine working. Found and worked through an OpenFAST issue, and a segfault on restart, which Nate fixed.

pscrozi commented 3 weeks ago

Should we continue this task, or call it off. Been on the board for 4 months. How much time is Nate spending on this? (Wait for Nate to discuss.)

pscrozi commented 2 weeks ago

Why are we doing this? Phil: we want to evaluate the numerical convergence of our FSI algorithm. We need to evaluate the solver convergence. With Nalu-Nalu, we can do outer iterations and evaluate the convergence of our algorithm.

Where are we on this right now? Ilker is working on AMR-Wind outer iteration capability. Would our resources be better spent there?

There would still be questions about using different discretizations between Nalu-Wind and AMR-Wind.

Nate: we dropped this temporarily due to getting the milestone work done, and this task wasn't on the critical path. But the case is still important, meshed, and ready to go.

Mike: If it can be done in short-order, we should proceed. We want this in a journal article.

Nate: does anyone have a script that can be used to extract iterations from log files? (No answer.)

pscrozi commented 2 weeks ago

No updates

pscrozi commented 6 days ago

Nate: have a Nalu-Nalu running slowly with 15 MW. Need to sit down with some folks to decide what we want to see with a paper on this. We want to try high wind speeds. Might want to track the convergence on lower wind speed cases as well. Explore parameter space: solution iterations, wind speeds, etc.

Phil: run 2 or 3 cases to start. (1) reproduce lower wind speed coupled Nalu-AMR; (2) redo the case that's failing with AMR-Wind that we couldn't do with the high wind speed with ExaWind driver. Nate will run these.