Open rybchuk opened 2 weeks ago
I did some more investigating, but have not been able to figure out a quick fix. I changed the time tolerance from 1e-12
to the aggressive 1e-4
, and that didn't work.
what(): Assertion `std::abs(time - m_in_data.tinterp()) < 1e-4' failed, file "/kfs2/projects/ai4wind/orybchuk/exawind-manager/environments/ai4wind-nov24/amr-wind/amr-wind/wind_energy/ABLBoundaryPlane.cpp", line 900
I also added some print statements into ABLBoundaryPlane.cpp
to see the values of everything:
amrex::Print() << "time: " << time << std::endl;
amrex::Print() << "m_in_data.tn(): " << m_in_data.tn() << std::endl;
amrex::Print() << "m_in_data.tnp1(): " << m_in_data.tnp1() << std::endl;
amrex::Print() << "m_in_data.tinterp(): " << m_in_data.tinterp() << std::endl;
amrex::Print() << "std::abs(time - m_in_data.tinterp()): " << std::abs(time - m_in_data.tinterp()) << std::endl;
AMREX_ALWAYS_ASSERT(
((m_in_data.tn() <= time) || (time <= m_in_data.tnp1())));
AMREX_ALWAYS_ASSERT(std::abs(time - m_in_data.tinterp()) < 1e-6);
Surprisingly, the values of time
and m_in_data.tinterp()
are printed out as identical and the difference truly appears to be 0
:
time: 6542.942863
m_in_data.tn(): -1
m_in_data.tnp1(): -1
m_in_data.tinterp(): 6542.942863
std::abs(time - m_in_data.tinterp()): 0
I assume that m_in_data.tn()
and m_in_data.tnp1()
are supposed to be -1
?
One more update: if I comment out AMREX_ALWAYS_ASSERT(std::abs(time - m_in_data.tinterp()) < 1e-6);
, then my simulation does run through to timestepping. It's unclear yet if I'm seeing any unphysical artifacts.
I'm confident I found the problem. The time passed to a vof fillpatch (within the advection step) is not valid, leading to a problem with this assertion. I'm not sure what the best solution is, but to confirm this is the problem, you could try running without vof as one of the bndry variables. This would also explain why this problem is showing up for offshore ABLs in particular.
Yep, I can confirm! If I run with my code that was compiled back in October and I set ABL.bndry_var_names = velocity temperature
, the code makes it through the first few timesteps without any errors.
Putting these here for myself and other developers: https://github.com/Exawind/amr-wind/blob/main/amr-wind/equation_systems/vof/SplitAdvection.cpp#L102 https://github.com/Exawind/amr-wind/blob/main/amr-wind/equation_systems/vof/vof_ops.H#L73
both of these would lead to breaking the assertion. The second one is easy to fix. The first one is a problem for two reasons:
When you get a chance, could you also check with the more recent code, and just taking vof out of the input bndry variable names? No hurry
It's a quick thing for me to check :) I un-uncommented the AMREX_ALWAYS_ASSERT
with yesterday's code, and I can confirm that that also runs through the first few timestamps without issue. And, the code fails if I add vof
back into the BCs.
Adding to our previous discussion, I randomly get a code crash at :
AMREX_ALWAYS_ASSERT(
((m_in_data.tn() <= time) || (time <= m_in_data.tnp1())));
It happens on both CPUs and GPUs. I would say 10% of my cases see this issue.
@hgopalan could you make a separate issue with a testable example? That's a different problem than what Alex is seeing.
So... when I submitted the issue, I unfortunately set up my inflow-outflow simulation incorrectly. While I set ABL.bndry_io_mode = 1
, I was also running geometry.is_periodic = 1 1 0
. As a result, it's ambiguous if I was accidentally running my inflow-outflow simulation as a periodic simulation.
I'm now trying to run the inflow-outflow simulation with:
ABL.bndry_io_mode = 1
ABL.bndry_planes = xlo
...
geometry.is_periodic = 0 1 0
xlo.type = mass_inflow
xlo.density = 1.027
xlo.temperature = 265.0
xlo.tke = 0.0
xhi.type = pressure_outflow
The simulation now errors out with
AMReX (24.09-45-g6d9c25b989f1) initialized
Initializing AMR-Wind ...
ParmParse::sgetarr ParmParse::sgetarr(): xlo.vof not found in table
This error is being reported by amrex/Src/Base/AMReX_ParmParse.cpp
. Do you have any suggestions on what to do about xlo.vof
? I don't see any examples of this line in the repo. Also, what should I do about xlo.density
when MultiPhase
expects two densities? And should xhi.type
still be pressure_outflow
?
Harish suggested setting xlo.vof = 0
(where 0 is a dummy value), and my simulation is now running again :) I hope to have results soon with a longer inflow-outflow simulation
Thanks, @hgopalan, that's a great suggestion for this scenario
I ran the simulation out for 200 iterations (10 seconds). Based off the vof field by the end of the simulation, the numerical beach is indeed damping the waves. Here's pressure at that last step too.
Unfortunately, u, v, w, and T look funny at the outflow. This feels like I botched something with the inflow-outflow .i
file though. I haven't run inflow-outflow simulations in a bit, so let me confirm that I remember how to properly run them on land.
are you using ABL Mean Boussinesq in there too?
Nope, good catch.
I've added ABL Mean Boussinesq in there. I've also run the calc_inflow_stats.py
script but commented out all the ABL
lines because these values feel like they're specific to land-based ABLs with Monin-Obukhov.
ICNS.source_terms = BoussinesqBuoyancy CoriolisForcing GeostrophicForcing GravityForcing ABLMeanBoussinesq
##--------- Additions by calc_inflow_stats.py ---------#
# ABL.wall_shear_stress_type = "local"
# ABL.inflow_outflow_mode = true
# ABL.wf_velocity = -0.04061319424974897 0.01560499433783477
# ABL.wf_vmag = 0.05284818547325939
# ABL.wf_theta = 265.0000040814566
# BodyForce.magnitude = 0.0 0.0 0.0
BoussinesqBuoyancy.read_temperature_profile = true
BoussinesqBuoyancy.tprofile_filename = avg_theta.dat
##-----------------------------------------------------#
After making that change, my inflow-outflow simulation ran for the full requested two minutes without any obvious issues (u, v, w, T)!
Whereas the above simulations didn't have any refinement zones, I also ran ~20 seconds of an inflow-outflow simulation with two levels of refinement. The u, v, w, and T fields also look good there!
I've attached the read and write .i
files below for reference for posterity.
offshore_io.zip
I'm going to do some more futzing around now, where I take BCs from a spun-up simulation and I try to drive a simulation with quiescent ICs, but that's not relevant to this issue. The VoF fillpatch workaround that I've been using looks like it works great for the time being, so there's no pressing need from me to thinking about the VoF fillpatch code more rigorously. Thanks for the help!
That's awesome news @rybchuk . Good to close the issue then?
Bug description
Likely related to this PR, and I am using the most recent commit on
main
,740679c8
. I'm running an offshore ABL simulation with linear waves. I have achieved a spun-up state at iteration80000
, and now I am trying to run an inflow-outflow simulation. I believe that I have successfully written out boundary planes using the following:(I know that I'm probably saving out too many boundary variables, but I saw the same error with
= velocity temperature vof
)I then kick off the inflow-outflow simulation using the following variables:
The simulation errors out during the first timestep.
Steps to reproduce
I have shared all the files needed to run the bc-write simulation and bc-read simulation in
/scratch/orybchuk/share/debug_inflow_outflow
on Kestrel. I've uploaded many of the files (aside from the large ones) here too.debug_inflow_outflow.zip
Steps to reproduce the behavior:
.zip
file.Godunov: System Iters Initial residual Final residual
terminate called after throwing an instance of 'std::runtime_error' ... what(): Assertion `std::abs(time - m_in_data.tinterp()) < 1e-12' failed, file "/kfs2/projects/ai4wind/orybchuk/exawind-manager/environments/ai4wind-nov24/amr-wind/amr-wind/wind_energy/ABLBoundaryPlane.cpp", line 896
============================================================================== AMR-Wind (https://github.com/exawind/amr-wind)
AMR-Wind version :: v3.2.0-8-g740679c8-DIRTY AMR-Wind Git SHA :: 740679c87925db41521f477e8724daa0c69cc670-DIRTY AMReX version :: 24.09-45-g6d9c25b989f1
Exec. time :: Mon Nov 11 10:48:20 2024 Build time :: Nov 11 2024 10:17:52 C++ compiler :: IntelLLVM 2023.2.0
MPI :: ON (Num. ranks = 384) GPU :: OFF OpenMP :: OFF
Enabled third-party libraries: NetCDF 4.9.2
See https://github.com/Exawind/amr-wind/blob/development/LICENSE for details.
80000 6542.4428627258721 80001 6542.5721400248704 80002 6542.6563112443018