search

Exawind / amr-wind

AMReX-based structured wind solver
https://exawind.github.io/amr-wind
Other
103 stars 78 forks source link

Instability in some simulation #1080

Closed AsimUdel closed 4 weeks ago

AsimUdel commented 1 month ago

Hello All, I have a couple of questions and queries, 1) I have upgraded to AMR-Wind (v.2.1.0) from v.1.0, I am running three different simulation for (stable, neutral and unstable atmospheric stabilities), but I am getting an error in only neutral case. (Basically after some time I am getting "WARNING: fixed_dt does not satisfy CFL condition" and then solution does solve for variables (velcocity, tke, temperature), and my stable and unstable cases successfully completed. (I will attach my input files and screenshot of outputfiles of all the simulations (as those are too big to big). Spinupneutral.txt Spinupstable.txt Spinupunstable.txt Unstable_output Stable_output Neutral_output

2) I am getting a "WARNING: Sampling: netcdf output will negatively impact performance" message in my AMR-Wind(v.2.1.0), did not not get in v.1.0.

3) Due to space storage constraint in my working directory I used to save my (checkpoint and plot files in some other place by using this feature io.check_file = "/lustre/scratch/asim/Wind_Energy_Cases/Stable/chk" io.plot_file = "/lustre/scratch/asim/Wind_Energy_Cases/Stable/plt" my question is, can I use any feature to save (output files( sampling.nc and ablstatistics.nc)) files somewhere else to save space storage in working directory? Can I do as: "sampling.file = "" "

4) Does AMR-Wind only produce (sampling and abl_statistics) output files ?

Thank you so much in advance. Asim

marchdf commented 1 month ago

Hi, thanks for your questions.

  1. This is concerning. Did the same exact input file for the neutral case work with your previous version of amr-wind?
  2. Yes. Netcdf always had performance issues. There was a discussion recently here: https://github.com/Exawind/amr-wind/pull/1052 and here: https://github.com/Exawind/amr-wind/issues/971. We decided to make it explicit that there was a performance issue and signal a warning. If you want to move to speedier sampling: sampling.output_format= native. But the postprocessing tools are lagging so you might have to devise your own. Though the outputs are easily parsed in paraview/visit/yt if you are comfortable with those.
  3. We don't support that feature for sampling files. Would be neat though. Can you create a separate feature request issue with just this?
  4. I am not sure... hard to know all the files produced. Openfast will do a bunch of files, there are error norm files for certain cases, etc.
AsimUdel commented 1 month ago

Hi @marchdf , Thank you for your reply. 1) I have not run cases on the previous version, will do that, then I can confirm and get back to you. 2) I will look into those, thank you. I have not used paraview/visit/yt, I always use python to visualize the netCDF (sampling files). 3)Sure, I will make that. 4) Sure, thank you Best Regards Asim

moprak-nrel commented 1 month ago

  1. I took a quick look at your input files and incflo.velocity is 12.0 for the neutral case, and 7.0 for the other two. This could be a potential reason why that case is failing with a fixed dt while the other two cases are running. The warning seems to indicate that for a CFL of 0.95, the fixed dt set is too high.

  2. Would you be able to copy the amr_wind executable to the path where you want to store the sampling files? This could be a workaround.

AsimUdel commented 1 month ago

@moprak-nrel Thank you 1) Actually I wanted to run Neutral case for low thrust coefficient (at relatively high wind speed than Stable and Unstable), that is why I set wind speed 12 m/s here and 7 m/s in other two cases. I have reduced my dt = 0.25 and have run again now. (Mean while I am running exactly same input file on previous version too) 2) I want to store files in (temporary storage "lustre"), and my amr_wind executable is in my "workgroup" directory, right now I do not know how to link, maybe in "job submission script", if this what you are asking, I am sorry I did not understand it correctly.

moprak-nrel commented 1 month ago

This is just general practice, and not specific to amr_wind, you should be able to move the executable file to the lustre or other termporary storage directories and run from there. Example job script

cp $WORKING_DIR/amr_wind $TEMP_DIR/
cd $TEMP_DIR
mpirun -n 32 ./amr_wind input_file
AsimUdel commented 1 month ago

@moprak-nrel Thank you for the this suggestion.

AsimUdel commented 1 month ago

@marchdf I can confirm that for the previous version (AMR-Wind v.1.0.0) have also got "fixed_dt does not satisfy CFL condition" warning for the exact input file previous_version_amr_wind and solution stopped calculating (variables values). I suspect that by keeping "time.fixed_dt =-1 " will resolve this issue, can someone please correct me if I am wrong.

marchdf commented 1 month ago

Well the good news is that we didn't cause a regression...

yeah you can reduce your dt until you find one that works for these velocities. Or you can have it use a variable dt ("time.fixed_dt =-1") and a CFL condition.

moprak-nrel commented 4 weeks ago

Hi @AsimUdel, did settingfixed_dt = -1 resolve this issue?

AsimUdel commented 4 weeks ago

Hi @moprak-nrel , yes, I have tried running both with fixed_dt = -1 and fixed_dt = 0.25 (half of initial sertting), both completed successfully.