search

AMReX-Astro / Castro

Castro (Compressible Astrophysics): An adaptive mesh, astrophysical compressible (radiation-, magneto-) hydrodynamics simulation code for massively parallel CPU and GPU architectures.
http://amrex-astro.github.io/Castro
Other
300 stars 98 forks source link

Zone Edge Effects in subchandra Castro Runs #550

Closed abigailbishop closed 5 years ago

abigailbishop commented 5 years ago

When running the model in Castro/Exec/science/subchandra on Edison at Nersc and our computers at Stony Brook, the plots of magvel and eint_E (aka eint_e) show inexplicable edge effects on zones. Please see the attached two images of how this runs on Bender at Stony Brook. The magvel plot is on a log scale to better show these effects. They begin as soon as the simulation takes place, allowing them to escalate to dramatic effects with more plot files.

To run this simulation, make sure inputs_2d has amr.plot_int = 1 to plot every plot file, compile the simulation in Castro/Exec/science/subchandra via make, then run the executable via ./Castro2d.gnu.MPI.ex inputs_2d. After 2 plot files, the effects become apparent on the magvel and eint_e plots (you may have to set them to log scale).

subch_plt00002_eint_e_slice subch_plt00002_magvel_slice

zingale commented 5 years ago

Max tracked this down to the lack of an error buffer at jumps in refinement. The fix, which I will check in to the subchandra/ problem is to set

amr.n_error_buf     = 2 2 2 2

in the inputs file (it is currently 0 0 0 0.

zingale commented 5 years ago

@abigailbishop can you update, and make sure your inputs has this change and start up a new run?

abigailbishop commented 5 years ago

For sure! I'm running it now on Edison and Bender and will update with results soon.

abigailbishop commented 5 years ago

That didn't work so @zingale is suggesting adding castro.update_sources_after_reflux = 0 to inputs_2d since he noticed these effects happen at jumps in refinement. He suspects that it may be from a reflux which verifies conservations of certain variables across the jump.

Currently running this on Bender.

abigailbishop commented 5 years ago

On Bender we saw improvements by adding castro.update_sources_after_reflux = 0 to the inputs file (images below). I've submitted jobs to Edison to see the long term difference between a run with castro.update_sources_after_reflux = 0 in the inputs file and without.

The image below shows eint_E in the simulation after 1 time step without the reflux added to the inputs file. subch_plt00001_eint_e_slice_withoutReflux

The image below shows eint_E in the simulation after 1 time step with the reflux added to the inputs file. There are faint signatures of effects at (r=3E8, z=0.2E9) and again at (5.5E8, 1.1E9). subch_plt00001_eint_e_slice_withReflux

maxpkatz commented 5 years ago

Do you see this effect with castro.do_react = 0?

abigailbishop commented 5 years ago

It's currently set to 1, but I just kicked off a run with castro.do_react = 0 and castro.update_sources_after_reflux = 0 so we can see how that goes.

abigailbishop commented 5 years ago

TLDR: Adding castro.update_sources_after_reflux = 0 and updating to castro.do_react = 1 in the inputs file seems to resolve the issue.

Below is an image of the first plot file of a simulation without castro.update_sources_after_reflux = 0 and castro.do_react = 1

subch_plt00001_eint_e_slice_withoutReflux

Below is an image of the first plot file with castro.update_sources_after_reflux = 0 and castro.do_react = 1.

subch_plt00001_eint_e_slice_withReflux

Lastly, below is an image of the first plot file with castro.update_sources_after_reflux = 0 and castro.do_react = 0. Note that the latter option means that there will be no reactions.

subch_plt00001_eint_e_slice-with_reflux_and_doReact0

The simulations of all scenarios have gotten farther than before, up to about plot file 85,000, before quitting where the first scenario gave the error of low pressure and rho_e we think is associate with the reflux problem and the second scenario had the excuse of a seg fault. The third quit at 15,000 because that what the inputs file said to do, and I have since increased that and am running it with the hopes that it runs far. Also, now that I've updated Castro, Microphysics, and Amrex (which are all on the development branch), they are all running the same simulation.

zingale commented 5 years ago

can you also try running with:

gravity.gravity_type = PoissonGrav instead of MonopoleGrav?

abigailbishop commented 5 years ago

It looks like gravity.gravity_type = PoissonGrav coupled with the castro.update_sources_after_reflux = 0 and castro.do_react = 0 causes goofy trends. Here's the eint_e plot for the first plot file:

subch_plt00001_eint_e_slicesubch_PoissonGrav

I ran it with this command: mpiexec -n 6 ./Castro2d.gnu.MPI.ex inputs_2d.txt And with this inputs file: inputs_2d.txt

zingale commented 5 years ago

the issue appears to be due to castro.small_dens being the same as the lowest density in the initial model. So we are seeing the effect of density resets. Not sure why they are so dramatic though. In my runs (with only 2 levels of refinement), if I set castro.small_dens = 1.e-5, then all the funniness went away.

abigailbishop commented 5 years ago

The Castro.small_dens along with castro.update_sources_after_reflux = 0 have resolved the issue! A current run has made it 45,000 time steps without any weird effects on the magvel plot.