search

AMReX-Astro / Nyx

An adaptive mesh, N-body hydro cosmological simulation code
https://amrex-astro.github.io/Nyx/
Other
90 stars 36 forks source link

issues to run the MiniSB experiment #73

Closed wangzhezhe closed 3 years ago

wangzhezhe commented 3 years ago

Hello, I'm a new user for NYX, I tried to run the MiniSB code to get familiar with the nyx. I got this error when I run it on Cori cluster by this way:

$srun -n 2 -C haswell nyx_MiniSB ~/cworkspace/src/Nyx/Exec/MiniSB/inputs.32
MPI initialized with 2 MPI processes
MPI initialized with thread support level 0
OMP initialized with 32 OMP threads
AMReX (21.02-10-geb9b4242a070) initialized
Successfully read inputs file ... 

AMReX git describe: 21.02-10-geb9b4242a070

Nyx git describe:   21.02-dirty
amrex::Error::0::Nyx::cant have do_hydro == 0 and do_santa_barbara == 1 !!!
SIGABRT
amrex::Error::1::Nyx::cant have do_hydro == 0 and do_santa_barbara == 1 !!!
SIGABRT
See Backtrace.1.0 file for details
See Backtrace.0.0 file for details
Rank 1 [Mon Feb  8 19:54:04 2021] [c4-0c0s2n2] application called MPI_Abort(MPI_COMM_WORLD, 6) - process 1
Rank 0 [Mon Feb  8 19:54:04 2021] [c4-0c0s2n2] application called MPI_Abort(MPI_COMM_WORLD, 6) - process 0
srun: error: nid00778: tasks 0-1: Aborted
srun: Terminating job step 38955083.0
$ cat ~/cworkspace/src/Nyx/Exec/MiniSB/inputs.32 | grep "do_hydro"
nyx.do_hydro = 1
$ cat ~/cworkspace/src/Nyx/Exec/MiniSB/inputs.32 | grep "do_santa_barbara"
nyx.do_santa_barbara = 1

Then I tried to add 

pp.query("do_hydro", do_hydro);

at this line (https://github.com/AMReX-Astro/Nyx/blob/development/Source/Initialization/Nyx_initdata.cpp#L77) it seems code could move forward, but I got this output now:

MPI initialized with 2 MPI processes
MPI initialized with thread support level 0
OMP initialized with 32 OMP threads
AMReX (21.02-10-geb9b4242a070) initialized
Successfully read inputs file ... 

AMReX git describe: 21.02-10-geb9b4242a070

Nyx git describe:   21.02-dirty
Successfully read inputs file ... 
Getting Gconst from nyx_constants: 4.300927161e-09
Using -5.404704468e-08 for 4 pi G in Gravity.cpp 
Initializing the data at level 0

Initializing DM particles from "ic_sb_32.ascii" ...

Redistributing from processor 0 to 0
ParticleContainer byte spread across MPI nodes: [0 (0) ... 653720 (16343)] total particles: (16343)
ParticleContainer::Redistribute() time: 0.016965574

Redistributing from processor 1 to 1
ParticleContainer byte spread across MPI nodes: [0 (0) ... 1307440 (32686)] total particles: (32686)
ParticleContainer::Redistribute() time: 0.004900744

Total number of particles: 32686
Done initializing the level 0 data
...estdt from particles at level 0: 9.238322837e-05
...estdt after call to comoving   : 3.91455618e-07
...change in a is 0.9996324954 percent
Nyx::est_time_step at level 0:  estdt = 3.91455618e-07
Integrating a from time 0 by dt = 3.91455618e-07
Old / new A time      0 3.91455618e-07
Old / new A           0.015625 0.01578119258
Old / new z           63 62.3665672
Setting the current time in the state data to 0
BA SIZE 0
FLEV    0
Calling redistribute because the size of BoxArray changed 
ParticleContainer byte spread across MPI nodes: [0 (0) ... 1307440 (32686)] total particles: (32686)
ParticleContainer::Redistribute() time: 0.002811674

Gravity ... defining average density in Gravity::set_mass_offset to be 6.92105131e+10
Gravity ... multilevel solve for new phi at base level 0 to finest level 0
 ... subtracting average density 6.92105131e+10 from RHS at each level 
 ... subtracting -3.147870302e-07 to ensure solvability 
INITIAL GRIDS 
  Level 0   1 grids  32768 cells  100 % of domain

CHECKPOINT: file = chk00000
checkPoint() time = 0.018486771 secs.
PLOTFILE: file = plt00000
Output file plt00000.temp at time 0.015625 and step 0
Write plotfile time = 0.020314764  seconds

...estdt from particles at level 0: 9.238322837e-05
...advancing from old_a_time rather than new_a_time! 
amrex::Abort::1::In `advance_particles_only` but `do_hydro` is true !!!
SIGABRT
...estdt after call to comoving   : 3.91455618e-07
...change in a is 0.9996324954 percent
Nyx::est_time_step at level 0:  estdt = 3.91455618e-07
Re-integrating a from time 0 by dt = 3.91455618e-07
Old / new A time         0 3.91455618e-07
Old / new A              0.015625 0.01578119258
Old / new z              63 62.3665672
[Level 0 step 1] ADVANCE with dt = 3.91455618e-07
amrex::Abort::0::In `advance_particles_only` but `do_hydro` is true !!!
SIGABRT

Could you provide some ideas for solving this issue and make the MiniSB example works? Thanks for your help!

asalmgren commented 3 years ago

You should not have needed to add that line in the code -- we already read "do_hydro" at line 432 of Source/Driver/Nyx.cpp

Please do "git pull" and make sure you have the latest version in the development branch.

The line "nyx.do_hydro = 1" is in the inputs files provided. The examples in MiniSB should "just work" :-)

On Mon, Feb 8, 2021 at 7:39 PM wangzhezhe notifications@github.com wrote:

Hello, I'm a new user for NYX, I tried to run the MiniSB code to get familiar with the nyx. I got this error when I run it on Cori cluster:

MPI initialized with 2 MPI processes MPI initialized with thread support level 0 OMP initialized with 32 OMP threads AMReX (21.02-10-geb9b4242a070) initialized Successfully read inputs file ... amrex::Error::1::Nyx::cant have do_hydro == 0 and do_santa_barbara == 1 !!! SIGABRT

Then I tried to add

pp.query("do_hydro", do_hydro);

at this line ( https://github.com/AMReX-Astro/Nyx/blob/development/Source/Initialization/Nyx_initdata.cpp#L77 ) it seems code could move forward, but I got this output now:

MPI initialized with 2 MPI processes MPI initialized with thread support level 0 OMP initialized with 32 OMP threads AMReX (21.02-10-geb9b4242a070) initialized Successfully read inputs file ...

AMReX git describe: 21.02-10-geb9b4242a070

Nyx git describe: 21.02-dirty Successfully read inputs file ... Getting Gconst from nyx_constants: 4.300927161e-09 Using -5.404704468e-08 for 4 pi G in Gravity.cpp Initializing the data at level 0

Initializing DM particles from "ic_sb_32.ascii" ...

Redistributing from processor 0 to 0 ParticleContainer byte spread across MPI nodes: [0 (0) ... 653720 (16343)] total particles: (16343) ParticleContainer::Redistribute() time: 0.016965574

Redistributing from processor 1 to 1 ParticleContainer byte spread across MPI nodes: [0 (0) ... 1307440 (32686)] total particles: (32686) ParticleContainer::Redistribute() time: 0.004900744

Total number of particles: 32686 Done initializing the level 0 data ...estdt from particles at level 0: 9.238322837e-05 ...estdt after call to comoving : 3.91455618e-07 ...change in a is 0.9996324954 percent Nyx::est_time_step at level 0: estdt = 3.91455618e-07 Integrating a from time 0 by dt = 3.91455618e-07 Old / new A time 0 3.91455618e-07 Old / new A 0.015625 0.01578119258 Old / new z 63 62.3665672 Setting the current time in the state data to 0 BA SIZE 0 FLEV 0 Calling redistribute because the size of BoxArray changed ParticleContainer byte spread across MPI nodes: [0 (0) ... 1307440 (32686)] total particles: (32686) ParticleContainer::Redistribute() time: 0.002811674

Gravity ... defining average density in Gravity::set_mass_offset to be 6.92105131e+10 Gravity ... multilevel solve for new phi at base level 0 to finest level 0 ... subtracting average density 6.92105131e+10 from RHS at each level ... subtracting -3.147870302e-07 to ensure solvability INITIAL GRIDS Level 0 1 grids 32768 cells 100 % of domain

CHECKPOINT: file = chk00000 checkPoint() time = 0.018486771 secs. PLOTFILE: file = plt00000 Output file plt00000.temp at time 0.015625 and step 0 Write plotfile time = 0.020314764 seconds

...estdt from particles at level 0: 9.238322837e-05 ...advancing from old_a_time rather than new_a_time! amrex::Abort::1::In advance_particles_only but do_hydro is true !!! SIGABRT ...estdt after call to comoving : 3.91455618e-07 ...change in a is 0.9996324954 percent Nyx::est_time_step at level 0: estdt = 3.91455618e-07 Re-integrating a from time 0 by dt = 3.91455618e-07 Old / new A time 0 3.91455618e-07 Old / new A 0.015625 0.01578119258 Old / new z 63 62.3665672 [Level 0 step 1] ADVANCE with dt = 3.91455618e-07 amrex::Abort::0::In advance_particles_only but do_hydro is true !!! SIGABRT

Could you provide some ideas for solving this issue and make the MiniSB example works? Thanks for your help!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/AMReX-Astro/Nyx/issues/73, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YTHRZH6NWW3OU7HXNDS6CVAZANCNFSM4XKGRJPQ .

-- Ann Almgren Senior Scientist; CCSE Group Lead

wangzhezhe commented 3 years ago

Thanks for the reply, the latest log for my current source code is 

commit db3549343e22d9c86592677a732dc8f834b6e4d5 (HEAD -> development, origin/development, origin/HEAD)
Author: Ann Almgren <asalmgren@lbl.gov>
Date:   Mon Feb 8 16:34:38 2021 -0800

    Add Andrew Myers

but I still get this error:

srun -n 2 -C haswell nyx_MiniSB ~/cworkspace/src/Nyx/Exec/MiniSB/inputs.32
MPI initialized with 2 MPI processes
MPI initialized with thread support level 0
OMP initialized with 32 OMP threads
AMReX (21.02-10-geb9b4242a070) initialized
Successfully read inputs file ... 
amrex::Error::1::Nyx::cant have do_hydro == 0 and do_santa_barbara == 1 !!!
SIGABRT

AMReX git describe: 21.02-10-geb9b4242a070

Nyx git describe:   21.02-17-gdb3549343e22
amrex::Error::0::Nyx::cant have do_hydro == 0 and do_santa_barbara == 1 !!!
SIGABRT

and there are:

$ cat ~/cworkspace/src/Nyx/Exec/MiniSB/inputs.32 | grep do_hydro
nyx.do_hydro = 1
$ cat ~/cworkspace/src/Nyx/Exec/MiniSB/inputs.32 | grep nyx.do_santa_barbara
nyx.do_santa_barbara = 1
asalmgren commented 3 years ago

I've just successfully built and run on my workstation with 2 MPI ranks, with and without OpenMP.

Everything "just worked" as expected.

I'm not sure what's going on.

Can you try running with just one OMP thread and see if that makes any difference?

On Mon, Feb 8, 2021 at 8:20 PM wangzhezhe notifications@github.com wrote:

Thanks for the reply, the latest log for my current source code is

commit db3549343e22d9c86592677a732dc8f834b6e4d5 (HEAD -> development, origin/development, origin/HEAD) Author: Ann Almgren asalmgren@lbl.gov Date: Mon Feb 8 16:34:38 2021 -0800

Add Andrew Myers

but I still get this error:

srun -n 2 -C haswell nyx_MiniSB ~/cworkspace/src/Nyx/Exec/MiniSB/inputs.32 MPI initialized with 2 MPI processes MPI initialized with thread support level 0 OMP initialized with 32 OMP threads AMReX (21.02-10-geb9b4242a070) initialized Successfully read inputs file ... amrex::Error::1::Nyx::cant have do_hydro == 0 and do_santa_barbara == 1 !!! SIGABRT

AMReX git describe: 21.02-10-geb9b4242a070

Nyx git describe: 21.02-17-gdb3549343e22 amrex::Error::0::Nyx::cant have do_hydro == 0 and do_santa_barbara == 1 !!! SIGABRT

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AMReX-Astro/Nyx/issues/73#issuecomment-775652049, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YQS4YG6AJMQBQJ7ZHDS6CZ2PANCNFSM4XKGRJPQ .

-- Ann Almgren Senior Scientist; CCSE Group Lead

wangzhezhe commented 3 years ago

yeah, it looks weird:

srun -n 1 -C haswell nyx_MiniSB ~/cworkspace/src/Nyx/Exec/MiniSB/inputs.32
srun: job 38959759 queued and waiting for resources
srun: job 38959759 has been allocated resources
MPI initialized with 1 MPI processes
MPI initialized with thread support level 0
OMP initialized with 1 OMP threads
AMReX (21.02-10-geb9b4242a070) initialized
Successfully read inputs file ... 

AMReX git describe: 21.02-10-geb9b4242a070

Nyx git describe:   21.02-17-gdb3549343e22
amrex::Error::0::Nyx::cant have do_hydro == 0 and do_santa_barbara == 1 !!!
SIGABRT
See Backtrace.0.0 file for details
Rank 0 [Mon Feb  8 21:53:45 2021] [c4-0c0s10n2] application called MPI_Abort(MPI_COMM_WORLD, 6) - process 0
srun: error: nid00810: task 0: Aborted
srun: Terminating job step 38959759.0

I just use all default options when I build the code based on cmake, is there particular option that I should set?

WeiqunZhang commented 3 years ago

This is a cmake issue.

wangzhezhe commented 3 years ago

I added two log messages in the function read_hydro_params (in the Nyx.cpp) and function read_init_params (in the Nyx_initdata.cpp), but there is only this one:

debug do_hydro in read_init_params 0

and it looks the read_hydro_params is not called for my code

WeiqunZhang commented 3 years ago

If you use gnumake, it will just work.

wangzhezhe commented 3 years ago

let me try it, thanks

WeiqunZhang commented 3 years ago

Probably because cmake builds the test with -DNO_HYDRO.

wangzhezhe commented 3 years ago

Thanks for the information, I set the Nyx_HYDRO as ON this time, and there are following issues, maybe I could try gnumake.

$srun -n 1 -C haswell nyx_MiniSB ~/cworkspace/src/Nyx/Exec/MiniSB/inputs.32
MPI initialized with 1 MPI processes
MPI initialized with thread support level 0
OMP initialized with 1 OMP threads
AMReX (21.02-10-geb9b4242a070) initialized
Successfully read inputs file ... 

AMReX git describe: 21.02-10-geb9b4242a070

Nyx git describe:   21.02-17-gdb3549343e22-dirty
debug do_hydro in read_hydro_params 1
ParmParse::getval ParmParse::getval(): nyx.do_forcing not found in table
max_step(nvals = 1)  :: [2000]
max_step(nvals = 1)  :: [10]
nyx.h_species(nvals = 1)  :: [0.75]
nyx.he_species(nvals = 1)  :: [0.25]
amr.refine_grid_layout(nvals = 1)  :: [0]
amr.checkpoint_files_output(nvals = 1)  :: [1]
amr.plot_files_output(nvals = 1)  :: [1]
nyx.ppm_type(nvals = 1)  :: [0]
nyx.small_dens(nvals = 1)  :: [6921.0511053125001]
nyx.small_temp(nvals = 1)  :: [1e-3]
nyx.small_pres(nvals = 1)  :: [9.7106852723886791E-002]
nyx.do_santa_barbara(nvals = 1)  :: [1]
gravity.sl_tol(nvals = 1)  :: [1.e-12]
nyx.initial_z(nvals = 1)  :: [63.]
nyx.final_a(nvals = 1)  :: [1.0]
amr.data_log(nvals = 1)  :: [runlog]
gravity.no_sync(nvals = 1)  :: [1]
gravity.no_composite(nvals = 1)  :: [1]
geometry.is_periodic(nvals = 3)  :: [1, 1, 1]
geometry.coord_sys(nvals = 1)  :: [0]
geometry.prob_lo(nvals = 3)  :: [0, 0, 0]
geometry.prob_hi(nvals = 3)  :: [64., 64., 64.]
amr.n_cell(nvals = 3)  :: [32, 32, 32]
amr.max_grid_size(nvals = 1)  :: [16]
amr.max_grid_size(nvals = 1)  :: [32]
amr.subcycling_mode(nvals = 1)  :: [Optimal]
particles.replicate(nvals = 1)  :: [(1,1,1)]
nyx.lo_bc(nvals = 3)  :: [0, 0, 0]
nyx.hi_bc(nvals = 3)  :: [0, 0, 0]
nyx.do_hydro(nvals = 1)  :: [1]
nyx.do_grav(nvals = 1)  :: [1]
nyx.comoving_OmM(nvals = 1)  :: [1.0]
nyx.comoving_OmB(nvals = 1)  :: [0.1]
nyx.comoving_h(nvals = 1)  :: [0.5e0]
nyx.do_dm_particles(nvals = 1)  :: [1]
nyx.particle_init_type(nvals = 1)  :: [AsciiFile]
nyx.ascii_particle_file(nvals = 1)  :: [ic_sb_32.ascii]
particles.aggregation_type(nvals = 1)  :: [Cell]
particles.aggregation_buffer(nvals = 1)  :: [2]
nyx.cfl(nvals = 1)  :: [0.9]
nyx.init_shrink(nvals = 1)  :: [1.0]
nyx.change_max(nvals = 1)  :: [1.1]
nyx.dt_cutoff(nvals = 1)  :: [5.e-20]
nyx.sum_interval(nvals = 1)  :: [-1]
nyx.v(nvals = 1)  :: [1]
gravity.v(nvals = 1)  :: [1]
amr.v(nvals = 1)  :: [1]
mg.v(nvals = 1)  :: [0]
particles.v(nvals = 1)  :: [1]
amr.max_level(nvals = 1)  :: [0]
amr.ref_ratio(nvals = 4)  :: [2, 2, 2, 2]
amr.regrid_int(nvals = 4)  :: [2, 2, 2, 2]
amr.n_error_buf(nvals = 4)  :: [2, 2, 2, 2]
amr.grid_eff(nvals = 1)  :: [0.7]
amr.blocking_factor(nvals = 1)  :: [8]
amr.check_file(nvals = 1)  :: [chk]
amr.check_int(nvals = 1)  :: [100]
amr.plot_file(nvals = 1)  :: [plt]
amr.plot_int(nvals = 1)  :: [100]
amr.plot_vars(nvals = 1)  :: [ALL]
amr.derive_plot_vars(nvals = 4)  :: [particle_count, particle_mass_density, pressure, magvel]
amr.probin_file(nvals = 1)  :: []
amrex::Abort::SIGABRT
asalmgren commented 3 years ago

Yes, please use gmake.

It looks like we have the wrong default options with cmake.

On Tue, Feb 9, 2021 at 12:07 AM wangzhezhe notifications@github.com wrote:

Thanks for the information, I set the Nyx_HYDRO as ON this time, and there are following errors, maybe I could try gnumake.

srun -n 1 -C haswell nyx_MiniSB ~/cworkspace/src/Nyx/Exec/MiniSB/inputs.32 MPI initialized with 1 MPI processes MPI initialized with thread support level 0 OMP initialized with 1 OMP threads AMReX (21.02-10-geb9b4242a070) initialized Successfully read inputs file ...

AMReX git describe: 21.02-10-geb9b4242a070

Nyx git describe: 21.02-17-gdb3549343e22-dirty debug do_hydro in read_hydro_params 1 ParmParse::getval ParmParse::getval(): nyx.do_forcing not found in table max_step(nvals = 1) :: [2000] max_step(nvals = 1) :: [10] nyx.h_species(nvals = 1) :: [0.75] nyx.he_species(nvals = 1) :: [0.25] amr.refine_grid_layout(nvals = 1) :: [0] amr.checkpoint_files_output(nvals = 1) :: [1] amr.plot_files_output(nvals = 1) :: [1] nyx.ppm_type(nvals = 1) :: [0] nyx.small_dens(nvals = 1) :: [6921.0511053125001] nyx.small_temp(nvals = 1) :: [1e-3] nyx.small_pres(nvals = 1) :: [9.7106852723886791E-002] nyx.do_santa_barbara(nvals = 1) :: [1] gravity.sl_tol(nvals = 1) :: [1.e-12] nyx.initial_z(nvals = 1) :: [63.] nyx.final_a(nvals = 1) :: [1.0] amr.data_log(nvals = 1) :: [runlog] gravity.no_sync(nvals = 1) :: [1] gravity.no_composite(nvals = 1) :: [1] geometry.is_periodic(nvals = 3) :: [1, 1, 1] geometry.coord_sys(nvals = 1) :: [0] geometry.prob_lo(nvals = 3) :: [0, 0, 0] geometry.prob_hi(nvals = 3) :: [64., 64., 64.] amr.n_cell(nvals = 3) :: [32, 32, 32] amr.max_grid_size(nvals = 1) :: [16] amr.max_grid_size(nvals = 1) :: [32] amr.subcycling_mode(nvals = 1) :: [Optimal] particles.replicate(nvals = 1) :: [(1,1,1)] nyx.lo_bc(nvals = 3) :: [0, 0, 0] nyx.hi_bc(nvals = 3) :: [0, 0, 0] nyx.do_hydro(nvals = 1) :: [1] nyx.do_grav(nvals = 1) :: [1] nyx.comoving_OmM(nvals = 1) :: [1.0] nyx.comoving_OmB(nvals = 1) :: [0.1] nyx.comoving_h(nvals = 1) :: [0.5e0] nyx.do_dm_particles(nvals = 1) :: [1] nyx.particle_init_type(nvals = 1) :: [AsciiFile] nyx.ascii_particle_file(nvals = 1) :: [ic_sb_32.ascii] particles.aggregation_type(nvals = 1) :: [Cell] particles.aggregation_buffer(nvals = 1) :: [2] nyx.cfl(nvals = 1) :: [0.9] nyx.init_shrink(nvals = 1) :: [1.0] nyx.change_max(nvals = 1) :: [1.1] nyx.dt_cutoff(nvals = 1) :: [5.e-20] nyx.sum_interval(nvals = 1) :: [-1] nyx.v(nvals = 1) :: [1] gravity.v(nvals = 1) :: [1] amr.v(nvals = 1) :: [1] mg.v(nvals = 1) :: [0] particles.v(nvals = 1) :: [1] amr.max_level(nvals = 1) :: [0] amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] amr.n_error_buf(nvals = 4) :: [2, 2, 2, 2] amr.grid_eff(nvals = 1) :: [0.7] amr.blocking_factor(nvals = 1) :: [8] amr.check_file(nvals = 1) :: [chk] amr.check_int(nvals = 1) :: [100] amr.plot_file(nvals = 1) :: [plt] amr.plot_int(nvals = 1) :: [100] amr.plot_vars(nvals = 1) :: [ALL] amr.derive_plot_vars(nvals = 4) :: [particle_count, particle_mass_density, pressure, magvel] amr.probin_file(nvals = 1) :: [] amrex::Abort::SIGABRT

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AMReX-Astro/Nyx/issues/73#issuecomment-775750779, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YR6KOGYJDFO6TTXSPDS6DUNXANCNFSM4XKGRJPQ .

-- Ann Almgren Senior Scientist; CCSE Group Lead