enzo-project / enzo-e

A version of Enzo designed for exascale and built on charm++.
Other
29 stars 35 forks source link

Building Enzo-E with CMake #79

Closed forrestglines closed 2 years ago

forrestglines commented 3 years ago

In preparation for incorporating Kokkos into Enzo-E, I need to get Enzo-E compiling and running with a build system that is compatible with Kokkos - either cmake or plain Makefile. cmake should be more flexible.

I've started a branch here on my fork of Enzo-E for building Enzo-E with cmake. However, it's currently not working, even with a charm++ build that works fine with the master SCons built Enzo-E repo.

From a high level, my cmake does these things to successfully compile Enzo-E (although the executable is broken).

  1. Compile a libexternal_library.a from all the Source and header files in src/External using g++
  2. Generate *.decl.h and *.def.h headers from all the *.ci files in src/ using charmc
  3. Compile a libcello_library.a from most of the source and header files in src/Cello using g++/gcc/gfortran
  4. Compile object files from most of the source and header files in src/Enzo using g++/gcc/gfortran
  5. Link libexternal_library.a, libcello_library.a, and the source files from src/Enzo together into an executable using charmc, also linking with dl, pthread, z, hdf5, png, gfortran, boost

However, the resulting executable still doesn't work

$CHARM_DIR/bin/charmrun +p4 src/enzo_e_exe input/test_cosmo-bcg.in
Running on 4 processors:  src/enzo_e_exe input/test_cosmo-bcg.in 
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 4  src/enzo_e_exe input/test_cosmo-bcg.in 
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: MPI_THREAD_SINGLE (desired: MPI_THREAD_SINGLE)
Charm++> Running in non-SMP mode: 4 processes (PEs)
Converse/Charm++ Commit ID: v6.10.2-0-g7bf00fad3
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 hosts (1 sockets x 6 cores x 2 PUs = 12-way SMP)
Charm++> cpu topology info is gathered in 0.000 seconds.
0 00000.00 Parameters  Adapt:list = [ "mass_baryon" "mass_dark" ]
0 00000.00 Parameters  Adapt:mass_baryon:field_list = [ "density" ]
0 00000.00 Parameters  Adapt:mass_baryon:mass_type = "baryon"
0 00000.00 Parameters  Adapt:mass_baryon:max_coarsen = 0.1250000000000000
0 00000.00 Parameters  Adapt:mass_baryon:min_refine = 8.000000000000000
0 00000.00 Parameters  Adapt:mass_baryon:type = "mass"
0 00000.00 Parameters  Adapt:mass_dark:field_list = [ "density_particle_accumulate" ]
0 00000.00 Parameters  Adapt:mass_dark:mass_type = "dark"
0 00000.00 Parameters  Adapt:mass_dark:max_coarsen = 0.1250000000000000
0 00000.00 Parameters  Adapt:mass_dark:min_refine = 8.000000000000000
0 00000.00 Parameters  Adapt:mass_dark:type = "mass"
0 00000.00 Parameters  Adapt:max_initial_level = 0
0 00000.00 Parameters  Adapt:max_level = 4
0 00000.00 Parameters  Adapt:min_level = 0
0 00000.00 Parameters  Balance:start = 10
0 00000.00 Parameters  Balance:value = 20
0 00000.00 Parameters  Balance:var = "cycle"
0 00000.00 Parameters  Boundary:type = "periodic"
0 00000.00 Parameters  Domain:lower = [ 0.000000000000000 0.000000000000000 0.000000000000000 ]
0 00000.00 Parameters  Domain:upper = [ 1.000000000000000 1.000000000000000 1.000000000000000 ]
0 00000.00 Parameters  Field:alignment = 8
0 00000.00 Parameters  Field:gamma = 1.666700000000000
0 00000.00 Parameters  Field:ghost_depth = 4
0 00000.00 Parameters  Field:history = 1
0 00000.00 Parameters  Field:list = [ "density" "velocity_x" "velocity_y" "velocity_z" "acceleration_x" "acceleration_y" "acceleration_z" "total_energy" "internal_energy" "pressure" "density_total" "density_particle" "density_particle_accumulate" "potential" "density_gas" "X_copy" "B" "R0" "DD_po_1" "DD_B" "Y1_bcg" "Y2_bcg" "P1_bcg" "Q2_bcg" "V1_bcg" "R1_bcg" "B_COPY" "potential_copy" ]
0 00000.00 Parameters  Field:padding = 0
0 00000.00 Parameters  Initial:list = [ "music" "cosmology" ]
0 00000.00 Parameters  Initial:music:FD:coords = "tzyx"
0 00000.00 Parameters  Initial:music:FD:dataset = "GridDensity"
0 00000.00 Parameters  Initial:music:FD:file = "input/cosmo_grid_density.h5"
0 00000.00 Parameters  Initial:music:FD:name = "density"
0 00000.00 Parameters  Initial:music:FD:type = "field"
0 00000.00 Parameters  Initial:music:FVX:coords = "tzyx"
0 00000.00 Parameters  Initial:music:FVX:dataset = "GridVelocities_x"
0 00000.00 Parameters  Initial:music:FVX:file = "input/cosmo_grid_velocities_x.h5"
0 00000.00 Parameters  Initial:music:FVX:name = "velocity_x"
0 00000.00 Parameters  Initial:music:FVX:type = "field"
0 00000.00 Parameters  Initial:music:FVY:coords = "tzyx"
0 00000.00 Parameters  Initial:music:FVY:dataset = "GridVelocities_y"
0 00000.00 Parameters  Initial:music:FVY:file = "input/cosmo_grid_velocities_y.h5"
0 00000.00 Parameters  Initial:music:FVY:name = "velocity_y"
0 00000.00 Parameters  Initial:music:FVY:type = "field"
0 00000.00 Parameters  Initial:music:FVZ:coords = "tzyx"
0 00000.00 Parameters  Initial:music:FVZ:dataset = "GridVelocities_z"
0 00000.00 Parameters  Initial:music:FVZ:file = "input/cosmo_grid_velocities_z.h5"
0 00000.00 Parameters  Initial:music:FVZ:name = "velocity_z"
0 00000.00 Parameters  Initial:music:FVZ:type = "field"
0 00000.00 Parameters  Initial:music:PVX:attribute = "vx"
0 00000.00 Parameters  Initial:music:PVX:coords = "tzyx"
0 00000.00 Parameters  Initial:music:PVX:dataset = "ParticleVelocities_x"
0 00000.00 Parameters  Initial:music:PVX:file = "input/cosmo_particle_velocities_x.h5"
0 00000.00 Parameters  Initial:music:PVX:name = "dark"
0 00000.00 Parameters  Initial:music:PVX:type = "particle"
0 00000.00 Parameters  Initial:music:PVY:attribute = "vy"
0 00000.00 Parameters  Initial:music:PVY:coords = "tzyx"
0 00000.00 Parameters  Initial:music:PVY:dataset = "ParticleVelocities_y"
0 00000.00 Parameters  Initial:music:PVY:file = "input/cosmo_particle_velocities_y.h5"
0 00000.00 Parameters  Initial:music:PVY:name = "dark"
0 00000.00 Parameters  Initial:music:PVY:type = "particle"
0 00000.00 Parameters  Initial:music:PVZ:attribute = "vz"
0 00000.00 Parameters  Initial:music:PVZ:coords = "tzyx"
0 00000.00 Parameters  Initial:music:PVZ:dataset = "ParticleVelocities_z"
0 00000.00 Parameters  Initial:music:PVZ:file = "input/cosmo_particle_velocities_z.h5"
0 00000.00 Parameters  Initial:music:PVZ:name = "dark"
0 00000.00 Parameters  Initial:music:PVZ:type = "particle"
0 00000.00 Parameters  Initial:music:PX:attribute = "x"
0 00000.00 Parameters  Initial:music:PX:coords = "tzyx"
0 00000.00 Parameters  Initial:music:PX:dataset = "ParticleDisplacements_x"
0 00000.00 Parameters  Initial:music:PX:file = "input/cosmo_particle_displacements_x.h5"
0 00000.00 Parameters  Initial:music:PX:name = "dark"
0 00000.00 Parameters  Initial:music:PX:type = "particle"
0 00000.00 Parameters  Initial:music:PY:attribute = "y"
0 00000.00 Parameters  Initial:music:PY:coords = "tzyx"
0 00000.00 Parameters  Initial:music:PY:dataset = "ParticleDisplacements_y"
0 00000.00 Parameters  Initial:music:PY:file = "input/cosmo_particle_displacements_y.h5"
0 00000.00 Parameters  Initial:music:PY:name = "dark"
0 00000.00 Parameters  Initial:music:PY:type = "particle"
0 00000.00 Parameters  Initial:music:PZ:attribute = "z"
0 00000.00 Parameters  Initial:music:PZ:coords = "tzyx"
0 00000.00 Parameters  Initial:music:PZ:dataset = "ParticleDisplacements_z"
0 00000.00 Parameters  Initial:music:PZ:file = "input/cosmo_particle_displacements_z.h5"
0 00000.00 Parameters  Initial:music:PZ:name = "dark"
0 00000.00 Parameters  Initial:music:PZ:type = "particle"
0 00000.00 Parameters  Initial:music:file_list = [ "FD" "FVX" "FVY" "FVZ" "PX" "PY" "PZ" "PVX" "PVY" "PVZ" ]
0 00000.00 Parameters  Initial:music:throttle_group_size = 64
0 00000.00 Parameters  Initial:music:throttle_internode = false
0 00000.00 Parameters  Initial:music:throttle_intranode = false
0 00000.00 Parameters  Initial:music:throttle_node_files = false
0 00000.00 Parameters  Initial:music:throttle_seconds_delay = 0.01000000000000000
0 00000.00 Parameters  Initial:music:throttle_seconds_stagger = 0.1000000000000000
0 00000.00 Parameters  Memory:limit_gb = 1.000000000000000
0 00000.00 Parameters  Mesh:root_blocks = [ 4 4 4 ]
0 00000.00 Parameters  Mesh:root_rank = 3
0 00000.00 Parameters  Mesh:root_size = [ 32 32 32 ]
0 00000.00 Parameters  Method:gravity:accumulate = true
0 00000.00 Parameters  Method:gravity:grav_const = 1.000000000000000
0 00000.00 Parameters  Method:gravity:order = 2
0 00000.00 Parameters  Method:gravity:solver = "bcg"
0 00000.00 Parameters  Method:list = [ "pm_deposit" "gravity" "pm_update" "comoving_expansion" ]
0 00000.00 Parameters  Method:ppm:courant = 0.5000000000000000
0 00000.00 Parameters  Method:ppm:diffusion = false
0 00000.00 Parameters  Method:ppm:dual_energy = true
0 00000.00 Parameters  Output:ax:dir = [ "Dir_COSMO_BCG_%04d" "cycle" ]
0 00000.00 Parameters  Output:ax:field_list = [ "acceleration_x" ]
0 00000.00 Parameters  Output:ax:image_ghost = false
0 00000.00 Parameters  Output:ax:image_size = [ 512 512 ]
0 00000.00 Parameters  Output:ax:image_type = "data"
0 00000.00 Parameters  Output:ax:name = [ "ax-%02d.png" "count" ]
0 00000.00 Parameters  Output:ax:schedule:step = 20
0 00000.00 Parameters  Output:ax:schedule:var = "cycle"
0 00000.00 Parameters  Output:ax:type = "image"
0 00000.00 Parameters  Output:ay:dir = [ "Dir_COSMO_BCG_%04d" "cycle" ]
0 00000.00 Parameters  Output:ay:field_list = [ "acceleration_y" ]
0 00000.00 Parameters  Output:ay:image_size = [ 512 512 ]
0 00000.00 Parameters  Output:ay:image_type = "data"
0 00000.00 Parameters  Output:ay:name = [ "ay-%02d.png" "count" ]
0 00000.00 Parameters  Output:ay:schedule:step = 20
0 00000.00 Parameters  Output:ay:schedule:var = "cycle"
0 00000.00 Parameters  Output:ay:type = "image"
0 00000.00 Parameters  Output:az:dir = [ "Dir_COSMO_BCG_%04d" "cycle" ]
0 00000.00 Parameters  Output:az:field_list = [ "acceleration_z" ]
0 00000.00 Parameters  Output:az:image_size = [ 512 512 ]
0 00000.00 Parameters  Output:az:image_type = "data"
0 00000.00 Parameters  Output:az:name = [ "az-%02d.png" "count" ]
0 00000.00 Parameters  Output:az:schedule:step = 20
0 00000.00 Parameters  Output:az:schedule:var = "cycle"
0 00000.00 Parameters  Output:az:type = "image"
0 00000.00 Parameters  Output:check:dir = [ "Dir_COSMO_BCG_%04d-checkpoint" "count" ]
0 00000.00 Parameters  Output:check:schedule:start = 100
0 00000.00 Parameters  Output:check:schedule:step = 200
0 00000.00 Parameters  Output:check:schedule:var = "cycle"
0 00000.00 Parameters  Output:check:type = "checkpoint"
0 00000.00 Parameters  Output:dark:colormap = [ 0.000000000000000 0.000000000000000 0.000000000000000 1.000000000000000 0.000000000000000 0.000000000000000 1.000000000000000 1.000000000000000 0.000000000000000 1.000000000000000 1.000000000000000 1.000000000000000 ]
0 00000.00 Parameters  Output:dark:dir = [ "Dir_COSMO_BCG_%04d" "cycle" ]
0 00000.00 Parameters  Output:dark:image_ghost = false
0 00000.00 Parameters  Output:dark:image_size = [ 512 512 ]
0 00000.00 Parameters  Output:dark:image_type = "data"
0 00000.00 Parameters  Output:dark:name = [ "dark-%02d.png" "count" ]
0 00000.00 Parameters  Output:dark:particle_list = [ "dark" ]
0 00000.00 Parameters  Output:dark:schedule:step = 20
0 00000.00 Parameters  Output:dark:schedule:var = "cycle"
0 00000.00 Parameters  Output:dark:type = "image"
0 00000.00 Parameters  Output:de:dir = [ "Dir_COSMO_BCG_%04d" "cycle" ]
0 00000.00 Parameters  Output:de:field_list = [ "density" ]
0 00000.00 Parameters  Output:de:image_ghost = false
0 00000.00 Parameters  Output:de:image_size = [ 512 512 ]
0 00000.00 Parameters  Output:de:image_type = "data"
0 00000.00 Parameters  Output:de:name = [ "de-%02d.png" "count" ]
0 00000.00 Parameters  Output:de:schedule:step = 20
0 00000.00 Parameters  Output:de:schedule:var = "cycle"
0 00000.00 Parameters  Output:de:type = "image"
0 00000.00 Parameters  Output:dep:dir = [ "Dir_COSMO_BCG_%04d" "cycle" ]
0 00000.00 Parameters  Output:dep:field_list = [ "density_particle" ]
0 00000.00 Parameters  Output:dep:image_ghost = false
0 00000.00 Parameters  Output:dep:image_size = [ 512 512 ]
0 00000.00 Parameters  Output:dep:image_type = "data"
0 00000.00 Parameters  Output:dep:name = [ "dep-%02d.png" "count" ]
0 00000.00 Parameters  Output:dep:schedule:step = 20
0 00000.00 Parameters  Output:dep:schedule:var = "cycle"
0 00000.00 Parameters  Output:dep:type = "image"
0 00000.00 Parameters  Output:depa:dir = [ "Dir_COSMO_BCG_%04d" "cycle" ]
0 00000.00 Parameters  Output:depa:field_list = [ "density_particle_accumulate" ]
0 00000.00 Parameters  Output:depa:image_ghost = false
0 00000.00 Parameters  Output:depa:image_size = [ 512 512 ]
0 00000.00 Parameters  Output:depa:image_type = "data"
0 00000.00 Parameters  Output:depa:name = [ "depa-%02d.png" "count" ]
0 00000.00 Parameters  Output:depa:schedule:step = 20
0 00000.00 Parameters  Output:depa:schedule:var = "cycle"
0 00000.00 Parameters  Output:depa:type = "image"
0 00000.00 Parameters  Output:hdf5:dir = [ "Dir_COSMO_BCG_%04d" "cycle" ]
0 00000.00 Parameters  Output:hdf5:field_list = [ "density" "velocity_x" "velocity_y" "velocity_z" "acceleration_x" "acceleration_y" "acceleration_z" "total_energy" "internal_energy" "pressure" ]
0 00000.00 Parameters  Output:hdf5:name = [ "data-%02d-%02d.h5" "count" "proc" ]
0 00000.00 Parameters  Output:hdf5:particle_list = [ "dark" ]
0 00000.00 Parameters  Output:hdf5:schedule:step = 20
0 00000.00 Parameters  Output:hdf5:schedule:var = "cycle"
0 00000.00 Parameters  Output:hdf5:type = "data"
0 00000.00 Parameters  Output:list = [ "de" "depa" "ax" "dark" "mesh" "po" "check" ]
0 00000.00 Parameters  Output:mesh:colormap = [ 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 1.000000000000000 0.000000000000000 1.000000000000000 1.000000000000000 0.000000000000000 1.000000000000000 0.000000000000000 1.000000000000000 1.000000000000000 0.000000000000000 1.000000000000000 0.000000000000000 0.000000000000000 ]
0 00000.00 Parameters  Output:mesh:dir = [ "Dir_COSMO_BCG_%04d" "cycle" ]
0 00000.00 Parameters  Output:mesh:image_reduce_type = "max"
0 00000.00 Parameters  Output:mesh:image_size = [ 513 513 ]
0 00000.00 Parameters  Output:mesh:image_type = "mesh"
0 00000.00 Parameters  Output:mesh:name = [ "mesh-%02d.png" "count" ]
0 00000.00 Parameters  Output:mesh:schedule:step = 20
0 00000.00 Parameters  Output:mesh:schedule:var = "cycle"
0 00000.00 Parameters  Output:mesh:type = "image"
0 00000.00 Parameters  Output:po:dir = [ "Dir_COSMO_BCG_%04d" "cycle" ]
0 00000.00 Parameters  Output:po:field_list = [ "potential_copy" ]
0 00000.00 Parameters  Output:po:image_size = [ 512 512 ]
0 00000.00 Parameters  Output:po:image_type = "data"
0 00000.00 Parameters  Output:po:name = [ "po-%02d.png" "count" ]
0 00000.00 Parameters  Output:po:schedule:step = 20
0 00000.00 Parameters  Output:po:schedule:var = "cycle"
0 00000.00 Parameters  Output:po:type = "image"
0 00000.00 Parameters  Particle:dark:attributes = [ "x" "default" "y" "default" "z" "default" "vx" "default" "vy" "default" "vz" "default" "ax" "default" "ay" "default" "az" "default" ]
0 00000.00 Parameters  Particle:dark:constants = [ "mass" "default" 0.8666666666667000 ]
0 00000.00 Parameters  Particle:dark:position = [ "x" "y" "z" ]
0 00000.00 Parameters  Particle:dark:velocity = [ "vx" "vy" "vz" ]
0 00000.00 Parameters  Particle:list = [ "dark" ]
0 00000.00 Parameters  Physics:cosmology:comoving_box_size = 3.000000000000000
0 00000.00 Parameters  Physics:cosmology:final_redshift = 3.000000000000000
0 00000.00 Parameters  Physics:cosmology:hubble_constant_now = 0.7000000000000000
0 00000.00 Parameters  Physics:cosmology:initial_redshift = 99.00000000000000
0 00000.00 Parameters  Physics:cosmology:max_expansion_rate = 0.01500000000000000
0 00000.00 Parameters  Physics:cosmology:omega_baryon_now = 0.04000000000000000
0 00000.00 Parameters  Physics:cosmology:omega_cdm_now = 0.2600000000000000
0 00000.00 Parameters  Physics:cosmology:omega_lambda_now = 0.7000000000000000
0 00000.00 Parameters  Physics:cosmology:omega_matter_now = 0.3000000000000000
0 00000.00 Parameters  Physics:list = [ "cosmology" ]
0 00000.00 Parameters  Solver:bcg:iter_max = 100
0 00000.00 Parameters  Solver:bcg:monitor_iter = 10
0 00000.00 Parameters  Solver:bcg:res_tol = 0.01000000000000000
0 00000.00 Parameters  Solver:bcg:type = "bicgstab"
0 00000.00 Parameters  Solver:list = [ "bcg" ]
0 00000.00 Parameters  Stopping:cycle = 160
0 00000.05 WARNING parameters_Config.cpp:927
0 00000.05 WARNING Config::read()
0 00000.05 WARNING output_axis[0] set to z
0 00000.05 WARNING parameters_Config.cpp:927
0 00000.05 WARNING Config::read()
0 00000.05 WARNING output_axis[1] set to z
0 00000.05 WARNING parameters_Config.cpp:927
0 00000.05 WARNING Config::read()
0 00000.05 WARNING output_axis[2] set to z
0 00000.05 WARNING parameters_Config.cpp:927
0 00000.05 WARNING Config::read()
0 00000.05 WARNING output_axis[3] set to z
0 00000.05 WARNING parameters_Config.cpp:927
0 00000.05 WARNING Config::read()
0 00000.05 WARNING output_axis[4] set to z
0 00000.05 WARNING parameters_Config.cpp:927
0 00000.05 WARNING Config::read()
0 00000.05 WARNING output_axis[5] set to z
UNIT TEST BEGIN
0 00000.05  ==============================================
0 00000.05   
0 00000.05    .oooooo.             oooo  oooo            
0 00000.05   d8P'  `Y8b            `888  `888            
0 00000.05  888           .ooooo.   888   888   .ooooo.  
0 00000.05  888          d88' `88b  888   888  d88' `88b 
0 00000.05  888          888ooo888  888   888  888   888 
0 00000.05  `88b    ooo  888    .o  888   888  888   888 
0 00000.05   `Y8bood8P'  `Y8bod8P' o888o o888o `Y8bod8P' 
0 00000.05   
0 00000.05  A Parallel Adaptive Mesh Refinement Framework
0 00000.05   
0 00000.05    Laboratory for Computational Astrophysics
0 00000.05          San Diego Supercomputer Center
0 00000.05       University of California, San Diego
0 00000.05   
0 00000.05  See 'LICENSE_CELLO' for software license information
0 00000.05   
0 00000.05  BEGIN CELLO: Feb 08 15:30:12
0 00000.05 Define Simulation processors 4
0 00000.05 CHARM CkNumPes()           4
0 00000.05 CHARM CkNumNodes()         4
0 00000.05  BEGIN ENZO-P
0 00000.05 Memory bytes 272858 bytes_high 272912
Charm++: late entry method registration happened after init
------------- Processor 0 Exiting: Called CmiAbort ------------
Reason: Did you forget to import a module or instantiate a templated entry method in a .ci file?

Entry point: EnzoSimulation(const char *filename, int n), addr: 0x555555808370
[0] Stack Traceback:
  [0:0] enzo_e_exe 0x555555b1c280 CmiAbortHelper(char const*, char const*, char const*, int, int)
  [0:1] enzo_e_exe 0x555555b1c3a4 
  [0:2] enzo_e_exe 0x555555a2f2f8 
  [0:3] enzo_e_exe 0x55555580a7cd CkIndex_EnzoSimulation::reg_EnzoSimulation_marshall1()
  [0:4] enzo_e_exe 0x55555580acd1 CProxy_EnzoSimulation::ckNew(char const*, int, CkEntryOptions const*)
  [0:5] enzo_e_exe 0x5555558185d0 Main::Main(CkArgMsg*)
  [0:6] enzo_e_exe 0x555555a2c3f6 _initCharm(int, char**)
  [0:7] enzo_e_exe 0x555555b1ea1e ConverseInit
  [0:8] enzo_e_exe 0x555555a2b2cc charm_main
  [0:9] libc.so.6 0x7ffff717eb25 __libc_start_main
  [0:10] enzo_e_exe 0x5555556fb2ce _start
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[1] Stack Traceback:
  [1:0] enzo_e_exe 0x555555b1eb67 
  [1:1] libpthread.so.0 0x7ffff7f85960 
  [1:2] enzo_e_exe 0x555555b1cec3 
  [1:3] enzo_e_exe 0x555555b1da99 LrtsAdvanceCommunication(int)
  [1:4] enzo_e_exe 0x555555b1db47 CmiGetNonLocal
  [1:5] enzo_e_exe 0x555555b1fa1d CsdNextMessage
  [1:6] enzo_e_exe 0x555555b1fad3 CsdScheduleForever
  [1:7] enzo_e_exe 0x555555b1fd85 CsdScheduler
  [1:8] enzo_e_exe 0x555555b1ea9a ConverseInit
  [1:9] enzo_e_exe 0x555555a2b2cc charm_main
  [1:10] libc.so.6 0x7ffff717eb25 __libc_start_main
  [1:11] enzo_e_exe 0x5555556fb2ce _start
[2] Stack Traceback:
  [2:0] enzo_e_exe 0x555555b1eb67 
  [2:1] libpthread.so.0 0x7ffff7f85960 
  [2:2] libmpi.so.40 0x7ffff75e33dd 
  [2:3] mca_pml_ob1.so 0x7ffff4ec2281 mca_pml_ob1_iprobe
  [2:4] libmpi.so.40 0x7ffff7596510 MPI_Iprobe
  [2:5] enzo_e_exe 0x555555b1d195 
  [2:6] enzo_e_exe 0x555555b1dabe 
  [2:7] enzo_e_exe 0x555555b2385e CcdRaiseCondition
  [2:8] enzo_e_exe 0x555555b1fb65 CsdScheduleForever
  [2:9] enzo_e_exe 0x555555b1fd85 CsdScheduler
  [2:10] enzo_e_exe 0x555555b1ea9a ConverseInit
  [2:11] enzo_e_exe 0x555555a2b2cc charm_main
  [2:12] libc.so.6 0x7ffff717eb25 __libc_start_main
  [2:13] enzo_e_exe 0x5555556fb2ce _start
------------- Processor 1 Exiting: Caught Signal ------------
Reason: Terminated
------------- Processor 2 Exiting: Caught Signal ------------
Reason: Terminated
------------- Processor 3 Exiting: Caught Signal ------------
Reason: Terminated
[3] Stack Traceback:
  [3:0] enzo_e_exe 0x555555b1eb67 
  [3:1] libpthread.so.0 0x7ffff7f85960 
  [3:2] enzo_e_exe 0x555555b1cdf0 
  [3:3] enzo_e_exe 0x555555b1da99 LrtsAdvanceCommunication(int)
  [3:4] enzo_e_exe 0x555555b1db47 CmiGetNonLocal
  [3:5] enzo_e_exe 0x555555b1fa1d CsdNextMessage
  [3:6] enzo_e_exe 0x555555b1fad3 CsdScheduleForever
  [3:7] enzo_e_exe 0x555555b1fd85 CsdScheduler
  [3:8] enzo_e_exe 0x555555b1ea9a ConverseInit
  [3:9] enzo_e_exe 0x555555a2b2cc charm_main
  [3:10] libc.so.6 0x7ffff717eb25 __libc_start_main
  [3:11] enzo_e_exe 0x5555556fb2ce _start
[forrest-razer:758669] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[forrest-razer:758669] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Any suggestions for the cmake build process? You can look at my repo and cmake build instructions at https://github.com/forrestglines/enzo-e/tree/cmake

forrestglines commented 3 years ago

As a note, I did remove the generated auto_config.def header file from my build, and all references to it in src/Cell/monitor_Monitor.cpp. I don't think that it's causing issues.

mabruzzo commented 3 years ago

I spent some time looking at this today. I've got to cut my losses for now, but I wanted to share what I did (in case it helps). Maybe I can take another look tomorrow afternoon or Thursday.

First, I managed to replicate your issue using a netlrts-linux-x86_64 build of charm++

My best guess is that the issue relates to the inclusion of the following files when compiling cello_library:

${CMAKE_CURRENT_SOURCE_DIR}/Cello/main.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Cello/main_enzo.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Cello/main_mesh.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Cello/main_simulation.cpp If you look at lines 194-202 from src/Cello/SConscript, only some of these are included while compiling constituent libraries. But I'm not super confident in that assessment

While messing around with this I tried replacing the section of src/Cello/CMakeLists.txt related to the Cello layer with instructions that more closely reflected the instructions in src/Cello/SConscript. Basically, I went and tried to compile each of the individual libraries that we currently build. I did this so that I could try to compile some of the unit tests to see if I could get those to work (I thought that they might be easier to compile than the full Enzo-E binary). This turned out to be more work than I though, but I ultimately had some success. I was able to get some of the lightest-weight tests to run (and pass) like test_Error.cpp, test_memory.cpp, and test_monitor.cpp. I also attempted 3 other tests but those all failed (I think those are all linker errors)

I'm not sure how helpful this will be, but here is a copy of my src/CMakeLists.txt file. This is a little hacky/messy, but it should be readable. I wrote this by hand, so things are probably not perfectly transcribed (which is probably causing the linker errors - note that a bug is also preventing the Enzo-E binary from compiling). In hindsight, it would probably have been better to inject some code into src/Cello/SConscript to directly generate a CMakeLists.txt file. In the event that you do want to try to use this file, I think you may to know that I tweaked the root level CMakeLists.txt to include the following 2 lines:

set(HDF5_USE_STATIC_LIBS OFF) find_package(HDF5)

and I removed "hdf5" from "ENZO_E_LIBRARIES"

forrestglines commented 3 years ago

Thanks @mabruzzo! Compiling Kokkos with SConstruct turned out to be easier than I thought, so for now I'm abandoning my cmake attempt. I might return to this later though.

jwise77 commented 2 years ago

This was addressed by #139