apontzen
commented
3 years ago Could you provide the full log for this run up to the exception? It might provide some clues...
Closed Martin-Rey closed 3 years ago
apontzen
commented
3 years ago Could you provide the full log for this run up to the exception? It might provide some clues...
Martin-Rey
commented
3 years ago The logs are pretty longs, as this was a 1000 jobs run interactively with mpi. After more careful investigation of the log, there is second "shape" related RuntimeError from the timer which I had not noticed. Both are copied below with a longer chunk of log output.
The tangos command was
mpirun -np 5 tangos write shrink_center_stars --include-only="contamination_fraction<0.01" --include-only="M200c_stars>1" --hmax 1000 --with-prerequisites --backend=mpi4pyRuntimeError1:
[ 3] 2021-02-05 10:19:36,072 : Processing <TimeStep u’Halo600_fiducial_UVB_nohighzdamping/output_00009' z=11.50 t=0.39 Gyr>
[ 3] 2021-02-05 10:19:36,082 : Missing prerequisites - added class <class ‘tangos.properties.pynbody.centring.CentreAndRadius’>
[ 3] 2021-02-05 10:19:36,082 : providing properties (‘shrink_center’, ‘max_radius’)
[ 3] 2021-02-05 10:19:36,082 : Gathering existing properties for all halos in timestep <TimeStep u’Halo600_fiducial_UVB_nohighzdamping/output_00009' z=11.50 t=0.39 Gyr>
[ 3] 2021-02-05 10:19:36,216 : User-specified inclusion criterion excluded 514 of 516 halos
[ 3] 2021-02-05 10:19:36,219 : Successfully gathered existing properties; calculating halo properties now...
[ 3] 2021-02-05 10:19:36,219 : 2 halos to consider; 2 property calculations for each of them
[ 2] 2021-02-05 10:19:49,835 : Attempting to commit 2 halo properties...
[ 2] 2021-02-05 10:19:50,206 : 2 properties were committed
[ 2] 2021-02-05 10:19:50,206 : CUMULATIVE RUNNING TIMES (just this node)
[ 2] 2021-02-05 10:19:50,206 : StellarAndGasCenter 31.6s | 100.0%
[ 2] 2021-02-05 10:19:50,206 : ------ INTERNAL BREAKDOWN ------
[ 2] 2021-02-05 10:19:50,206 : start cen 2.4s | 7.6% | 7.6%
[ 2] 2021-02-05 10:19:50,206 : cen cen 27.6s | 87.3% | 87.3%
[ 2] 2021-02-05 10:19:50,206 : cen end 1.6s | 5.1% | 5.1%
[ 2] 2021-02-05 10:19:50,206 : --------------------------------
[ 2] 2021-02-05 10:20:50,977 : Attempting to commit 58 halo properties...
[ 2] 2021-02-05 10:20:51,328 : 58 properties were committed
[ 2] 2021-02-05 10:20:51,328 : CUMULATIVE RUNNING TIMES (just this node)
[ 2] 2021-02-05 10:20:51,328 : StellarAndGasCenter 76.7s | 100.0%
[ 2] 2021-02-05 10:20:51,328 : ------ INTERNAL BREAKDOWN ------
[ 2] 2021-02-05 10:20:51,328 : start cen 2.7s | 3.5% | 3.5%
[ 2] 2021-02-05 10:20:51,328 : cen cen 67.1s | 87.5% | 87.5%
[ 2] 2021-02-05 10:20:51,328 : cen end 7.0s | 9.1% | 9.1%
[ 2] 2021-02-05 10:20:51,328 : --------------------------------
[ 3] 2021-02-05 10:21:04,512 : Attempting to commit 12 halo properties...
[ 3] 2021-02-05 10:21:05,005 : 12 properties were committed
[ 3] 2021-02-05 10:21:05,005 : CUMULATIVE RUNNING TIMES (just this node)
[ 3] 2021-02-05 10:21:05,005 : StellarAndGasCenter 134.9s | 100.0%
[ 3] 2021-02-05 10:21:05,005 : ------ INTERNAL BREAKDOWN ------
[ 3] 2021-02-05 10:21:05,005 : start cen 2.3s | 1.7% | 1.7%
[ 3] 2021-02-05 10:21:05,005 : cen cen 117.4s | 87.0% | 87.0%
[ 3] 2021-02-05 10:21:05,005 : cen end 15.2s | 11.3% | 11.3%
[ 3] 2021-02-05 10:21:05,005 : --------------------------------
[ 3] 2021-02-05 10:21:06,955 : Done with <TimeStep u’Halo600_fiducial_UVB_nohighzdamping/output_00009' z=11.50 t=0.39 Gyr>
[ 3] 2021-02-05 10:21:07,828 : Succeeded: 2 property calculations
[ 3] 2021-02-05 10:21:07,828 : Errored: 0 property calculations
[ 3] 2021-02-05 10:21:07,828 : Errored during load: 0 property calculations
[ 3] 2021-02-05 10:21:07,828 : Already exists: 2 property calculations
[ 3] 2021-02-05 10:21:07,828 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:21:07,829 : Send job 84 of 1097 to node 3
[ 3] 2021-02-05 10:21:07,829 : Processing <TimeStep u’Halo600_fiducial/output_00009' z=11.50 t=0.39 Gyr>
[ 3] 2021-02-05 10:21:07,838 : Missing prerequisites - added class <class ‘tangos.properties.pynbody.centring.CentreAndRadius’>
[ 3] 2021-02-05 10:21:07,838 : providing properties (‘shrink_center’, ‘max_radius’)
[ 3] 2021-02-05 10:21:07,838 : Gathering existing properties for all halos in timestep <TimeStep u’Halo600_fiducial/output_00009' z=11.50 t=0.39 Gyr>
[ 3] 2021-02-05 10:21:07,933 : User-specified inclusion criterion excluded 513 of 515 halos
[ 3] 2021-02-05 10:21:07,936 : Successfully gathered existing properties; calculating halo properties now...
[ 3] 2021-02-05 10:21:07,936 : 2 halos to consider; 2 property calculations for each of them
[ 2] 2021-02-05 10:21:20,801 : Done with <TimeStep u’Halo600_RT/output_00010' z=11.50 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:21,928 : Succeeded: 48 property calculations
[ 2] 2021-02-05 10:21:21,928 : Errored: 0 property calculations
[ 2] 2021-02-05 10:21:21,928 : Errored during load: 0 property calculations
[ 2] 2021-02-05 10:21:21,928 : Already exists: 48 property calculations
[ 2] 2021-02-05 10:21:21,928 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:21:21,928 : Send job 85 of 1097 to node 2
[ 2] 2021-02-05 10:21:21,928 : Processing <TimeStep u’Halo600_DMO_hires/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:21,934 : Missing prerequisites - added class <class ‘tangos.properties.pynbody.centring.CentreAndRadius’>
[ 2] 2021-02-05 10:21:21,934 : providing properties (‘shrink_center’, ‘max_radius’)
[ 2] 2021-02-05 10:21:21,935 : Gathering existing properties for all halos in timestep <TimeStep u’Halo600_DMO_hires/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,154 : User-specified inclusion criterion excluded 1000 of 1000 halos
[ 2] 2021-02-05 10:21:22,155 : Successfully gathered existing properties; calculating halo properties now...
[ 2] 2021-02-05 10:21:22,155 : 0 halos to consider; 2 property calculations for each of them
[ 2] 2021-02-05 10:21:22,155 : Done with <TimeStep u’Halo600_DMO_hires/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,187 : Succeeded: 0 property calculations
[ 2] 2021-02-05 10:21:22,187 : Errored: 0 property calculations
[ 2] 2021-02-05 10:21:22,188 : Errored during load: 0 property calculations
[ 2] 2021-02-05 10:21:22,188 : Already exists: 0 property calculations
[ 2] 2021-02-05 10:21:22,188 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:21:22,188 : Send job 86 of 1097 to node 2
[ 2] 2021-02-05 10:21:22,188 : Processing <TimeStep u’Halo600_DMO/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,193 : Missing prerequisites - added class <class ‘tangos.properties.pynbody.centring.CentreAndRadius’>
[ 2] 2021-02-05 10:21:22,193 : providing properties (‘shrink_center’, ‘max_radius’)
[ 2] 2021-02-05 10:21:22,193 : Gathering existing properties for all halos in timestep <TimeStep u’Halo600_DMO/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,273 : User-specified inclusion criterion excluded 550 of 550 halos
[ 2] 2021-02-05 10:21:22,273 : Successfully gathered existing properties; calculating halo properties now...
[ 2] 2021-02-05 10:21:22,273 : 0 halos to consider; 2 property calculations for each of them
[ 2] 2021-02-05 10:21:22,273 : Done with <TimeStep u’Halo600_DMO/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,302 : Succeeded: 0 property calculations
[ 2] 2021-02-05 10:21:22,302 : Errored: 0 property calculations
[ 2] 2021-02-05 10:21:22,302 : Errored during load: 0 property calculations
[ 2] 2021-02-05 10:21:22,302 : Already exists: 0 property calculations
[ 2] 2021-02-05 10:21:22,302 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:21:22,302 : Send job 87 of 1097 to node 2
[ 2] 2021-02-05 10:21:22,302 : Processing <TimeStep u’Halo600_fiducial_later_mergers/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,309 : Missing prerequisites - added class <class ‘tangos.properties.pynbody.centring.CentreAndRadius’>
[ 2] 2021-02-05 10:21:22,310 : providing properties (‘shrink_center’, ‘max_radius’)
[ 2] 2021-02-05 10:21:22,310 : Gathering existing properties for all halos in timestep <TimeStep u’Halo600_fiducial_later_mergers/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,406 : User-specified inclusion criterion excluded 399 of 399 halos
[ 2] 2021-02-05 10:21:22,406 : Successfully gathered existing properties; calculating halo properties now...
[ 2] 2021-02-05 10:21:22,406 : 0 halos to consider; 2 property calculations for each of them
[ 2] 2021-02-05 10:21:22,406 : Done with <TimeStep u’Halo600_fiducial_later_mergers/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,434 : Succeeded: 0 property calculations
[ 2] 2021-02-05 10:21:22,434 : Errored: 0 property calculations
[ 2] 2021-02-05 10:21:22,434 : Errored during load: 0 property calculations
[ 2] 2021-02-05 10:21:22,434 : Already exists: 0 property calculations
[ 2] 2021-02-05 10:21:22,434 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:21:22,434 : Send job 88 of 1097 to node 2
[ 2] 2021-02-05 10:21:22,434 : Processing <TimeStep u’Halo600_DMO_lowres/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,439 : Missing prerequisites - added class <class ‘tangos.properties.pynbody.centring.CentreAndRadius’>
[ 2] 2021-02-05 10:21:22,439 : providing properties (‘shrink_center’, ‘max_radius’)
[ 2] 2021-02-05 10:21:22,440 : Gathering existing properties for all halos in timestep <TimeStep u’Halo600_DMO_lowres/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,463 : User-specified inclusion criterion excluded 3 of 3 halos
[ 2] 2021-02-05 10:21:22,463 : Successfully gathered existing properties; calculating halo properties now...
[ 2] 2021-02-05 10:21:22,463 : 0 halos to consider; 2 property calculations for each of them
[ 2] 2021-02-05 10:21:22,463 : Done with <TimeStep u’Halo600_DMO_lowres/output_00009' z=11.49 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,488 : Succeeded: 0 property calculations
[ 2] 2021-02-05 10:21:22,488 : Errored: 0 property calculations
[ 2] 2021-02-05 10:21:22,488 : Errored during load: 0 property calculations
[ 2] 2021-02-05 10:21:22,488 : Already exists: 0 property calculations
[ 2] 2021-02-05 10:21:22,488 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:21:22,489 : Send job 89 of 1097 to node 2
[ 2] 2021-02-05 10:21:22,489 : Processing <TimeStep u’Halo600_fiducial_fblim/output_00009' z=11.48 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,492 : Missing prerequisites - added class <class ‘tangos.properties.pynbody.centring.CentreAndRadius’>
[ 2] 2021-02-05 10:21:22,492 : providing properties (‘shrink_center’, ‘max_radius’)
[ 2] 2021-02-05 10:21:22,492 : Gathering existing properties for all halos in timestep <TimeStep u’Halo600_fiducial_fblim/output_00009' z=11.48 t=0.39 Gyr>
[ 2] 2021-02-05 10:21:22,570 : User-specified inclusion criterion excluded 516 of 518 halos
[ 2] 2021-02-05 10:21:22,573 : Successfully gathered existing properties; calculating halo properties now...
[ 2] 2021-02-05 10:21:22,573 : 2 halos to consider; 2 property calculations for each of them
[ 3] 2021-02-05 10:22:34,538 : Attempting to commit 4 halo properties...
[ 3] 2021-02-05 10:22:35,083 : 4 properties were committed
[ 3] 2021-02-05 10:22:35,083 : CUMULATIVE RUNNING TIMES (just this node)
[ 3] 2021-02-05 10:22:35,083 : StellarAndGasCenter 140.5s | 100.0%
[ 3] 2021-02-05 10:22:35,083 : ------ INTERNAL BREAKDOWN ------
[ 3] 2021-02-05 10:22:35,083 : start cen 2.8s | 2.0% | 2.0%
[ 3] 2021-02-05 10:22:35,083 : cen cen 122.3s | 87.1% | 87.1%
[ 3] 2021-02-05 10:22:35,084 : cen end 15.4s | 11.0% | 11.0%
[ 3] 2021-02-05 10:22:35,084 : --------------------------------
[ 3] 2021-02-05 10:22:35,665 : Done with <TimeStep u’Halo600_fiducial/output_00009' z=11.50 t=0.39 Gyr>
[ 3] 2021-02-05 10:22:36,646 : Succeeded: 2 property calculations
[ 3] 2021-02-05 10:22:36,646 : Errored: 0 property calculations
[ 3] 2021-02-05 10:22:36,646 : Errored during load: 0 property calculations
[ 3] 2021-02-05 10:22:36,647 : Already exists: 2 property calculations
[ 3] 2021-02-05 10:22:36,647 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:22:36,647 : Send job 90 of 1097 to node 3
[ 3] 2021-02-05 10:22:36,647 : Processing <TimeStep u’Halo600_DMO_later_mergers/output_00009' z=11.32 t=0.40 Gyr>
[ 3] 2021-02-05 10:22:36,653 : Missing prerequisites - added class <class ‘tangos.properties.pynbody.centring.CentreAndRadius’>
[ 3] 2021-02-05 10:22:36,653 : providing properties (‘shrink_center’, ‘max_radius’)
[ 3] 2021-02-05 10:22:36,654 : Gathering existing properties for all halos in timestep <TimeStep u’Halo600_DMO_later_mergers/output_00009' z=11.32 t=0.40 Gyr>
[ 3] 2021-02-05 10:22:36,757 : User-specified inclusion criterion excluded 484 of 484 halos
[ 3] 2021-02-05 10:22:36,757 : Successfully gathered existing properties; calculating halo properties now...
[ 3] 2021-02-05 10:22:36,758 : 0 halos to consider; 2 property calculations for each of them
[ 3] 2021-02-05 10:22:36,758 : Done with <TimeStep u’Halo600_DMO_later_mergers/output_00009' z=11.32 t=0.40 Gyr>
[ 3] 2021-02-05 10:22:36,788 : Succeeded: 0 property calculations
[ 3] 2021-02-05 10:22:36,788 : Errored: 0 property calculations
[ 3] 2021-02-05 10:22:36,788 : Errored during load: 0 property calculations
[ 3] 2021-02-05 10:22:36,788 : Already exists: 0 property calculations
[ 3] 2021-02-05 10:22:36,788 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:22:36,788 : Send job 91 of 1097 to node 3
[ 3] 2021-02-05 10:22:36,788 : Processing <TimeStep u’Halo600_fiducial_hires/output_00010' z=10.52 t=0.44 Gyr>
[ 3] 2021-02-05 10:22:36,793 : Missing prerequisites - added class <class ‘tangos.properties.pynbody.centring.CentreAndRadius’>
[ 3] 2021-02-05 10:22:36,793 : providing properties (‘shrink_center’, ‘max_radius’)
[ 3] 2021-02-05 10:22:36,793 : Gathering existing properties for all halos in timestep <TimeStep u’Halo600_fiducial_hires/output_00010' z=10.52 t=0.44 Gyr>
[ 3] 2021-02-05 10:22:37,007 : User-specified inclusion criterion excluded 994 of 1000 halos
[ 3] 2021-02-05 10:22:37,011 : Successfully gathered existing properties; calculating halo properties now...
[ 3] 2021-02-05 10:22:37,011 : 6 halos to consider; 2 property calculations for each of them
[ 2] 2021-02-05 10:22:48,934 : Attempting to commit 38 halo properties...
[ 2] 2021-02-05 10:22:49,438 : 38 properties were committed
[ 2] 2021-02-05 10:22:49,438 : CUMULATIVE RUNNING TIMES (just this node)
[ 2] 2021-02-05 10:22:49,438 : StellarAndGasCenter 100.6s | 100.0%
[ 2] 2021-02-05 10:22:49,438 : ------ INTERNAL BREAKDOWN ------
[ 2] 2021-02-05 10:22:49,438 : start cen 3.4s | 3.4% | 3.4%
[ 2] 2021-02-05 10:22:49,438 : cen cen 87.8s | 87.3% | 87.3%
[ 2] 2021-02-05 10:22:49,438 : cen end 9.4s | 9.3% | 9.3%
[ 2] 2021-02-05 10:22:49,438 : --------------------------------
[ 2] 2021-02-05 10:22:50,226 : Done with <TimeStep u’Halo600_fiducial_fblim/output_00009' z=11.48 t=0.39 Gyr>
[ 2] 2021-02-05 10:22:51,375 : Succeeded: 2 property calculations
[ 2] 2021-02-05 10:22:51,375 : Errored: 0 property calculations
[ 2] 2021-02-05 10:22:51,375 : Errored during load: 0 property calculations
[ 2] 2021-02-05 10:22:51,375 : Already exists: 2 property calculations
[ 2] 2021-02-05 10:22:51,375 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:22:51,375 : Send job 92 of 1097 to node 2
[ 2] 2021-02-05 10:22:51,375 : Processing <TimeStep u’Halo600_fiducial_hires/output_00011' z=10.11 t=0.47 Gyr>
[ 2] 2021-02-05 10:22:51,381 : Missing prerequisites - added class <class ‘tangos.properties.pynbody.centring.CentreAndRadius’>
[ 2] 2021-02-05 10:22:51,381 : providing properties (‘shrink_center’, ‘max_radius’)
[ 2] 2021-02-05 10:22:51,382 : Gathering existing properties for all halos in timestep <TimeStep u’Halo600_fiducial_hires/output_00011' z=10.11 t=0.47 Gyr>
[ 2] 2021-02-05 10:22:51,777 : User-specified inclusion criterion excluded 995 of 1000 halos
[ 2] 2021-02-05 10:22:51,782 : Successfully gathered existing properties; calculating halo properties now...
[ 2] 2021-02-05 10:22:51,782 : 5 halos to consider; 2 property calculations for each of them
Traceback (most recent call last):
File “/scratch/dp101/shared/python/anaconda2/bin/tangos”, line 11, in <module>
load_entry_point(‘tangos’, ‘console_scripts’, ‘tangos’)()
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/scripts/__init__.py”, line 49, in main
args.func(args)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/__init__.py”, line 54, in run
parallel_tasks.launch(obj.run_calculation_loop, 2, [])
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/__init__.py”, line 54, in launch
backend.launch(_exec_function_or_server, num_procs, [function, args])
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/backends/mpi4py.py”, line 61, in launch
function(*args)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/__init__.py”, line 86, in _exec_function_or_server
function(*args)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 508, in run_calculation_loop
self.run_timestep_calculation(f_obj)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 471, in run_timestep_calculation
self.run_halo_calculation(db_halo, existing_properties)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 438, in run_halo_calculation
self.run_property_calculation(db_halo, calculator, existing_properties)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 423, in run_property_calculation
results = self._get_property_value(db_halo, property_calculator, existing_properties)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 367, in _get_property_value
snapshot_data = self._get_halo_snapshot_data_if_appropriate(db_halo, db_data, property_calculator)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 347, in _get_halo_snapshot_data_if_appropriate
return self._get_current_halo_specified_region_particles(db_halo, property_calculator.region_specification(db_data))
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 340, in _get_current_halo_specified_region_particles
return db_halo.timestep.load_region(region_spec,self.options.load_mode)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/core/timestep.py”, line 49, in load_region
return handler.load_region(self.extension, region_specification, mode=mode)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/input_handlers/pynbody.py”, line 89, in load_region
return timestep[region_specification]
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 271, in __getitem__
return IndexedSubSnap(self, i)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 1850, in __init__
index_array = index_array.where(base)[0]
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/filt.py”, line 27, in where
return np.where(self(sim))
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/filt.py”, line 150, in __call__
pos = sim[‘pos’]
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 263, in __getitem__
return self._get_array_with_lazy_actions(i)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 354, in _get_array_with_lazy_actions
self.__load_if_required(name)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 365, in __load_if_required
self.__load_array_and_perform_postprocessing(name)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 917, in __load_array_and_perform_postprocessing
self._load_array(array_name, fam)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/ramses.py”, line 980, in _load_array
self._create_array(‘pos’, 3)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 1118, in _create_array
new_array = array._array_factory(dims, dtype, zeros, shared)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/array.py”, line 1026, in _array_factory
raise MemoryError, “Unable to create shared memory region”
MemoryError: Unable to create shared memory region
Traceback (most recent call last):
File “/scratch/dp101/shared/python/anaconda2/bin/tangos”, line 11, in <module>
load_entry_point(‘tangos’, ‘console_scripts’, ‘tangos’)()
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/scripts/__init__.py”, line 49, in main
args.func(args)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/__init__.py”, line 54, in run
parallel_tasks.launch(obj.run_calculation_loop, 2, [])
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/__init__.py”, line 54, in launch
backend.launch(_exec_function_or_server, num_procs, [function, args])
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/backends/mpi4py.py”, line 61, in launch
function(*args)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/__init__.py”, line 86, in _exec_function_or_server
function(*args)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 508, in run_calculation_loop
self.run_timestep_calculation(f_obj)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 471, in run_timestep_calculation
self.run_halo_calculation(db_halo, existing_properties)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 438, in run_halo_calculation
self.run_property_calculation(db_halo, calculator, existing_properties)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 423, in run_property_calculation
results = self._get_property_value(db_halo, property_calculator, existing_properties)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 367, in _get_property_value
snapshot_data = self._get_halo_snapshot_data_if_appropriate(db_halo, db_data, property_calculator)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 347, in _get_halo_snapshot_data_if_appropriate
return self._get_current_halo_specified_region_particles(db_halo, property_calculator.region_specification(db_data))
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 340, in _get_current_halo_specified_region_particles
return db_halo.timestep.load_region(region_spec,self.options.load_mode)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/core/timestep.py”, line 49, in load_region
return handler.load_region(self.extension, region_specification, mode=mode)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/input_handlers/pynbody.py”, line 89, in load_region
return timestep[region_specification]
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 271, in __getitem__
return IndexedSubSnap(self, i)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 1850, in __init__
index_array = index_array.where(base)[0]
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/filt.py”, line 27, in where
return np.where(self(sim))
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/filt.py”, line 150, in __call__
pos = sim[‘pos’]
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 263, in __getitem__
return self._get_array_with_lazy_actions(i)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 354, in _get_array_with_lazy_actions
self.__load_if_required(name)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 365, in __load_if_required
self.__load_array_and_perform_postprocessing(name)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 917, in __load_array_and_perform_postprocessing
self._load_array(array_name, fam)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/ramses.py”, line 988, in _load_array
self._load_gas_vars()
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/ramses.py”, line 797, in _load_gas_vars
self.gas._create_array(i)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 1950, in _create_array
array_name, self._unifamily, ndim, dtype, derived, shared)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 1208, in _create_family_array
new_ar = array._array_factory(dims, dtype, False, shared)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/array.py”, line 1026, in _array_factory
raise MemoryError, “Unable to create shared memory region”
MemoryError: Unable to create shared memory region
[ 4] 2021-02-05 10:23:27,626 : Uncaught exception during property calculation <edge_tangos_properties.component_centering.StellarAndGasCenter object at 0x2aaafce6e710> applied to <Halo u’Halo600_fiducial_hires/output_00009/halo_1' | NDM=101297 Nstar=0 Ngas=0>
Traceback (most recent call last):
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 376, in _get_property_value
result = property_calculator.calculate(snapshot_data, db_data)
File “/lustre/dirac/scratch/dp101/shared/python/edge-tangos-properties/edge_tangos_properties/component_centering.py”, line 13, in calculate
star_center = self._get_center(halo.st)
File “/lustre/dirac/scratch/dp101/shared/python/edge-tangos-properties/edge_tangos_properties/component_centering.py”, line 25, in _get_center
center = pynbody.analysis.halo.shrink_sphere_center(particle_data, shrink_factor=0.8, velocity=False)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/analysis/halo.py”, line 90, in shrink_sphere_center
mass = np.asarray(sim[‘mass’], dtype=‘double’)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 263, in __getitem__
return self._get_array_with_lazy_actions(i)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 354, in _get_array_with_lazy_actions
self.__load_if_required(name)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 365, in __load_if_required
self.__load_array_and_perform_postprocessing(name)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 917, in __load_array_and_perform_postprocessing
self._load_array(array_name, fam)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 1968, in _load_array
self.base._load_array(array_name, self._unifamily)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 1793, in _load_array
self.base._load_array(array_name, fam)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/ramses.py”, line 956, in _load_array
self._load_particle_block(array_name)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/ramses.py”, line 857, in _load_particle_block
self._create_array_for_particles(blockname, _type)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/ramses.py”, line 850, in _create_array_for_particles
self[f]._create_array(name, dtype=type_)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 1950, in _create_array
array_name, self._unifamily, ndim, dtype, derived, shared)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py”, line 1208, in _create_family_array
new_ar = array._array_factory(dims, dtype, False, shared)
File “/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/array.py”, line 1026, in _array_factory
raise MemoryError, “Unable to create shared memory region”
MemoryError: Unable to create shared memory region
[ 4] 2021-02-05 10:23:27,661 : Further errors from this calculation on this timestep will be counted but not individually reported.
[ 4] 2021-02-05 10:23:27,670 : Attempting to commit 2 halo properties...
[ 4] 2021-02-05 10:23:27,729 : 2 properties were committed
[ 4] 2021-02-05 10:23:27,729 : CUMULATIVE RUNNING TIMES (just this node)
[ 4] 2021-02-05 10:23:27,729 : StellarAndGasCenter 25.5s | 100.0%
[ 4] 2021-02-05 10:23:27,730 : ------ INTERNAL BREAKDOWN ------
[ 4] 2021-02-05 10:23:27,730 : start end 2.9s | 11.5% | 11.5%
Traceback (most recent call last):
File “/scratch/dp101/shared/python/anaconda2/bin/tangos”, line 11, in <module>
load_entry_point(‘tangos’, ‘console_scripts’, ‘tangos’)()
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/scripts/__init__.py”, line 49, in main
args.func(args)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/__init__.py”, line 54, in run
parallel_tasks.launch(obj.run_calculation_loop, 2, [])
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/__init__.py”, line 54, in launch
backend.launch(_exec_function_or_server, num_procs, [function, args])
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/backends/mpi4py.py”, line 61, in launch
function(*args)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/__init__.py”, line 86, in _exec_function_or_server
function(*args)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 508, in run_calculation_loop
self.run_timestep_calculation(f_obj)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 471, in run_timestep_calculation
self.run_halo_calculation(db_halo, existing_properties)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 445, in run_halo_calculation
self._commit_results_if_needed()
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py”, line 257, in _commit_results_if_needed
self.timing_monitor.summarise_timing(logger)
File “/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/util/timing_monitor.py”, line 86, in summarise_timing
(marks_info[i], marks_info[i + 1],
IndexError: list index out of range[ 0] 2021-02-05 10:16:42,673 : Send job 74 of 1097 to node 2
[ 2] 2021-02-05 10:16:42,673 : Processing <TimeStep u'Halo600_fiducial_UVB_nohighzdamping/output_00008' z=13.28 t=0.32 Gyr>
[ 2] 2021-02-05 10:16:42,680 : Missing prerequisites - added class <class 'tangos.properties.pynbody.centring.CentreAndRadius'>
[ 2] 2021-02-05 10:16:42,680 : providing properties ('shrink_center', 'max_radius')
[ 2] 2021-02-05 10:16:42,681 : Gathering existing properties for all halos in timestep <TimeStep u'Halo600_fiducial_UVB_nohighzdamping/output_00008' z=13.28 t=0.32 Gyr>
[ 2] 2021-02-05 10:16:42,743 : User-specified inclusion criterion excluded 299 of 300 halos
[ 2] 2021-02-05 10:16:42,746 : Successfully gathered existing properties; calculating halo properties now...
[ 2] 2021-02-05 10:16:42,746 : 1 halos to consider; 2 property calculations for each of them
[ 3] 2021-02-05 10:16:45,293 : Done with <TimeStep u'Halo600_RT/output_00008' z=13.29 t=0.32 Gyr>
[ 3] 2021-02-05 10:16:46,337 : Succeeded: 22 property calculations
[ 3] 2021-02-05 10:16:46,337 : Errored: 0 property calculations
[ 3] 2021-02-05 10:16:46,337 : Errored during load: 0 property calculations
[ 3] 2021-02-05 10:16:46,338 : Already exists: 22 property calculations
[ 3] 2021-02-05 10:16:46,338 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:16:46,338 : Send job 75 of 1097 to node 3
[ 3] 2021-02-05 10:16:46,338 : Processing <TimeStep u'Halo600_DMO_hires/output_00008' z=13.28 t=0.32 Gyr>
[ 3] 2021-02-05 10:16:46,343 : Missing prerequisites - added class <class 'tangos.properties.pynbody.centring.CentreAndRadius'>
[ 3] 2021-02-05 10:16:46,343 : providing properties ('shrink_center', 'max_radius')
[ 3] 2021-02-05 10:16:46,343 : Gathering existing properties for all halos in timestep <TimeStep u'Halo600_DMO_hires/output_00008' z=13.28 t=0.32 Gyr>
[ 3] 2021-02-05 10:16:46,506 : User-specified inclusion criterion excluded 1000 of 1000 halos
[ 3] 2021-02-05 10:16:46,507 : Successfully gathered existing properties; calculating halo properties now...
[ 3] 2021-02-05 10:16:46,507 : 0 halos to consider; 2 property calculations for each of them
[ 3] 2021-02-05 10:16:46,507 : Done with <TimeStep u'Halo600_DMO_hires/output_00008' z=13.28 t=0.32 Gyr>
[ 3] 2021-02-05 10:16:46,541 : Succeeded: 0 property calculations
[ 3] 2021-02-05 10:16:46,541 : Errored: 0 property calculations
[ 3] 2021-02-05 10:16:46,541 : Errored during load: 0 property calculations
[ 3] 2021-02-05 10:16:46,541 : Already exists: 0 property calculations
[ 3] 2021-02-05 10:16:46,541 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:16:46,541 : Send job 76 of 1097 to node 3
[ 3] 2021-02-05 10:16:46,541 : Processing <TimeStep u'Halo600_DMO/output_00008' z=13.28 t=0.32 Gyr>
[ 3] 2021-02-05 10:16:46,545 : Missing prerequisites - added class <class 'tangos.properties.pynbody.centring.CentreAndRadius'>
[ 3] 2021-02-05 10:16:46,545 : providing properties ('shrink_center', 'max_radius')
[ 3] 2021-02-05 10:16:46,546 : Gathering existing properties for all halos in timestep <TimeStep u'Halo600_DMO/output_00008' z=13.28 t=0.32 Gyr>
[ 3] 2021-02-05 10:16:46,606 : User-specified inclusion criterion excluded 340 of 340 halos
[ 3] 2021-02-05 10:16:46,606 : Successfully gathered existing properties; calculating halo properties now...
[ 3] 2021-02-05 10:16:46,606 : 0 halos to consider; 2 property calculations for each of them
[ 3] 2021-02-05 10:16:46,606 : Done with <TimeStep u'Halo600_DMO/output_00008' z=13.28 t=0.32 Gyr>
[ 3] 2021-02-05 10:16:46,637 : Succeeded: 0 property calculations
[ 3] 2021-02-05 10:16:46,637 : Errored: 0 property calculations
[ 3] 2021-02-05 10:16:46,637 : Errored during load: 0 property calculations
[ 3] 2021-02-05 10:16:46,637 : Already exists: 0 property calculations
[ 0] 2021-02-05 10:16:46,637 : Send job 77 of 1097 to node 3
[ 3] 2021-02-05 10:16:46,637 : Missing pre-requisite: 0 property calculations
[ 3] 2021-02-05 10:16:46,637 : Processing <TimeStep u'Halo600_fiducial_later_mergers/output_00008' z=13.27 t=0.32 Gyr>
[ 3] 2021-02-05 10:16:46,645 : Missing prerequisites - added class <class 'tangos.properties.pynbody.centring.CentreAndRadius'>
[ 3] 2021-02-05 10:16:46,645 : providing properties ('shrink_center', 'max_radius')
[ 3] 2021-02-05 10:16:46,645 : Gathering existing properties for all halos in timestep <TimeStep u'Halo600_fiducial_later_mergers/output_00008' z=13.27 t=0.32 Gyr>
[ 3] 2021-02-05 10:16:46,706 : User-specified inclusion criterion excluded 186 of 186 halos
[ 3] 2021-02-05 10:16:46,706 : Successfully gathered existing properties; calculating halo properties now...
[ 3] 2021-02-05 10:16:46,707 : 0 halos to consider; 2 property calculations for each of them
[ 3] 2021-02-05 10:16:46,707 : Done with <TimeStep u'Halo600_fiducial_later_mergers/output_00008' z=13.27 t=0.32 Gyr>
[ 3] 2021-02-05 10:16:46,738 : Succeeded: 0 property calculations
[ 3] 2021-02-05 10:16:46,738 : Errored: 0 property calculations
[ 3] 2021-02-05 10:16:46,738 : Errored during load: 0 property calculations
[ 3] 2021-02-05 10:16:46,738 : Already exists: 0 property calculations
[ 3] 2021-02-05 10:16:46,738 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:16:46,738 : Send job 78 of 1097 to node 3
[ 3] 2021-02-05 10:16:46,739 : Processing <TimeStep u'Halo600_DMO_lowres/output_00008' z=13.08 t=0.33 Gyr>
[ 3] 2021-02-05 10:16:46,742 : Missing prerequisites - added class <class 'tangos.properties.pynbody.centring.CentreAndRadius'>
[ 3] 2021-02-05 10:16:46,742 : providing properties ('shrink_center', 'max_radius')
[ 3] 2021-02-05 10:16:46,742 : Gathering existing properties for all halos in timestep <TimeStep u'Halo600_DMO_lowres/output_00008' z=13.08 t=0.33 Gyr>
[ 3] 2021-02-05 10:16:46,762 : User-specified inclusion criterion excluded 0 of 0 halos
[ 3] 2021-02-05 10:16:46,762 : Successfully gathered existing properties; calculating halo properties now...
[ 3] 2021-02-05 10:16:46,763 : 0 halos to consider; 2 property calculations for each of them
[ 3] 2021-02-05 10:16:46,763 : Done with <TimeStep u'Halo600_DMO_lowres/output_00008' z=13.08 t=0.33 Gyr>
[ 3] 2021-02-05 10:16:46,789 : Succeeded: 0 property calculations
[ 3] 2021-02-05 10:16:46,789 : Errored: 0 property calculations
[ 3] 2021-02-05 10:16:46,789 : Errored during load: 0 property calculations
[ 3] 2021-02-05 10:16:46,789 : Already exists: 0 property calculations
[ 3] 2021-02-05 10:16:46,790 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:16:46,790 : Send job 79 of 1097 to node 3
[ 3] 2021-02-05 10:16:46,790 : Processing <TimeStep u'Halo600_DMO_later_mergers/output_00008' z=13.04 t=0.33 Gyr>
[ 3] 2021-02-05 10:16:46,794 : Missing prerequisites - added class <class 'tangos.properties.pynbody.centring.CentreAndRadius'>
[ 3] 2021-02-05 10:16:46,794 : providing properties ('shrink_center', 'max_radius')
[ 3] 2021-02-05 10:16:46,794 : Gathering existing properties for all halos in timestep <TimeStep u'Halo600_DMO_later_mergers/output_00008' z=13.04 t=0.33 Gyr>
[ 3] 2021-02-05 10:16:46,845 : User-specified inclusion criterion excluded 242 of 242 halos
[ 3] 2021-02-05 10:16:46,845 : Successfully gathered existing properties; calculating halo properties now...
[ 3] 2021-02-05 10:16:46,845 : 0 halos to consider; 2 property calculations for each of them
[ 3] 2021-02-05 10:16:46,845 : Done with <TimeStep u'Halo600_DMO_later_mergers/output_00008' z=13.04 t=0.33 Gyr>
[ 3] 2021-02-05 10:16:46,876 : Succeeded: 0 property calculations
[ 3] 2021-02-05 10:16:46,876 : Errored: 0 property calculations
[ 3] 2021-02-05 10:16:46,876 : Errored during load: 0 property calculations
[ 3] 2021-02-05 10:16:46,876 : Already exists: 0 property calculations
[ 3] 2021-02-05 10:16:46,876 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:16:46,876 : Send job 80 of 1097 to node 3
[ 3] 2021-02-05 10:16:46,877 : Processing <TimeStep u'Halo600_RT/output_00009' z=12.66 t=0.34 Gyr>
[ 3] 2021-02-05 10:16:46,878 : Missing prerequisites - added class <class 'tangos.properties.pynbody.centring.CentreAndRadius'>
[ 3] 2021-02-05 10:16:46,878 : providing properties ('shrink_center', 'max_radius')
[ 3] 2021-02-05 10:16:46,879 : Gathering existing properties for all halos in timestep <TimeStep u'Halo600_RT/output_00009' z=12.66 t=0.34 Gyr>
[ 3] 2021-02-05 10:16:46,964 : User-specified inclusion criterion excluded 333 of 358 halos
[ 3] 2021-02-05 10:16:46,973 : Successfully gathered existing properties; calculating halo properties now...
[ 3] 2021-02-05 10:16:46,973 : 25 halos to consider; 2 property calculations for each of them
Traceback (most recent call last):
File "/scratch/dp101/shared/python/anaconda2/bin/tangos", line 11, in <module>
load_entry_point('tangos', 'console_scripts', 'tangos')()
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/scripts/__init__.py", line 49, in main
args.func(args)
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/__init__.py", line 54, in run
parallel_tasks.launch(obj.run_calculation_loop, 2, [])
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/__init__.py", line 54, in launch
backend.launch(_exec_function_or_server, num_procs, [function, args])
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/backends/mpi4py.py", line 61, in launch
function(*args)
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/parallel_tasks/__init__.py", line 86, in _exec_function_or_server
function(*args)
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py", line 508, in run_calculation_loop
self.run_timestep_calculation(f_obj)
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py", line 471, in run_timestep_calculation
self.run_halo_calculation(db_halo, existing_properties)
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py", line 438, in run_halo_calculation
self.run_property_calculation(db_halo, calculator, existing_properties)
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py", line 423, in run_property_calculation
results = self._get_property_value(db_halo, property_calculator, existing_properties)
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/tools/property_writer.py", line 387, in _get_property_value
pdb.post_mortem(tb)
File "/scratch/dp101/shared/python/anaconda2/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/util/timing_monitor.py", line 21, in __call__
self._end()
File "/lustre/dirac/scratch/dp101/shared/python/tangos/tangos/util/timing_monitor.py", line 56, in _end
cumulative_timings = np.diff(self._time_marks)+previous_timings
ValueError: operands could not be broadcast together with shapes (3,) (2,)
[ 2] 2021-02-05 10:18:06,067 : Attempting to commit 2 halo properties...
[ 2] 2021-02-05 10:18:06,522 : 2 properties were committed
[ 2] 2021-02-05 10:18:06,522 : CUMULATIVE RUNNING TIMES (just this node)
[ 2] 2021-02-05 10:18:06,523 : StellarAndGasCenter 18.6s | 100.0%
[ 2] 2021-02-05 10:18:06,523 : ------ INTERNAL BREAKDOWN ------
[ 2] 2021-02-05 10:18:06,523 : start cen 2.0s | 10.6% | 10.6%
[ 2] 2021-02-05 10:18:06,067 : Attempting to commit 2 halo properties...
[ 2] 2021-02-05 10:18:06,524 : --------------------------------
[ 2] 2021-02-05 10:18:06,524 : Done with <TimeStep u'Halo600_fiducial_UVB_nohighzdamping/output_00008' z=13.28 t=0.32 Gyr>
[ 2] 2021-02-05 10:18:07,490 : Succeeded: 1 property calculations
[ 2] 2021-02-05 10:18:07,490 : Errored: 0 property calculations
[ 2] 2021-02-05 10:18:07,490 : Errored during load: 0 property calculations
[ 2] 2021-02-05 10:18:07,490 : Already exists: 1 property calculations
[ 2] 2021-02-05 10:18:07,490 : Missing pre-requisite: 0 property calculations
[ 0] 2021-02-05 10:18:07,491 : Send job 81 of 1097 to node 2
[ 2] 2021-02-05 10:18:07,491 : Processing <TimeStep u'Halo600_RT/output_00010' z=11.50 t=0.39 Gyr>
[ 2] 2021-02-05 10:18:07,497 : Missing prerequisites - added class <class 'tangos.properties.pynbody.centring.CentreAndRadius'>
[ 2] 2021-02-05 10:18:07,497 : providing properties ('shrink_center', 'max_radius')
[ 2] 2021-02-05 10:18:07,500 : Gathering existing properties for all halos in timestep <TimeStep u'Halo600_RT/output_00010' z=11.50 t=0.39 Gyr>
[ 2] 2021-02-05 10:18:07,586 : User-specified inclusion criterion excluded 465 of 513 halos
[ 2] 2021-02-05 10:18:07,600 : Successfully gathered existing properties; calculating halo properties now...
[ 2] 2021-02-05 10:18:07,600 : 48 halos to consider; 2 property calculations for each of them
[ 3] 2021-02-05 10:18:17,734 : Attempting to commit 44 halo properties...
[ 3] 2021-02-05 10:18:18,342 : 44 properties were committed
[ 3] 2021-02-05 10:18:18,342 : CUMULATIVE RUNNING TIMES (just this node)
[ 3] 2021-02-05 10:18:18,342 : StellarAndGasCenter 64.8s | 100.0%
[ 3] 2021-02-05 10:18:18,342 : ------ INTERNAL BREAKDOWN ------
[ 3] 2021-02-05 10:18:18,342 : start cen 1.6s | 2.5% | 2.5%
[ 3] 2021-02-05 10:18:18,342 : cen cen 56.4s | 87.1% | 87.1%
[ 3] 2021-02-05 10:18:18,342 : cen end 6.8s | 10.4% | 10.4%
[ 3] 2021-02-05 10:18:18,342 : --------------------------------
[ 3] 2021-02-05 10:19:18,511 : Attempting to commit 38 halo properties...
[ 3] 2021-02-05 10:19:19,084 : 38 properties were committed
[ 3] 2021-02-05 10:19:19,084 : CUMULATIVE RUNNING TIMES (just this node)
[ 3] 2021-02-05 10:19:19,085 : StellarAndGasCenter 114.8s | 100.0%
[ 3] 2021-02-05 10:19:19,085 : ------ INTERNAL BREAKDOWN ------
[ 3] 2021-02-05 10:19:19,085 : start cen 1.8s | 1.6% | 1.6%
[ 3] 2021-02-05 10:19:19,085 : cen cen 99.8s | 87.0% | 87.0%
[ 3] 2021-02-05 10:19:19,085 : cen end 13.1s | 11.4% | 11.4%
[ 3] 2021-02-05 10:19:19,085 : --------------------------------
It looks like two things are going on here. The primary problem you have is a MemoryError: Unable to create shared memory region which is a common issue with loading ramses simulations where shared memory hacks are required -- probably your system is not allowing enough shared memory to be allocated, or there is 'hanging' shared memory from a previous session. (It is possible that this could be fixed at source in pynbody, but that's a separate issue.)
When it fails in this way, part-way through the calculation, it looks like the timer object is then left in an inconsistent state. The timer code should be made robust to finding itself in such an inconsistent state -- that clearly is an issue for tangos.
apontzen
commented
3 years ago @Martin-Rey Take a look at #136 - this should address the timer crashing
Martin-Rey
commented
3 years ago Sorry I should have said that the shared Memory Error is indeed a sidetrack and a consequence of a small analysis node with Ramses sims -- I had seen the timing failure both with mpi and normal Tangos runs where other Errors get generated before the timer crash -- but this was the log I had available.
Thanks for tracking this down @apontzen, I confirm that I don't see any problems running on #136 with calls that would previously make the timer fail. Feel free to merge and close this issue!
Hi all,
I might have found a small bug in the timer utility. It sometimes throws the following (has happened both in mpi and normal runs) when computing the centering property
The centering property has an internal timing marker: https://github.com/pynbody/tangos/blob/fedf83fb6535d38effaad0c5f481091a04cd1145/tangos/properties/pynbody/centring.py#L18-L22
but I am unsure what is causing this out-of-bound error. Logging the timing successfully writes in other parts of the calculation, e.g. the log 40s earlier on the same call to tangos.
I am on branch PR #134 (although I doubt this makes a difference) and am writing very small number of quick properties that do not always complete (they can throw
RuntimeErrorsfor physical reasons) if that helps.Martin