search

AdaptiveComputationLab / simcov

Other
4 stars 10 forks source link

Sampling error #16

Closed abbypribis closed 3 years ago

abbypribis commented 3 years ago

Hello!

I got an error when I set the sample period for one of my runs running on 14 nodes on Xena at CARC.

Here is the script that I submitted:

!/bin/bash

SBATCH --partition=singleGPU

SBATCH --nodes=14

SBATCH --ntasks-per-node=16

SBATCH --time=2-00:00:00

SBATCH --job-name=simcov_test

SBATCh --output=simcov.out

module load upcxx/2020.10.0-python3-3o75 module load cmake/3.18.4-2lmi

upcxx-run -n $SLURM_NTASKS -N $SLURM_NNODES -- ./install/bin/simcov --config=../simcov-results/configs/tcl_configs/ode_run_tcl_config/ode_run_tcl_20000_foi_600.config --sample-period 1440

The slurm.out file is VERY long, so I put it in this Google Drive folder: https://drive.google.com/drive/folders/16XnZj9sce_Cy5aqJ9MyzXgETjuNOnJdW?usp=sharing

Here is what is in the different files in the simcov-run-n... directory that was generated:

simcov.stats

time incb expr apop dead tvas ttis chem virs

simcov.log

2020-12-16 18:10:30 SimCov version SimCov version 0.1.1.cc7f682-dirty-master built on 20201215121229 2020-12-16 18:10:30 ____ 2020-12-16 18:10:30 Compiler definitions: 2020-12-16 18:10:30 _XOPEN_SOURCE=700 2020-12-16 18:10:30 CONFIG_USE_COLORS 2020-12-16 18:10:30 USE_BYTELL 2020-12-16 18:10:30 MAXVIRIONS=125000 2020-12-16 18:10:30 ____ 2020-12-16 18:10:30 Options: 2020-12-16 18:10:30 dim = 20000 20000 1 2020-12-16 18:10:30 timesteps = 20160 2020-12-16 18:10:30 infections = 600 2020-12-16 18:10:30 infection-coords = [] 2020-12-16 18:10:30 initial-infection = 300000 2020-12-16 18:10:30 infectable-spacing = 1 2020-12-16 18:10:30 incubation-period = 360 2020-12-16 18:10:30 apoptosis-period = 180 2020-12-16 18:10:30 expressing-period = 2057 2020-12-16 18:10:30 infectivity = 0.00247 2020-12-16 18:10:30 virion-production = 35 2020-12-16 18:10:30 virion-decay = 0.0069 2020-12-16 18:10:30 virion-diffusion = 1 2020-12-16 18:10:30 chemokine-production = 0.5 2020-12-16 18:10:30 chemokine-decay = 0.02 2020-12-16 18:10:30 chemokine-diffusion = 0.5 2020-12-16 18:10:30 min-chemokine = 1e-6 2020-12-16 18:10:30 antibody-factor = 1 2020-12-16 18:10:30 antibody-period = 5760 2020-12-16 18:10:30 tcell-generation-rate = 10 2020-12-16 18:10:30 tcell-initial-delay = 20160 2020-12-16 18:10:30 tcell-vascular-period = 5760 2020-12-16 18:10:30 tcell-tissue-period = 1440 2020-12-16 18:10:30 tcell-binding-period = 10 2020-12-16 18:10:30 max-binding-prob = 1 2020-12-16 18:10:30 tcells-follow-gradient = false 2020-12-16 18:10:30 seed = 29 2020-12-16 18:10:30 sample-period = 1440 2020-12-16 18:10:30 sample-resolution = 1 2020-12-16 18:10:30 min-blocks-per-proc = 16 2020-12-16 18:10:30 output = simcov-run-n224-N14-201216181030 2020-12-16 18:10:30 _____ 2020-12-16 18:10:30 Starting run with 224 processes on 14 nodes at 12/16/20 18:10:30 2020-12-16 18:10:30 Pinned processes, with process 0 (pid 3964) pinned to core 0 2020-12-16 18:10:30 Initial free memory across all nodes: 826.18GB 2020-12-16 18:10:30 Starting with 59.00GB free on node 0 2020-12-16 18:10:30 Dividing 400000000 grid points into 6400 squares of size 62500 (250^2), with 29 per process 2020-12-16 18:10:30 Total initial memory required per process is a max of 165.94MB 2020-12-16 18:10:34 --- Elapsed time for tissue.cpp:Tissue: (exiting barrier) 2020-12-16 18:10:34 ... 2020-12-16 18:10:34 ... (min/my/avg/max, bal) (3.60/3.60/3.60/3.60 s, 1.00) barrier(0.00 s) --- 2020-12-16 18:10:34 Memory used on node 0 after initialization is 9.15GB 2020-12-16 18:10:34 ... (min/my/avg/max, bal) No-Time barrier(0.00 s) --- 2020-12-16 18:10:34 # datetime elapsed step incb expr apop dead tvas ttis chem virs <%active lbln> 2020-12-16 18:10:34 Time step 0: initial infection at (13164, 15138, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7466, 1517, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8279, 9922, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18287, 353, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1056, 18379, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3445, 178, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17843, 11989, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4336, 11898, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15944, 15380, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16723, 14120, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3591, 979, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4219, 3463, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12721, 19866, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12424, 18047, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19321, 11421, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16088, 13182, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2895, 2028, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2927, 15591, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7639, 8537, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14203, 7701, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2546, 583, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4866, 6247, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6251, 11004, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2712, 3691, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5398, 650, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7815, 3841, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11180, 3622, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19529, 17890, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16820, 1658, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10619, 12186, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15150, 18572, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5744, 16552, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12026, 17102, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6174, 11242, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10894, 4357, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14836, 9600, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10563, 12001, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3208, 10514, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1351, 5985, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17823, 10198, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2572, 18386, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6125, 6544, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18133, 13571, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17695, 1673, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9263, 3659, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10749, 417, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (563, 18901, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10211, 15130, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14716, 2372, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15647, 18323, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9484, 668, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19391, 12659, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18541, 1281, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12517, 4356, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5683, 4767, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11990, 17515, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13218, 19832, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14752, 19339, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5903, 359, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (918, 7280, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9669, 4767, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6804, 17203, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12451, 19913, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (48, 5501, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6845, 6982, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (708, 8682, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14290, 8932, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17947, 19100, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13667, 9703, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13616, 13773, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12555, 13413, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1783, 3973, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1802, 16324, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16707, 5007, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (507, 13773, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10331, 3325, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11420, 4077, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7540, 10016, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10137, 19169, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12465, 14600, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13573, 6079, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6818, 7022, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9884, 10121, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12936, 4769, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19987, 15346, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (315, 5674, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14180, 1483, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12296, 7663, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15194, 229, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4051, 17493, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14588, 15725, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (603, 9374, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (819, 9033, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6825, 8930, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13342, 12699, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12277, 16456, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5919, 13514, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9849, 5306, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7905, 11143, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9639, 10027, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (613, 5869, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16604, 17093, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13537, 6136, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2236, 5070, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4554, 14752, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8482, 3395, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (294, 5124, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2083, 11848, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4000, 6176, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10301, 12362, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6036, 13877, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1585, 5035, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8930, 15125, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12646, 13071, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2607, 17004, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8576, 13926, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16425, 15934, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6004, 6670, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16780, 1294, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5037, 12029, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14917, 14136, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7888, 18733, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5341, 18165, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14253, 3692, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18001, 5162, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3623, 4874, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19085, 7956, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5098, 1479, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (996, 19632, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5689, 9577, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16061, 11187, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5757, 15225, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7958, 2275, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17965, 2373, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12981, 16099, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5398, 16293, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12435, 7956, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16642, 6020, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18215, 4664, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16884, 16756, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14423, 14112, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1273, 14531, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19708, 9442, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14703, 19689, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14368, 1546, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2466, 7757, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16583, 1286, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11473, 16589, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5482, 10720, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6351, 5836, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12045, 2030, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7445, 3534, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14706, 7688, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (348, 8139, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18124, 7686, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18112, 6069, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15000, 12850, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8731, 16066, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19920, 19377, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16017, 8425, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17289, 16332, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16949, 13541, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11586, 7881, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19484, 1487, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15429, 10674, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2470, 18996, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5175, 17153, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19237, 16445, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13136, 11921, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2798, 5181, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10346, 4716, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18255, 13276, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4275, 5111, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1774, 11106, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14879, 15476, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5683, 16734, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12343, 1359, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15241, 6005, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3483, 13101, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8826, 16680, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18026, 17863, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4185, 11075, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2410, 13875, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15178, 5137, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3703, 17113, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14681, 16020, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10357, 1521, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19706, 16839, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12975, 4729, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19673, 12893, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2696, 13279, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (954, 10290, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (840, 4714, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8408, 9535, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8016, 9580, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15558, 16656, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12879, 10197, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7608, 13137, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15622, 13455, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3234, 163, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12043, 19833, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1636, 17293, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2981, 1057, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6515, 10085, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18587, 19740, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5958, 8593, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7504, 10754, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11051, 16326, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6417, 16980, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16990, 2563, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9813, 5043, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10731, 14625, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10210, 11952, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16003, 12266, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9249, 19676, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15584, 19514, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9779, 15374, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15549, 18785, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8412, 13511, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10126, 7650, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15498, 19463, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5761, 13217, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8152, 666, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1481, 16786, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4971, 1736, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13409, 6234, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10860, 14942, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18574, 15816, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15645, 16592, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16823, 17650, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3847, 9784, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8856, 5540, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (197, 3352, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15623, 7656, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14, 4111, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6259, 16615, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12578, 2466, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12047, 6323, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6825, 14379, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7500, 12693, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16043, 3282, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13128, 11325, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7584, 19418, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5274, 16429, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1037, 15430, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5570, 14310, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15985, 19195, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11985, 19388, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2071, 12780, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (645, 2105, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3479, 18970, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17172, 10716, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11436, 13967, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2387, 2787, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8415, 8355, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8584, 18968, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16260, 11286, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9938, 459, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7946, 8775, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (348, 4192, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5439, 722, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10015, 14390, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5716, 7329, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11542, 12148, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16439, 4831, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1722, 18501, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6145, 3029, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3923, 3465, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11455, 268, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3540, 14560, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17323, 15312, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18527, 14870, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16327, 5447, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3282, 1790, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14108, 9311, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2951, 10288, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16031, 6443, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4983, 3082, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9976, 15753, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1656, 1829, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18264, 12678, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12847, 12653, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (646, 12549, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17809, 4451, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1408, 8909, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7059, 3092, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19442, 3253, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1294, 8553, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18786, 2500, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15683, 561, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (213, 17089, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13248, 1112, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18326, 5984, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3268, 13840, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17845, 4372, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7090, 2365, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1839, 18700, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14757, 3425, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16085, 4244, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4374, 19215, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6026, 18079, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10435, 2767, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17102, 7843, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15081, 512, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11137, 14012, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11885, 2606, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18279, 6719, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2796, 4366, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10131, 16038, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2716, 15195, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4035, 5409, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12043, 1258, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4279, 369, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13413, 18227, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12326, 7204, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11248, 5228, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2562, 19767, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5773, 2621, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1811, 5329, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14973, 14963, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (819, 15881, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3884, 6885, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14329, 6165, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11228, 16871, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15298, 99, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (551, 11307, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17695, 6195, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15947, 4009, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16124, 19147, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10348, 4409, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11521, 8734, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12463, 18430, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16203, 3725, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (176, 13944, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19403, 4098, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12810, 17904, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3386, 14071, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7670, 3646, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3524, 9754, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12297, 10852, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18343, 983, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17462, 15329, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17796, 7367, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15825, 9076, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16271, 655, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6922, 7713, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3581, 6092, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9498, 6410, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2867, 14588, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13018, 18662, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1085, 12154, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3667, 12918, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13745, 1778, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2986, 19502, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9169, 10210, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10264, 14315, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15487, 3976, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18681, 8502, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14389, 7022, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9940, 5250, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19841, 8302, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16897, 5339, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5439, 3811, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10021, 72, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13336, 15678, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16536, 16189, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5506, 17401, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6715, 15841, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17928, 13781, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5733, 6784, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4744, 14558, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17933, 11357, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6557, 1558, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16901, 3168, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10832, 14328, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6880, 18125, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18691, 14076, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16347, 5100, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2097, 19012, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11950, 19507, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16591, 17447, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4029, 10665, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10841, 8120, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (877, 11338, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7484, 8526, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4913, 14769, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11885, 18806, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2581, 13754, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8515, 5232, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11526, 2200, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7037, 3369, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5111, 13153, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1137, 17675, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11774, 15725, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10666, 8032, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16445, 15944, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19495, 14058, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16671, 1605, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14604, 13712, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3913, 17578, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14337, 7279, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5096, 14288, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6188, 11834, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6825, 10040, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13216, 11079, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11378, 14977, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4874, 7822, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5997, 17500, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16190, 17361, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19444, 14021, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2804, 8309, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17879, 5847, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1331, 13536, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10816, 3598, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13320, 16334, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16733, 11694, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5486, 17505, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13458, 7197, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16368, 12745, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10664, 723, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15306, 12277, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15899, 1778, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6391, 7243, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11119, 6784, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2879, 1293, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9583, 10666, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12904, 690, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9645, 9267, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8557, 4875, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11116, 1295, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9870, 894, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7539, 17951, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (790, 256, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19096, 17152, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18431, 3435, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16511, 7629, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2385, 14749, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2940, 12683, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9599, 12994, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18878, 12838, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6800, 12207, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7006, 8064, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17328, 9589, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14101, 18143, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1642, 15904, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3103, 15775, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15757, 6902, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9386, 5804, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14727, 7214, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16995, 4098, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7026, 7840, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3801, 1111, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18046, 6605, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19023, 12253, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15899, 14409, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18036, 13379, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5354, 5738, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17709, 4758, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4709, 10738, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13012, 1360, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19099, 5371, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6877, 3034, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16412, 8799, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3720, 1685, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13733, 18683, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7316, 17943, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10903, 2343, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6984, 14308, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18162, 10137, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1503, 16140, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6309, 13041, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19401, 1068, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17890, 6437, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9114, 3664, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5085, 4303, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16283, 4474, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1483, 10393, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15879, 12033, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9844, 19649, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6835, 8167, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13800, 56, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6853, 15205, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3805, 10312, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7039, 18388, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7133, 11773, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9987, 3683, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10586, 13210, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3573, 11000, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15159, 5762, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9987, 13375, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1422, 15527, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12929, 2746, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5125, 12939, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14530, 2783, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (782, 24, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18842, 1456, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4927, 6578, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17153, 7554, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19483, 18385, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18414, 8899, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14333, 4387, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1443, 7394, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8881, 15040, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1734, 10044, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16268, 3745, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1559, 16963, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11720, 18677, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18290, 5610, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18205, 2200, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5705, 4206, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3840, 5455, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5988, 15659, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8318, 14992, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17100, 6898, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14974, 15784, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12397, 14457, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1303, 11940, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3263, 13931, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (135, 6536, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3062, 10705, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14780, 8262, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4685, 5313, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18230, 2162, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5960, 4302, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7837, 9317, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5281, 9305, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5951, 10852, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15233, 18531, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4822, 12146, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19978, 11352, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17458, 5923, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15795, 16362, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1552, 4980, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6899, 9968, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (636, 13539, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9280, 13891, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14502, 8447, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11813, 15701, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6687, 12893, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12366, 3591, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16225, 9410, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (583, 14131, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17524, 16699, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9476, 3211, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (16066, 10100, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18935, 5904, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18194, 18373, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (725, 10999, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15702, 1087, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19625, 17926, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15236, 4735, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14083, 17776, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17580, 6632, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15007, 4, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9617, 12795, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6825, 19911, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (4809, 5249, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15653, 7400, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10536, 7859, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3893, 7168, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3118, 2137, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9013, 17619, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11622, 6655, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (10834, 8953, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (5151, 5444, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8489, 4118, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (12894, 2376, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6257, 18960, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (9756, 18877, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (999, 10537, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14885, 10734, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14895, 5252, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14300, 13451, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18250, 3419, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11734, 5904, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11003, 8154, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19935, 15803, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (6037, 2420, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3400, 18995, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (15978, 1986, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3242, 7922, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (276, 11801, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18816, 14968, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (18300, 9682, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (3526, 14705, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2568, 16926, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (17199, 16419, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13105, 1054, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (14270, 15453, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1925, 17480, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7327, 7486, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (2626, 12184, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (19161, 2333, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7338, 19239, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8593, 15295, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (1958, 14054, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (13814, 12671, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (11241, 6919, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (8359, 398, 0) 2020-12-16 18:10:34 Time step 0: initial infection at (7191, 17839, 0) 2020-12-16 18:10:34 [12/16/20 18:10:34 0.02s]: 0 600 0 0 0 0 0 0.00e+00 0.00e+00 < 0.000 0.383 >

samples directory

[apribis@xena simcov-run-n224-N14-201216181030]$ cd samples/ [apribis@xena samples]$ ls sample_epicell_0.vtk

Thanks!

abbypribis commented 3 years ago

I did some sleuthing and figured out that this error always occurs on Xena when I try to sample while running on multiple nodes.

For example, I ran the covid_default.config and changed the sample-period parameter to 1440, and I got the same error message in my slurm file as above.

stevenhofmeyr commented 3 years ago

It appears that all process running on any node other than 0 cannot access the file that was set up by process 0. This indicates to me that somehow they are not able to access the same shared file system. What directory are you running from? Usually these sorts of systems have a high speed parallel scratch file system. It appears that Xena has a scratch directory. You'll want to use that one.

abbypribis commented 3 years ago

I was running from inside the simcov directory I cloned into my home directory on Xena.

Does this mean that I should copy the simcov repo into the scratch directory? Or what do you mean by running it from that directory?

Thanks!

stevenhofmeyr commented 3 years ago

You don't need to copy the repo to the scratch directory, but just cd into an appropriate subdirectory in scratch in the batch script, and then execute simcov by specifying the path to the binary.

abbypribis commented 3 years ago

Hmm, this is the modification that I made to the batch script:

!/bin/bash

SBATCH --partition=singleGPU

SBATCH --nodes=2

SBATCH --ntasks-per-node=16

SBATCH --time=2-00:00:00

SBATCH --job-name=simcov_test

SBATCh --output=simcov.out

cd ../xena-scratch

module load upcxx/2020.10.0-python3-3o75 module load cmake/3.18.4-2lmi

upcxx-run -n $SLURM_NTASKS -N $SLURM_NNODES -- ../simcov/install/bin/simcov --config=../simcov/test_sample.config

But now I am getting the following error, I assume I am specifying the path to the binary incorrectly?

You have not been allocated GPUs. To request GPUs, add the following to you r submission script (where X is the number of GPUs to request): #SBATCH --g res=gpu:X

The following have been reloaded with a version change: 1) zlib/1.2.11-pkmj => zlib/1.2.11-lfst

Error: "../simcov/install/bin/simcov --config=../simcov/test_sample.config" does not appear to execute a UPC++/GASNet executable usage: upcxx-run [-h] [-n NUM] [-N NUM] [-shared-heap HEAPSZ] [-backtrace] [-show] [-info] [-ssh-servers HOSTS] [-localhost] [-v] [-E VAR1[,VAR2]] command ...

stevenhofmeyr commented 3 years ago

You should use the absolute path for both the cd and the binary to make sure it all works. I'd expect that you have a directory with your name under scratch, eg

/xena/scratch/username

Where username is your username.

You can check your path by making sure you can cd to that directory first before you submit the batch script, and also make sure from that directory that you have the correct path to the binary.

On Thu, Dec 17, 2020, 18:41 Abby Pribisova notifications@github.com wrote:

Hmm, this is the modification that I made to the batch script:

!/bin/bash

SBATCH --partition=singleGPU

SBATCH --nodes=2

SBATCH --ntasks-per-node=16

SBATCH --time=2-00:00:00

SBATCH --job-name=simcov_test

SBATCh --output=simcov.out

cd ../xena-scratch

module load upcxx/2020.10.0-python3-3o75 module load cmake/3.18.4-2lmi

upcxx-run -n $SLURM_NTASKS -N $SLURM_NNODES -- ../simcov/install/bin/simcov --config=../simcov/test_sample.config

But now I am getting the following error, I assume I am specifying the path to the binary incorrectly?

You have not been allocated GPUs. To request GPUs, add the following to you r submission script (where X is the number of GPUs to request): #SBATCH --g res=gpu:X

The following have been reloaded with a version change:

  1. zlib/1.2.11-pkmj => zlib/1.2.11-lfst

Error: "../simcov/install/bin/simcov --config=../simcov/test_sample.config" does not appear to execute a UPC++/GASNet executable usage: upcxx-run [-h] [-n NUM] [-N NUM] [-shared-heap HEAPSZ] [-backtrace] [-show] [-info] [-ssh-servers HOSTS] [-localhost] [-v] [-E VAR1[,VAR2]] command ...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AdaptiveComputationLab/simcov/issues/16#issuecomment-747830880, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2BOCFJHJTHUM4MMKFYRZLSVK6MPANCNFSM4U7CRSIA .

abbypribis commented 3 years ago

Ok, I specified the absolute path to xena-scratch and the binary, and it ran this time - but I got the same error as my first error:

datetime elapsed step incb expr apop dead tvas ttis chem virs <%active lbln> [12/17/20 20:11:21 0.00s]: 0 1 0 0 0 0 0 0.00e+00 0.00e+00 < 0.000 0.031 > terminate called after throwing an instance of 'terminate called after throwing an instance of 'std::runtime_error' what(): [18] ERROR: Cannot open file samples/sample_epicell_0.vtk: No such file or directory

I might be misunderstanding the scratch directory. In my home directory, I have:

[apribis@xena ~]$ ls simcov simcov-results wheeler-scratch xena-scratch

After running the script that cd's into xena-scratch, the run directory along with simcov.stats, samples directory, etc. showed up in xena-scratch:

[apribis@xena ~]$ cd xena-scratch/ [apribis@xena xena-scratch]$ ls simcov-run-n32-N2-201217201120 [apribis@xena xena-scratch]$ cd simcov-run-n32-N2-201217201120/ [apribis@xena simcov-run-n32-N2-201217201120]$ ls samples simcov.config simcov.log simcov.stats

Btw, I get the same result if I cd into /xena/scratch/apribis/ as opposed to /users/apribis/xena-scratch/.

Maybe the scratch space is not shared between nodes?

abbypribis commented 3 years ago

Set up a ticket with CARC - will keep y'all updated on what they say

abbypribis commented 3 years ago

Set up a ticket with CARC - will keep y'all updated on what they say

I just want to confirm - are we using MPI for multi-node?

stevenhofmeyr commented 3 years ago

It may be using MPI as a job spawner (unless it is using PMI, not sure which). Other than that, it does not use MPI - gasnet provides native support for infiniband via its ibv network conduit.

abbypribis commented 3 years ago

Ok, overall, this seems like an issue with how CARC loaded upcxx. The problem is that the file names for the vtk files cannot be relative paths (so, "simcov-run..../samples/sample.vtk"); they need to be absolute paths. Otherwise, we get the above "No such file or directory" error.

The hack is to save the vtk files to your home directory instead of the samples directory inside the "simcov-run..." directory.

Therefore, here is my horrible hack to get rid of that error:

  1. mkdir samples
  2. emacs src/main.cpp
  3. Change line 421 from string fname = "samples/sample_" + view_object_str(view_object) + "_" + to_string(time_step) + ".vtk"; TO string fname = "/users/<user>/simcov/samples/sample_" + view_object_str(view_object) + "_" + to_string(time_step) + ".vtk";
  4. export UPCXX_NETWORK=ibv
  5. ./build.sh clean && ./build.sh Release

Also, there are two fixes in the xena_simcov_run.sh script:

  1. Change line 9 from module load upcxx/2020.10.0-python3-3o75 TO module load upcxx/2020.10.0-python3-tsz6
  2. Change line 12 -N $SLURM_NNODES TO -N $SLURM_JOB_NUM_NODES

The reason we switched modules is that we suspected that the new module (3o75) was compiling with an MPI compiler, which was causing loading files from relative paths to fail. The old module (tsz6) does NOT have MPI support. Whether that is the issue or not is uncertain...

If anyone has any other suggestions, I'd be happy to hear them :)

stevenhofmeyr commented 3 years ago

I just pushed a change to master which uses getcwd where you are hardcoding the path. Can you try that out and see if it works?

stevenhofmeyr commented 3 years ago

And try it without explicitly creating the samples directory - let's see if simcov will do that as it should.

abbypribis commented 3 years ago

Hmm, similar error, but now with the absolute path.

[19] ERROR: Cannot open file /users/apribis/simcov/samples/sample_epicell_0.vtk: No such file or directory

I noticed that a bunch of other files changed when I pulled master, so I'll take a look and see if anything in there could be the culprit.

stevenhofmeyr commented 3 years ago

So is your output directory named "simcov"? With the latest master, I'd expect the failure to point to something like:

/users/apribis//samples

BTW, are you sure you can't use the scratch directory? It really is the place for the output...

abbypribis commented 3 years ago

It should deposit the vtk files in the samples directory that I manually created within the original simcov directory that I pulled from GitHub.

I did notice this error occurring sometimes - perhaps a missing package?

which: no lfs in (/opt/spack/opt/spack/linux-centos7-haswell/gcc-4.8.5/gcc-10.2.0-3kjqvw7masskmxqtxblo5khyshwe6zuw/bin:/opt/spack/opt/spack/linux-centos7-haswell/gcc-4.8.5/zstd-1.4.5-3boiausohz77fqap2ptmbwpoyy5a4ufg/bin:/opt/spack/bin:/usr/local/bin:/usr/bin)

stevenhofmeyr commented 3 years ago

That is harmless - the script looks for a lustre file system and attempts to configure the output directory if it is lustre. Otherwise it just prints a warning and continues.

stevenhofmeyr commented 3 years ago

Fixed - problem was using a timestamp in the default output directory with different ranks generating different timestamps. Now no timestamps are used.