Open icasallas opened 4 years ago
hvasbath
commented
4 years ago Ah ok saw that you posted everything. The last points make no sense why they dont work- I have not the slightest clue.
The workaround for now would be to downgrade. Does it work using the --oversubscribe option ergo running any of the mpiexec commands above for example with -n 9
icasallas
commented
4 years ago Hi. I put n_chains= 11 and ran beat sample Laquila_dc; as it was expected I got the error:
distributed - INFO MPI is installed! distributed - INFO Detected python3 distributed - INFO Done initialising mpi runner distributed - INFO sampler directory: /home/lab/bayesiano/beat/beat/sampler/pt.py distributed - DEBUG Running mpi with arguments: -n 12 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
However running in the temporary directory:
mpiexec --use-hwthread-cpus -n 12 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
I did not get any error and mpiexec started to execute the process.
When n_chains=12; mpiexec did not work.
Thanks.
hvasbath
commented
4 years ago Sure as this will try to create 13 processes which computer doesn't have threads. I was asking for the --oversubscribe flag. Did you try that? For example: mpiexec --oversubscribe -n 13 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
icasallas
commented
4 years ago Hi.
I ran n_chains=12 and n_chains=16:
mpiexec --oversubscribe -n 13 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
mpiexec --oversubscribe -n 17 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
I did not get any error and mpiexec started to execute the process.
hvasbath
commented
4 years ago Great so thats the flag then we have to include in the execution string. We will have to check if it is available across versions- I will do that.
hvasbath
commented
4 years ago This seems to have deeper roots connected to issues with MPI distributions. Collecting here a bunch of relevant threads to further investigate: https://github.com/h5py/h5py/pull/1117 https://github.com/radiasoft/sirepo/issues/1311 https://github.com/radiasoft/sirepo/issues/1323
hvasbath
commented
4 years ago A fast workaround for that issue could be to ensure to use older MPI versions (<3): sudo apt install openmpi-bin=2.1.1-8 libopenmpi-dev=2.1.1-8 -V
Also updated webpage install instructions with that...
hvasbath
commented
4 years ago need to cleanup master. ..
hvasbath
commented
4 years ago OpenMPI version 3.3.2 works with mpi4py earlier versions in 3 have the init issue: https://bitbucket.org/mpi4py/mpi4py/issues/95/mpi4py-openmpi-300-breaks-subprocess mpi4py does not support OpenMPI4
hvasbath
commented
3 years ago Users that used anaconda reported that this issue could be fixed installing mpi4py through conda-forge like this: conda install -c conda-forge mpi4py
Hi all,
1) Running the second example with n_chains=8, I got:
(beat) lab@lab:~$ beat sample Laquila_dc config - INFO All hierarchicals ok! config - INFO All hyperparameters ok! config - INFO All priors ok! models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ... models - INFO ---------------------
heart - INFO Consistent number of datasets and targets in any_P_0 wavemap! heart - INFO Number of seismic datasets for wavemap: any_P_0: 35 models - INFO ... Building model ...
models - INFO Optimization for 35 hyperparameters in total! seismic - INFO Seismic optimization on: depth, dip, duration, east_shift, magnitude, north_shift, rake, strike, time seismic - INFO Retrieving seismic data-covariances with structure "variance" for any_P_0 ... seismic - INFO Preparing data of "any_P_0" for optimization seismic - INFO Initializing synthesizer for "any_P_0" models - INFO Model building was successful!
models - INFO Using "bin" backend to store samples! models - INFO ... Initiate Metropolis for Parallel Tempering... proposal_distribution: Normal, tune_interval=10, n_chains=8
models.base - INFO ... Starting Parallel Tempering ...
distributed - INFO MPI is installed! distributed - INFO Detected python3 distributed - INFO Done initialising mpi runner distributed - INFO sampler directory: /home/lab/bayesiano/beat/beat/sampler/pt.py Traceback (most recent call last): File "/home/lab/anaconda3/envs/beat/bin/beat", line 11, in
load_entry_point('beat==1.0', 'console_scripts', 'beat')()
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x8664.egg/beat/apps/beat.py", line 1686, in main
globals()['command' + command](args)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 775, in command_sample
sample(step, problem)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/base.py", line 196, in sample
record_worker_chains=pa.record_worker_chains)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/pt.py", line 836, in pt_sample
project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 226, in run_mpi_sampler
samplerdir, n_jobs=n_jobs, loglevel=loglevel, project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 158, in run
in the directory %s'''.lstrip() % (program, self.tempdir))
beat.sampler.distributed.MPIError: mpiexec has been invoked as "mpiexec"
in the directory /tmp/mpiexec-56qm7lyu
distributed - WARNING not removing temporary directory: /tmp/mpiexec-56qm7lyu
------------------------------------------------------------*-
2. Running again in --loglevel=debug:
(beat) lab@lab:~$ beat sample Laquila_dc config - INFO All hierarchicals ok! config - INFO All hyperparameters ok! config - INFO All priors ok! models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ... models - INFO ---------------------
heart - INFO Consistent number of datasets and targets in any_P_0 wavemap! heart - INFO Number of seismic datasets for wavemap: any_P_0: 35 models - INFO ... Building model ...
models - INFO Optimization for 35 hyperparameters in total! seismic - INFO Seismic optimization on: depth, dip, duration, east_shift, magnitude, north_shift, rake, strike, time seismic - INFO Retrieving seismic data-covariances with structure "variance" for any_P_0 ... seismic - INFO Preparing data of "any_P_0" for optimization seismic - INFO Initializing synthesizer for "any_P_0" models - INFO Model building was successful!
models - INFO Using "bin" backend to store samples! models - INFO ... Initiate Metropolis for Parallel Tempering... proposal_distribution: Normal, tune_interval=10, n_chains=8
models.base - INFO ... Starting Parallel Tempering ...
distributed - INFO MPI is installed! distributed - INFO Detected python3 distributed - INFO Done initialising mpi runner distributed - INFO sampler directory: /home/lab/bayesiano/beat/beat/sampler/pt.py Traceback (most recent call last): File "/home/lab/anaconda3/envs/beat/bin/beat", line 11, in
load_entry_point('beat==1.0', 'console_scripts', 'beat')()
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x8664.egg/beat/apps/beat.py", line 1686, in main
globals()['command' + command](args)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 775, in command_sample
sample(step, problem)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/base.py", line 196, in sample
record_worker_chains=pa.record_worker_chains)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/pt.py", line 836, in pt_sample
project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 226, in run_mpi_sampler
samplerdir, n_jobs=n_jobs, loglevel=loglevel, project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 158, in run
in the directory %s'''.lstrip() % (program, self.tempdir))
beat.sampler.distributed.MPIError: mpiexec has been invoked as "mpiexec"
in the directory /tmp/mpiexec-56qm7lyu
distributed - WARNING not removing temporary directory: /tmp/mpiexec-56qm7lyu
(beat) lab@lab:~$ beat sample Laquila_dc --loglevel=debug
config - INFO All hierarchicals ok!
config - INFO All hyperparameters ok!
config - INFO All priors ok!
models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ... models - INFO ---------------------
seismic - DEBUG Setting up seismic structure ...
utility - DEBUG Valid distance range: [30.000000, 90.000000]! utility - DEBUG Distance of station TSUM: 61.349855 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station RCBR: 65.567308 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station RES: 54.143590 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station KMI: 73.109289 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station SSPA: 64.946397 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station BTDF: 89.437223 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station ANWB: 67.439618 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station MBO: 38.167185 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station TEIG: 84.816714 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station KBL: 43.633450 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station MSEY: 60.417579 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station DBIC: 38.986606 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station HIA: 68.244425 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station YAK: 64.054778 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station QIZ: 82.062195 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station KBS: 36.698309 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station LSZ: 58.931762 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station TLY: 58.456908 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station INCN: 79.971426 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station DWPF: 74.925103 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station WMQ: 52.564826 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station AAK: 44.192670 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station XAN: 71.681441 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station KMBO: 48.335702 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station KURK: 44.334011 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station ADYE: 36.168619 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station SDV: 79.664513 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station ARU: 31.995457 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station SAML: 86.199701 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station ALE: 46.405043 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station OTAV: 91.209244 [deg] utility - DEBUG Outside defined distance range! utility - DEBUG Distance of station ABKT: 34.203018 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station AKTO: 31.451507 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station BJT: 72.303754 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station SSE: 81.349799 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station CHTO: 74.397519 [deg] utility - DEBUG Inside defined distance range! heart - INFO Consistent number of datasets and targets in any_P_0 wavemap! heart - INFO Number of seismic datasets for wavemap: any_P_0: 35 models - INFO ... Building model ...
models - DEBUG Optimization for 1 sources models - INFO Optimization for 35 hyperparameters in total! seismic - INFO Seismic optimization on: depth, dip, duration, east_shift, magnitude, north_shift, rake, strike, time seismic - INFO Retrieving seismic data-covariances with structure "variance" for any_P_0 ... heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 61.349855 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 65.567308 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 54.143590 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 73.109289 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 64.946397 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 89.437223 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 67.439618 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 38.167185 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 84.816714 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 43.633450 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 60.417579 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 38.986606 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 68.244425 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 64.054778 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 82.062195 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 36.698309 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 58.931762 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 58.456908 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 79.971426 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 74.925103 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 52.564826 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 44.192670 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 71.681441 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 48.335702 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 44.334011 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 36.168619 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 79.664513 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 31.995457 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 86.199701 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 46.405043 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 34.203018 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 31.451507 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 72.303754 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 81.349799 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 74.397519 [deg] seismic - INFO Preparing data of "any_P_0" for optimization heart - DEBUG Arrival time for wavename "any_P" distance 61.349855 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 65.567308 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 54.143590 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 73.109289 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 64.946397 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 89.437223 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 67.439618 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 38.167185 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 84.816714 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 43.633450 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 60.417579 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 38.986606 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 68.244425 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 64.054778 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 82.062195 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 36.698309 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 58.931762 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 58.456908 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 79.971426 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 74.925103 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 52.564826 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 44.192670 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 71.681441 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 48.335702 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 44.334011 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 36.168619 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 79.664513 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 31.995457 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 86.199701 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 46.405043 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 34.203018 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 31.451507 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 72.303754 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 81.349799 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 74.397519 [deg] heart - DEBUG Pre-processing data ... heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1641 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982169.0 b: 1238982174.0 c: 1238982224.0 d: 1238982239.0
heart - DEBUG trace: Trace (IU, TSUM, filt, Z) timerange: 2009-04-06 01:32:44 - 2009-04-06 02:27:24 delta t: 2
heart - DEBUG chopped trace: Trace (IU, TSUM, filt, Z) timerange: 2009-04-06 01:42:54 - 2009-04-06 01:43:42 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1716 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982197.0 b: 1238982202.0 c: 1238982252.0 d: 1238982267.0
heart - DEBUG trace: Trace (IU, RCBR, filt, Z) timerange: 2009-04-06 01:33:12 - 2009-04-06 02:30:22 delta t: 2
heart - DEBUG chopped trace: Trace (IU, RCBR, filt, Z) timerange: 2009-04-06 01:43:22 - 2009-04-06 01:44:10 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1508 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982119.0 b: 1238982124.0 c: 1238982174.0 d: 1238982189.0
heart - DEBUG trace: Trace (CN, RES, filt, Z) timerange: 2009-04-06 01:31:52 - 2009-04-06 02:22:06 delta t: 2
heart - DEBUG chopped trace: Trace (CN, RES, filt, Z) timerange: 2009-04-06 01:42:04 - 2009-04-06 01:42:52 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1853 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982245.0 b: 1238982250.0 c: 1238982300.0 d: 1238982315.0
heart - DEBUG trace: Trace (IC, KMI, filt, Z) timerange: 2009-04-06 01:34:00 - 2009-04-06 02:35:44 delta t: 2
heart - DEBUG chopped trace: Trace (IC, KMI, filt, Z) timerange: 2009-04-06 01:44:10 - 2009-04-06 01:44:58 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1702 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982193.0 b: 1238982198.0 c: 1238982248.0 d: 1238982263.0
heart - DEBUG trace: Trace (IU, SSPA, filt, Z) timerange: 2009-04-06 01:33:08 - 2009-04-06 02:29:50 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SSPA, filt, Z) timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:06 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2169 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982331.0 b: 1238982336.0 c: 1238982386.0 d: 1238982401.0
heart - DEBUG trace: Trace (MS, BTDF, filt, Z) timerange: 2009-04-06 01:35:26 - 2009-04-06 02:47:42 delta t: 2
heart - DEBUG chopped trace: Trace (MS, BTDF, filt, Z) timerange: 2009-04-06 01:45:36 - 2009-04-06 01:46:24 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1748 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982209.0 b: 1238982214.0 c: 1238982264.0 d: 1238982279.0
heart - DEBUG trace: Trace (CU, ANWB, filt, Z) timerange: 2009-04-06 01:33:24 - 2009-04-06 02:31:38 delta t: 2
heart - DEBUG chopped trace: Trace (CU, ANWB, filt, Z) timerange: 2009-04-06 01:43:34 - 2009-04-06 01:44:22 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1229 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981993.0 b: 1238981998.0 c: 1238982048.0 d: 1238982063.0
heart - DEBUG trace: Trace (G, MBO, filt, Z) timerange: 2009-04-06 01:29:48 - 2009-04-06 02:10:44 delta t: 2
heart - DEBUG chopped trace: Trace (G, MBO, filt, Z) timerange: 2009-04-06 01:39:58 - 2009-04-06 01:40:46 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2078 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982309.0 b: 1238982314.0 c: 1238982364.0 d: 1238982379.0
heart - DEBUG trace: Trace (IU, TEIG, filt, Z) timerange: 2009-04-06 01:35:02 - 2009-04-06 02:44:16 delta t: 2
heart - DEBUG chopped trace: Trace (IU, TEIG, filt, Z) timerange: 2009-04-06 01:45:14 - 2009-04-06 01:46:02 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1324 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982039.0 b: 1238982044.0 c: 1238982094.0 d: 1238982109.0
heart - DEBUG trace: Trace (IU, KBL, filt, Z) timerange: 2009-04-06 01:30:32 - 2009-04-06 02:14:38 delta t: 2
heart - DEBUG chopped trace: Trace (IU, KBL, filt, Z) timerange: 2009-04-06 01:40:44 - 2009-04-06 01:41:32 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1624 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982163.0 b: 1238982168.0 c: 1238982218.0 d: 1238982233.0
heart - DEBUG trace: Trace (II, MSEY, filt, Z) timerange: 2009-04-06 01:32:38 - 2009-04-06 02:26:44 delta t: 2
heart - DEBUG chopped trace: Trace (II, MSEY, filt, Z) timerange: 2009-04-06 01:42:48 - 2009-04-06 01:43:36 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1244 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982001.0 b: 1238982006.0 c: 1238982056.0 d: 1238982071.0
heart - DEBUG trace: Trace (GT, DBIC, filt, Z) timerange: 2009-04-06 01:29:54 - 2009-04-06 02:11:20 delta t: 2
heart - DEBUG chopped trace: Trace (GT, DBIC, filt, Z) timerange: 2009-04-06 01:40:06 - 2009-04-06 01:40:54 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1763 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982215.0 b: 1238982220.0 c: 1238982270.0 d: 1238982285.0
heart - DEBUG trace: Trace (IC, HIA, filt, Z) timerange: 2009-04-06 01:33:30 - 2009-04-06 02:32:14 delta t: 2
heart - DEBUG chopped trace: Trace (IC, HIA, filt, Z) timerange: 2009-04-06 01:43:40 - 2009-04-06 01:44:28 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1686 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982187.0 b: 1238982192.0 c: 1238982242.0 d: 1238982257.0
heart - DEBUG trace: Trace (IU, YAK, filt, Z) timerange: 2009-04-06 01:33:02 - 2009-04-06 02:29:12 delta t: 2
heart - DEBUG chopped trace: Trace (IU, YAK, filt, Z) timerange: 2009-04-06 01:43:12 - 2009-04-06 01:44:00 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2026 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982295.0 b: 1238982300.0 c: 1238982350.0 d: 1238982365.0
heart - DEBUG trace: Trace (IC, QIZ, filt, Z) timerange: 2009-04-06 01:34:48 - 2009-04-06 02:42:18 delta t: 2
heart - DEBUG chopped trace: Trace (IC, QIZ, filt, Z) timerange: 2009-04-06 01:45:00 - 2009-04-06 01:45:48 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1201 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981981.0 b: 1238981986.0 c: 1238982036.0 d: 1238982051.0
heart - DEBUG trace: Trace (GE, KBS, filt, Z) timerange: 2009-04-06 01:29:34 - 2009-04-06 02:09:34 delta t: 2
heart - DEBUG chopped trace: Trace (GE, KBS, filt, Z) timerange: 2009-04-06 01:39:46 - 2009-04-06 01:40:34 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1598 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982153.0 b: 1238982158.0 c: 1238982208.0 d: 1238982223.0
heart - DEBUG trace: Trace (IU, LSZ, filt, Z) timerange: 2009-04-06 01:32:28 - 2009-04-06 02:25:42 delta t: 2
heart - DEBUG chopped trace: Trace (IU, LSZ, filt, Z) timerange: 2009-04-06 01:42:38 - 2009-04-06 01:43:26 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1586 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982149.0 b: 1238982154.0 c: 1238982204.0 d: 1238982219.0
heart - DEBUG trace: Trace (II, TLY, filt, Z) timerange: 2009-04-06 01:32:24 - 2009-04-06 02:25:14 delta t: 2
heart - DEBUG chopped trace: Trace (II, TLY, filt, Z) timerange: 2009-04-06 01:42:34 - 2009-04-06 01:43:22 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1984 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982283.0 b: 1238982288.0 c: 1238982338.0 d: 1238982353.0
heart - DEBUG trace: Trace (IU, INCN, filt, Z) timerange: 2009-04-06 01:34:38 - 2009-04-06 02:40:44 delta t: 2
heart - DEBUG chopped trace: Trace (IU, INCN, filt, Z) timerange: 2009-04-06 01:44:48 - 2009-04-06 01:45:36 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1888 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982255.0 b: 1238982260.0 c: 1238982310.0 d: 1238982325.0
heart - DEBUG trace: Trace (IU, DWPF, filt, Z) timerange: 2009-04-06 01:34:08 - 2009-04-06 02:37:02 delta t: 2
heart - DEBUG chopped trace: Trace (IU, DWPF, filt, Z) timerange: 2009-04-06 01:44:20 - 2009-04-06 01:45:08 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1481 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982107.0 b: 1238982112.0 c: 1238982162.0 d: 1238982177.0
heart - DEBUG trace: Trace (IC, WMQ, filt, Z) timerange: 2009-04-06 01:31:42 - 2009-04-06 02:21:02 delta t: 2
heart - DEBUG chopped trace: Trace (IC, WMQ, filt, Z) timerange: 2009-04-06 01:41:52 - 2009-04-06 01:42:40 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1334 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982043.0 b: 1238982048.0 c: 1238982098.0 d: 1238982113.0
heart - DEBUG trace: Trace (II, AAK, filt, Z) timerange: 2009-04-06 01:30:36 - 2009-04-06 02:15:02 delta t: 2
heart - DEBUG chopped trace: Trace (II, AAK, filt, Z) timerange: 2009-04-06 01:40:48 - 2009-04-06 01:41:36 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1827 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982235.0 b: 1238982240.0 c: 1238982290.0 d: 1238982305.0
heart - DEBUG trace: Trace (IC, XAN, filt, Z) timerange: 2009-04-06 01:33:50 - 2009-04-06 02:34:42 delta t: 2
heart - DEBUG chopped trace: Trace (IC, XAN, filt, Z) timerange: 2009-04-06 01:44:00 - 2009-04-06 01:44:48 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1408 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982075.0 b: 1238982080.0 c: 1238982130.0 d: 1238982145.0
heart - DEBUG trace: Trace (IU, KMBO, filt, Z) timerange: 2009-04-06 01:31:10 - 2009-04-06 02:18:04 delta t: 2
heart - DEBUG chopped trace: Trace (IU, KMBO, filt, Z) timerange: 2009-04-06 01:41:20 - 2009-04-06 01:42:08 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1335 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982045.0 b: 1238982050.0 c: 1238982100.0 d: 1238982115.0
heart - DEBUG trace: Trace (II, KURK, filt, Z) timerange: 2009-04-06 01:30:38 - 2009-04-06 02:15:06 delta t: 2
heart - DEBUG chopped trace: Trace (II, KURK, filt, Z) timerange: 2009-04-06 01:40:50 - 2009-04-06 01:41:38 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1195 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981977.0 b: 1238981982.0 c: 1238982032.0 d: 1238982047.0
heart - DEBUG trace: Trace (ZF, ADYE, filt, Z) timerange: 2009-04-06 01:29:32 - 2009-04-06 02:09:20 delta t: 2
heart - DEBUG chopped trace: Trace (ZF, ADYE, filt, Z) timerange: 2009-04-06 01:39:42 - 2009-04-06 01:40:30 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1978 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982281.0 b: 1238982286.0 c: 1238982336.0 d: 1238982351.0
heart - DEBUG trace: Trace (IU, SDV, filt, Z) timerange: 2009-04-06 01:34:36 - 2009-04-06 02:40:30 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SDV, filt, Z) timerange: 2009-04-06 01:44:46 - 2009-04-06 01:45:34 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1120 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981941.0 b: 1238981946.0 c: 1238981996.0 d: 1238982011.0
heart - DEBUG trace: Trace (II, ARU, filt, Z) timerange: 2009-04-06 01:28:54 - 2009-04-06 02:06:12 delta t: 2
heart - DEBUG chopped trace: Trace (II, ARU, filt, Z) timerange: 2009-04-06 01:39:06 - 2009-04-06 01:39:54 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2105 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982315.0 b: 1238982320.0 c: 1238982370.0 d: 1238982385.0
heart - DEBUG trace: Trace (IU, SAML, filt, Z) timerange: 2009-04-06 01:35:10 - 2009-04-06 02:45:18 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SAML, filt, Z) timerange: 2009-04-06 01:45:20 - 2009-04-06 01:46:08 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1370 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982061.0 b: 1238982066.0 c: 1238982116.0 d: 1238982131.0
heart - DEBUG trace: Trace (II, ALE, filt, Z) timerange: 2009-04-06 01:30:54 - 2009-04-06 02:16:32 delta t: 2
heart - DEBUG chopped trace: Trace (II, ALE, filt, Z) timerange: 2009-04-06 01:41:06 - 2009-04-06 01:41:54 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1159 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981959.0 b: 1238981964.0 c: 1238982014.0 d: 1238982029.0
heart - DEBUG trace: Trace (II, ABKT, filt, Z) timerange: 2009-04-06 01:29:14 - 2009-04-06 02:07:50 delta t: 2
heart - DEBUG chopped trace: Trace (II, ABKT, filt, Z) timerange: 2009-04-06 01:39:24 - 2009-04-06 01:40:12 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1110 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981935.0 b: 1238981940.0 c: 1238981990.0 d: 1238982005.0
heart - DEBUG trace: Trace (KZ, AKTO, filt, Z) timerange: 2009-04-06 01:28:50 - 2009-04-06 02:05:48 delta t: 2
heart - DEBUG chopped trace: Trace (KZ, AKTO, filt, Z) timerange: 2009-04-06 01:39:00 - 2009-04-06 01:39:48 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1838 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982239.0 b: 1238982244.0 c: 1238982294.0 d: 1238982309.0
heart - DEBUG trace: Trace (IC, BJT, filt, Z) timerange: 2009-04-06 01:33:54 - 2009-04-06 02:35:08 delta t: 2
heart - DEBUG chopped trace: Trace (IC, BJT, filt, Z) timerange: 2009-04-06 01:44:04 - 2009-04-06 01:44:52 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2012 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982291.0 b: 1238982296.0 c: 1238982346.0 d: 1238982361.0
heart - DEBUG trace: Trace (IC, SSE, filt, Z) timerange: 2009-04-06 01:34:44 - 2009-04-06 02:41:46 delta t: 2
heart - DEBUG chopped trace: Trace (IC, SSE, filt, Z) timerange: 2009-04-06 01:44:56 - 2009-04-06 01:45:44 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1879 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982251.0 b: 1238982256.0 c: 1238982306.0 d: 1238982321.0
heart - DEBUG trace: Trace (IU, CHTO, filt, Z) timerange: 2009-04-06 01:34:06 - 2009-04-06 02:36:42 delta t: 2
heart - DEBUG chopped trace: Trace (IU, CHTO, filt, Z) timerange: 2009-04-06 01:44:16 - 2009-04-06 01:45:04 delta t: 2
heart - DEBUG Returning chopped traces ... seismic - INFO Initializing synthesizer for "any_P_0" heart - DEBUG Synthetics generation time: 0.017754 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982169.0 b: 1238982174.0 c: 1238982224.0 d: 1238982239.0
heart - DEBUG trace: Trace (IU, TSUM, 0, Z) timerange: 2009-04-06 01:42:38 - 2009-04-06 01:44:18 delta t: 2
heart - DEBUG chopped trace: Trace (IU, TSUM, 0, Z) timerange: 2009-04-06 01:42:54 - 2009-04-06 01:43:42 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982197.0 b: 1238982202.0 c: 1238982252.0 d: 1238982267.0
heart - DEBUG trace: Trace (IU, RCBR, 0, Z) timerange: 2009-04-06 01:43:06 - 2009-04-06 01:44:46 delta t: 2
heart - DEBUG chopped trace: Trace (IU, RCBR, 0, Z) timerange: 2009-04-06 01:43:22 - 2009-04-06 01:44:10 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982119.0 b: 1238982124.0 c: 1238982174.0 d: 1238982189.0
heart - DEBUG trace: Trace (CN, RES, 0, Z) timerange: 2009-04-06 01:41:48 - 2009-04-06 01:43:28 delta t: 2
heart - DEBUG chopped trace: Trace (CN, RES, 0, Z) timerange: 2009-04-06 01:42:04 - 2009-04-06 01:42:52 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982245.0 b: 1238982250.0 c: 1238982300.0 d: 1238982315.0
heart - DEBUG trace: Trace (IC, KMI, 0, Z) timerange: 2009-04-06 01:43:54 - 2009-04-06 01:45:34 delta t: 2
heart - DEBUG chopped trace: Trace (IC, KMI, 0, Z) timerange: 2009-04-06 01:44:10 - 2009-04-06 01:44:58 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982193.0 b: 1238982198.0 c: 1238982248.0 d: 1238982263.0
heart - DEBUG trace: Trace (IU, SSPA, 0, Z) timerange: 2009-04-06 01:43:02 - 2009-04-06 01:44:42 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SSPA, 0, Z) timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:06 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982331.0 b: 1238982336.0 c: 1238982386.0 d: 1238982401.0
heart - DEBUG trace: Trace (MS, BTDF, 0, Z) timerange: 2009-04-06 01:45:20 - 2009-04-06 01:47:00 delta t: 2
heart - DEBUG chopped trace: Trace (MS, BTDF, 0, Z) timerange: 2009-04-06 01:45:36 - 2009-04-06 01:46:24 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982209.0 b: 1238982214.0 c: 1238982264.0 d: 1238982279.0
heart - DEBUG trace: Trace (CU, ANWB, 0, Z) timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:58 delta t: 2
heart - DEBUG chopped trace: Trace (CU, ANWB, 0, Z) timerange: 2009-04-06 01:43:34 - 2009-04-06 01:44:22 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981993.0 b: 1238981998.0 c: 1238982048.0 d: 1238982063.0
heart - DEBUG trace: Trace (G, MBO, 0, Z) timerange: 2009-04-06 01:39:42 - 2009-04-06 01:41:22 delta t: 2
heart - DEBUG chopped trace: Trace (G, MBO, 0, Z) timerange: 2009-04-06 01:39:58 - 2009-04-06 01:40:46 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982309.0 b: 1238982314.0 c: 1238982364.0 d: 1238982379.0
heart - DEBUG trace: Trace (IU, TEIG, 0, Z) timerange: 2009-04-06 01:44:58 - 2009-04-06 01:46:38 delta t: 2
heart - DEBUG chopped trace: Trace (IU, TEIG, 0, Z) timerange: 2009-04-06 01:45:14 - 2009-04-06 01:46:02 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982039.0 b: 1238982044.0 c: 1238982094.0 d: 1238982109.0
heart - DEBUG trace: Trace (IU, KBL, 0, Z) timerange: 2009-04-06 01:40:28 - 2009-04-06 01:42:08 delta t: 2
heart - DEBUG chopped trace: Trace (IU, KBL, 0, Z) timerange: 2009-04-06 01:40:44 - 2009-04-06 01:41:32 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982163.0 b: 1238982168.0 c: 1238982218.0 d: 1238982233.0
heart - DEBUG trace: Trace (II, MSEY, 0, Z) timerange: 2009-04-06 01:42:32 - 2009-04-06 01:44:12 delta t: 2
heart - DEBUG chopped trace: Trace (II, MSEY, 0, Z) timerange: 2009-04-06 01:42:48 - 2009-04-06 01:43:36 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982001.0 b: 1238982006.0 c: 1238982056.0 d: 1238982071.0
heart - DEBUG trace: Trace (GT, DBIC, 0, Z) timerange: 2009-04-06 01:39:50 - 2009-04-06 01:41:30 delta t: 2
heart - DEBUG chopped trace: Trace (GT, DBIC, 0, Z) timerange: 2009-04-06 01:40:06 - 2009-04-06 01:40:54 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982215.0 b: 1238982220.0 c: 1238982270.0 d: 1238982285.0
heart - DEBUG trace: Trace (IC, HIA, 0, Z) timerange: 2009-04-06 01:43:24 - 2009-04-06 01:45:04 delta t: 2
heart - DEBUG chopped trace: Trace (IC, HIA, 0, Z) timerange: 2009-04-06 01:43:40 - 2009-04-06 01:44:28 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982187.0 b: 1238982192.0 c: 1238982242.0 d: 1238982257.0
heart - DEBUG trace: Trace (IU, YAK, 0, Z) timerange: 2009-04-06 01:42:56 - 2009-04-06 01:44:36 delta t: 2
heart - DEBUG chopped trace: Trace (IU, YAK, 0, Z) timerange: 2009-04-06 01:43:12 - 2009-04-06 01:44:00 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982295.0 b: 1238982300.0 c: 1238982350.0 d: 1238982365.0
heart - DEBUG trace: Trace (IC, QIZ, 0, Z) timerange: 2009-04-06 01:44:44 - 2009-04-06 01:46:24 delta t: 2
heart - DEBUG chopped trace: Trace (IC, QIZ, 0, Z) timerange: 2009-04-06 01:45:00 - 2009-04-06 01:45:48 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981981.0 b: 1238981986.0 c: 1238982036.0 d: 1238982051.0
heart - DEBUG trace: Trace (GE, KBS, 0, Z) timerange: 2009-04-06 01:39:30 - 2009-04-06 01:41:10 delta t: 2
heart - DEBUG chopped trace: Trace (GE, KBS, 0, Z) timerange: 2009-04-06 01:39:46 - 2009-04-06 01:40:34 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982153.0 b: 1238982158.0 c: 1238982208.0 d: 1238982223.0
heart - DEBUG trace: Trace (IU, LSZ, 0, Z) timerange: 2009-04-06 01:42:22 - 2009-04-06 01:44:02 delta t: 2
heart - DEBUG chopped trace: Trace (IU, LSZ, 0, Z) timerange: 2009-04-06 01:42:38 - 2009-04-06 01:43:26 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982149.0 b: 1238982154.0 c: 1238982204.0 d: 1238982219.0
heart - DEBUG trace: Trace (II, TLY, 0, Z) timerange: 2009-04-06 01:42:18 - 2009-04-06 01:43:58 delta t: 2
heart - DEBUG chopped trace: Trace (II, TLY, 0, Z) timerange: 2009-04-06 01:42:34 - 2009-04-06 01:43:22 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982283.0 b: 1238982288.0 c: 1238982338.0 d: 1238982353.0
heart - DEBUG trace: Trace (IU, INCN, 0, Z) timerange: 2009-04-06 01:44:32 - 2009-04-06 01:46:12 delta t: 2
heart - DEBUG chopped trace: Trace (IU, INCN, 0, Z) timerange: 2009-04-06 01:44:48 - 2009-04-06 01:45:36 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982255.0 b: 1238982260.0 c: 1238982310.0 d: 1238982325.0
heart - DEBUG trace: Trace (IU, DWPF, 0, Z) timerange: 2009-04-06 01:44:04 - 2009-04-06 01:45:44 delta t: 2
heart - DEBUG chopped trace: Trace (IU, DWPF, 0, Z) timerange: 2009-04-06 01:44:20 - 2009-04-06 01:45:08 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982107.0 b: 1238982112.0 c: 1238982162.0 d: 1238982177.0
heart - DEBUG trace: Trace (IC, WMQ, 0, Z) timerange: 2009-04-06 01:41:36 - 2009-04-06 01:43:16 delta t: 2
heart - DEBUG chopped trace: Trace (IC, WMQ, 0, Z) timerange: 2009-04-06 01:41:52 - 2009-04-06 01:42:40 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982043.0 b: 1238982048.0 c: 1238982098.0 d: 1238982113.0
heart - DEBUG trace: Trace (II, AAK, 0, Z) timerange: 2009-04-06 01:40:32 - 2009-04-06 01:42:12 delta t: 2
heart - DEBUG chopped trace: Trace (II, AAK, 0, Z) timerange: 2009-04-06 01:40:48 - 2009-04-06 01:41:36 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982235.0 b: 1238982240.0 c: 1238982290.0 d: 1238982305.0
heart - DEBUG trace: Trace (IC, XAN, 0, Z) timerange: 2009-04-06 01:43:44 - 2009-04-06 01:45:24 delta t: 2
heart - DEBUG chopped trace: Trace (IC, XAN, 0, Z) timerange: 2009-04-06 01:44:00 - 2009-04-06 01:44:48 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982075.0 b: 1238982080.0 c: 1238982130.0 d: 1238982145.0
heart - DEBUG trace: Trace (IU, KMBO, 0, Z) timerange: 2009-04-06 01:41:04 - 2009-04-06 01:42:44 delta t: 2
heart - DEBUG chopped trace: Trace (IU, KMBO, 0, Z) timerange: 2009-04-06 01:41:20 - 2009-04-06 01:42:08 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982045.0 b: 1238982050.0 c: 1238982100.0 d: 1238982115.0
heart - DEBUG trace: Trace (II, KURK, 0, Z) timerange: 2009-04-06 01:40:34 - 2009-04-06 01:42:14 delta t: 2
heart - DEBUG chopped trace: Trace (II, KURK, 0, Z) timerange: 2009-04-06 01:40:50 - 2009-04-06 01:41:38 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981977.0 b: 1238981982.0 c: 1238982032.0 d: 1238982047.0
heart - DEBUG trace: Trace (ZF, ADYE, 0, Z) timerange: 2009-04-06 01:39:26 - 2009-04-06 01:41:06 delta t: 2
heart - DEBUG chopped trace: Trace (ZF, ADYE, 0, Z) timerange: 2009-04-06 01:39:42 - 2009-04-06 01:40:30 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982281.0 b: 1238982286.0 c: 1238982336.0 d: 1238982351.0
heart - DEBUG trace: Trace (IU, SDV, 0, Z) timerange: 2009-04-06 01:44:30 - 2009-04-06 01:46:10 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SDV, 0, Z) timerange: 2009-04-06 01:44:46 - 2009-04-06 01:45:34 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981941.0 b: 1238981946.0 c: 1238981996.0 d: 1238982011.0
heart - DEBUG trace: Trace (II, ARU, 0, Z) timerange: 2009-04-06 01:38:50 - 2009-04-06 01:40:30 delta t: 2
heart - DEBUG chopped trace: Trace (II, ARU, 0, Z) timerange: 2009-04-06 01:39:06 - 2009-04-06 01:39:54 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982315.0 b: 1238982320.0 c: 1238982370.0 d: 1238982385.0
heart - DEBUG trace: Trace (IU, SAML, 0, Z) timerange: 2009-04-06 01:45:04 - 2009-04-06 01:46:44 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SAML, 0, Z) timerange: 2009-04-06 01:45:20 - 2009-04-06 01:46:08 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982061.0 b: 1238982066.0 c: 1238982116.0 d: 1238982131.0
heart - DEBUG trace: Trace (II, ALE, 0, Z) timerange: 2009-04-06 01:40:50 - 2009-04-06 01:42:30 delta t: 2
heart - DEBUG chopped trace: Trace (II, ALE, 0, Z) timerange: 2009-04-06 01:41:06 - 2009-04-06 01:41:54 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981959.0 b: 1238981964.0 c: 1238982014.0 d: 1238982029.0
heart - DEBUG trace: Trace (II, ABKT, 0, Z) timerange: 2009-04-06 01:39:08 - 2009-04-06 01:40:48 delta t: 2
heart - DEBUG chopped trace: Trace (II, ABKT, 0, Z) timerange: 2009-04-06 01:39:24 - 2009-04-06 01:40:12 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981935.0 b: 1238981940.0 c: 1238981990.0 d: 1238982005.0
heart - DEBUG trace: Trace (KZ, AKTO, 0, Z) timerange: 2009-04-06 01:38:44 - 2009-04-06 01:40:24 delta t: 2
heart - DEBUG chopped trace: Trace (KZ, AKTO, 0, Z) timerange: 2009-04-06 01:39:00 - 2009-04-06 01:39:48 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982239.0 b: 1238982244.0 c: 1238982294.0 d: 1238982309.0
heart - DEBUG trace: Trace (IC, BJT, 0, Z) timerange: 2009-04-06 01:43:48 - 2009-04-06 01:45:28 delta t: 2
heart - DEBUG chopped trace: Trace (IC, BJT, 0, Z) timerange: 2009-04-06 01:44:04 - 2009-04-06 01:44:52 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982291.0 b: 1238982296.0 c: 1238982346.0 d: 1238982361.0
heart - DEBUG trace: Trace (IC, SSE, 0, Z) timerange: 2009-04-06 01:44:40 - 2009-04-06 01:46:20 delta t: 2
heart - DEBUG chopped trace: Trace (IC, SSE, 0, Z) timerange: 2009-04-06 01:44:56 - 2009-04-06 01:45:44 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982251.0 b: 1238982256.0 c: 1238982306.0 d: 1238982321.0
heart - DEBUG trace: Trace (IU, CHTO, 0, Z) timerange: 2009-04-06 01:44:00 - 2009-04-06 01:45:40 delta t: 2
heart - DEBUG chopped trace: Trace (IU, CHTO, 0, Z) timerange: 2009-04-06 01:44:16 - 2009-04-06 01:45:04 delta t: 2
heart - DEBUG Post-process time 0.023978 heart - DEBUG Stack traces time 0.000000 heart - DEBUG Returning... seismic - DEBUG Teleseismic forward model on test model takes: 0.827361 models - INFO Model building was successful!
models - INFO Using "bin" backend to store samples! models - INFO ... Initiate Metropolis for Parallel Tempering... proposal_distribution: Normal, tune_interval=10, n_chains=8
models.base - INFO ... Starting Parallel Tempering ...
distributed - INFO MPI is installed! distributed - INFO Detected python3 distributed - INFO Done initialising mpi runner distributed - INFO sampler directory: /home/lab/bayesiano/beat/beat/sampler/pt.py distributed - DEBUG Running mpi with arguments: -n 9 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc Traceback (most recent call last): File "/home/lab/anaconda3/envs/beat/bin/beat", line 11, in
load_entry_point('beat==1.0', 'console_scripts', 'beat')()
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x8664.egg/beat/apps/beat.py", line 1686, in main
globals()['command' + command](args)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 775, in command_sample
sample(step, problem)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/base.py", line 196, in sample
record_worker_chains=pa.record_worker_chains)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/pt.py", line 836, in pt_sample
project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 226, in run_mpi_sampler
samplerdir, n_jobs=n_jobs, loglevel=loglevel, project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 158, in run
in the directory %s'''.lstrip() % (program, self.tempdir))
beat.sampler.distributed.MPIError: mpiexec has been invoked as "mpiexec"
in the directory /tmp/mpiexec-7bw3hmzo
distributed - WARNING not removing temporary directory: /tmp/mpiexec-7bw3hmzo
------------------------------------------------------------*-
3) Running:
mpiexec -n 9 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
in the temporary directory: /tmp/mpiexec-7bw3hmzo
I got:
(beat) lab@lab:/tmp/mpiexec-7bw3hmzo$ mpiexec -n 9 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
There are not enough slots available in the system to satisfy the 9 slots that were requested by the application:
python3
Either request fewer slots for your application, or make more slots available for use.
A "slot" is the Open MPI term for an allocatable unit where we can launch a process. The number of slots available are defined by the environment in which Open MPI processes are run:
In all the above cases, if you want Open MPI to default to the number of hardware threads instead of the number of processor cores, use the --use-hwthread-cpus option.
Alternatively, you can use the --oversubscribe option to ignore the number of available slots when deciding the number of processes to launch.
------------------------------------------------------------*-
4) running the line:
mpiexec -n 9 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc --use-hwthread-cpus
Got:
(beat) lab@lab:/tmp/mpiexec-7bw3hmzo$ mpiexec -n 9 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py --use-hwthread-cpus
There are not enough slots available in the system to satisfy the 9 slots that were requested by the application:
python3
Either request fewer slots for your application, or make more slots available for use.
A "slot" is the Open MPI term for an allocatable unit where we can launch a process. The number of slots available are defined by the environment in which Open MPI processes are run:
In all the above cases, if you want Open MPI to default to the number of hardware threads instead of the number of processor cores, use the --use-hwthread-cpus option.
Alternatively, you can use the --oversubscribe option to ignore the number of available slots when deciding the number of processes to launch.
------------------------------------------------------------*-
5) Now, Running the second example with n_chains=5, I got:
(beat) lab@lab:~$ beat sample Laquila_dc --loglevel=debug config - INFO All hierarchicals ok! config - INFO All hyperparameters ok! config - INFO All priors ok! models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ... models - INFO ---------------------
seismic - DEBUG Setting up seismic structure ...
utility - DEBUG Valid distance range: [30.000000, 90.000000]! utility - DEBUG Distance of station TSUM: 61.349855 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station RCBR: 65.567308 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station RES: 54.143590 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station KMI: 73.109289 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station SSPA: 64.946397 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station BTDF: 89.437223 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station ANWB: 67.439618 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station MBO: 38.167185 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station TEIG: 84.816714 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station KBL: 43.633450 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station MSEY: 60.417579 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station DBIC: 38.986606 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station HIA: 68.244425 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station YAK: 64.054778 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station QIZ: 82.062195 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station KBS: 36.698309 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station LSZ: 58.931762 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station TLY: 58.456908 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station INCN: 79.971426 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station DWPF: 74.925103 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station WMQ: 52.564826 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station AAK: 44.192670 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station XAN: 71.681441 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station KMBO: 48.335702 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station KURK: 44.334011 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station ADYE: 36.168619 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station SDV: 79.664513 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station ARU: 31.995457 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station SAML: 86.199701 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station ALE: 46.405043 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station OTAV: 91.209244 [deg] utility - DEBUG Outside defined distance range! utility - DEBUG Distance of station ABKT: 34.203018 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station AKTO: 31.451507 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station BJT: 72.303754 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station SSE: 81.349799 [deg] utility - DEBUG Inside defined distance range! utility - DEBUG Distance of station CHTO: 74.397519 [deg] utility - DEBUG Inside defined distance range! heart - INFO Consistent number of datasets and targets in any_P_0 wavemap! heart - INFO Number of seismic datasets for wavemap: any_P_0: 35 models - INFO ... Building model ...
models - DEBUG Optimization for 1 sources models - INFO Optimization for 35 hyperparameters in total! seismic - INFO Seismic optimization on: depth, dip, duration, east_shift, magnitude, north_shift, rake, strike, time seismic - INFO Retrieving seismic data-covariances with structure "variance" for any_P_0 ... heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 61.349855 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 65.567308 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 54.143590 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 73.109289 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 64.946397 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 89.437223 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 67.439618 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 38.167185 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 84.816714 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 43.633450 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 60.417579 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 38.986606 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 68.244425 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 64.054778 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 82.062195 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 36.698309 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 58.931762 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 58.456908 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 79.971426 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 74.925103 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 52.564826 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 44.192670 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 71.681441 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 48.335702 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 44.334011 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 36.168619 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 79.664513 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 31.995457 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 86.199701 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 46.405043 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 34.203018 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 31.451507 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 72.303754 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 81.349799 [deg] heart - DEBUG Wavename not specified using first tabulated phase! any_P heart - DEBUG Arrival time for wavename "any_P" distance 74.397519 [deg] seismic - INFO Preparing data of "any_P_0" for optimization heart - DEBUG Arrival time for wavename "any_P" distance 61.349855 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 65.567308 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 54.143590 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 73.109289 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 64.946397 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 89.437223 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 67.439618 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 38.167185 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 84.816714 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 43.633450 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 60.417579 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 38.986606 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 68.244425 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 64.054778 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 82.062195 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 36.698309 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 58.931762 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 58.456908 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 79.971426 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 74.925103 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 52.564826 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 44.192670 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 71.681441 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 48.335702 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 44.334011 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 36.168619 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 79.664513 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 31.995457 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 86.199701 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 46.405043 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 34.203018 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 31.451507 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 72.303754 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 81.349799 [deg] heart - DEBUG Arrival time for wavename "any_P" distance 74.397519 [deg] heart - DEBUG Pre-processing data ... heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1641 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982169.0 b: 1238982174.0 c: 1238982224.0 d: 1238982239.0
heart - DEBUG trace: Trace (IU, TSUM, filt, Z) timerange: 2009-04-06 01:32:44 - 2009-04-06 02:27:24 delta t: 2
heart - DEBUG chopped trace: Trace (IU, TSUM, filt, Z) timerange: 2009-04-06 01:42:54 - 2009-04-06 01:43:42 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1716 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982197.0 b: 1238982202.0 c: 1238982252.0 d: 1238982267.0
heart - DEBUG trace: Trace (IU, RCBR, filt, Z) timerange: 2009-04-06 01:33:12 - 2009-04-06 02:30:22 delta t: 2
heart - DEBUG chopped trace: Trace (IU, RCBR, filt, Z) timerange: 2009-04-06 01:43:22 - 2009-04-06 01:44:10 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1508 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982119.0 b: 1238982124.0 c: 1238982174.0 d: 1238982189.0
heart - DEBUG trace: Trace (CN, RES, filt, Z) timerange: 2009-04-06 01:31:52 - 2009-04-06 02:22:06 delta t: 2
heart - DEBUG chopped trace: Trace (CN, RES, filt, Z) timerange: 2009-04-06 01:42:04 - 2009-04-06 01:42:52 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1853 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982245.0 b: 1238982250.0 c: 1238982300.0 d: 1238982315.0
heart - DEBUG trace: Trace (IC, KMI, filt, Z) timerange: 2009-04-06 01:34:00 - 2009-04-06 02:35:44 delta t: 2
heart - DEBUG chopped trace: Trace (IC, KMI, filt, Z) timerange: 2009-04-06 01:44:10 - 2009-04-06 01:44:58 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1702 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982193.0 b: 1238982198.0 c: 1238982248.0 d: 1238982263.0
heart - DEBUG trace: Trace (IU, SSPA, filt, Z) timerange: 2009-04-06 01:33:08 - 2009-04-06 02:29:50 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SSPA, filt, Z) timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:06 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2169 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982331.0 b: 1238982336.0 c: 1238982386.0 d: 1238982401.0
heart - DEBUG trace: Trace (MS, BTDF, filt, Z) timerange: 2009-04-06 01:35:26 - 2009-04-06 02:47:42 delta t: 2
heart - DEBUG chopped trace: Trace (MS, BTDF, filt, Z) timerange: 2009-04-06 01:45:36 - 2009-04-06 01:46:24 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1748 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982209.0 b: 1238982214.0 c: 1238982264.0 d: 1238982279.0
heart - DEBUG trace: Trace (CU, ANWB, filt, Z) timerange: 2009-04-06 01:33:24 - 2009-04-06 02:31:38 delta t: 2
heart - DEBUG chopped trace: Trace (CU, ANWB, filt, Z) timerange: 2009-04-06 01:43:34 - 2009-04-06 01:44:22 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1229 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981993.0 b: 1238981998.0 c: 1238982048.0 d: 1238982063.0
heart - DEBUG trace: Trace (G, MBO, filt, Z) timerange: 2009-04-06 01:29:48 - 2009-04-06 02:10:44 delta t: 2
heart - DEBUG chopped trace: Trace (G, MBO, filt, Z) timerange: 2009-04-06 01:39:58 - 2009-04-06 01:40:46 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2078 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982309.0 b: 1238982314.0 c: 1238982364.0 d: 1238982379.0
heart - DEBUG trace: Trace (IU, TEIG, filt, Z) timerange: 2009-04-06 01:35:02 - 2009-04-06 02:44:16 delta t: 2
heart - DEBUG chopped trace: Trace (IU, TEIG, filt, Z) timerange: 2009-04-06 01:45:14 - 2009-04-06 01:46:02 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1324 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982039.0 b: 1238982044.0 c: 1238982094.0 d: 1238982109.0
heart - DEBUG trace: Trace (IU, KBL, filt, Z) timerange: 2009-04-06 01:30:32 - 2009-04-06 02:14:38 delta t: 2
heart - DEBUG chopped trace: Trace (IU, KBL, filt, Z) timerange: 2009-04-06 01:40:44 - 2009-04-06 01:41:32 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1624 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982163.0 b: 1238982168.0 c: 1238982218.0 d: 1238982233.0
heart - DEBUG trace: Trace (II, MSEY, filt, Z) timerange: 2009-04-06 01:32:38 - 2009-04-06 02:26:44 delta t: 2
heart - DEBUG chopped trace: Trace (II, MSEY, filt, Z) timerange: 2009-04-06 01:42:48 - 2009-04-06 01:43:36 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1244 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982001.0 b: 1238982006.0 c: 1238982056.0 d: 1238982071.0
heart - DEBUG trace: Trace (GT, DBIC, filt, Z) timerange: 2009-04-06 01:29:54 - 2009-04-06 02:11:20 delta t: 2
heart - DEBUG chopped trace: Trace (GT, DBIC, filt, Z) timerange: 2009-04-06 01:40:06 - 2009-04-06 01:40:54 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1763 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982215.0 b: 1238982220.0 c: 1238982270.0 d: 1238982285.0
heart - DEBUG trace: Trace (IC, HIA, filt, Z) timerange: 2009-04-06 01:33:30 - 2009-04-06 02:32:14 delta t: 2
heart - DEBUG chopped trace: Trace (IC, HIA, filt, Z) timerange: 2009-04-06 01:43:40 - 2009-04-06 01:44:28 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1686 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982187.0 b: 1238982192.0 c: 1238982242.0 d: 1238982257.0
heart - DEBUG trace: Trace (IU, YAK, filt, Z) timerange: 2009-04-06 01:33:02 - 2009-04-06 02:29:12 delta t: 2
heart - DEBUG chopped trace: Trace (IU, YAK, filt, Z) timerange: 2009-04-06 01:43:12 - 2009-04-06 01:44:00 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2026 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982295.0 b: 1238982300.0 c: 1238982350.0 d: 1238982365.0
heart - DEBUG trace: Trace (IC, QIZ, filt, Z) timerange: 2009-04-06 01:34:48 - 2009-04-06 02:42:18 delta t: 2
heart - DEBUG chopped trace: Trace (IC, QIZ, filt, Z) timerange: 2009-04-06 01:45:00 - 2009-04-06 01:45:48 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1201 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981981.0 b: 1238981986.0 c: 1238982036.0 d: 1238982051.0
heart - DEBUG trace: Trace (GE, KBS, filt, Z) timerange: 2009-04-06 01:29:34 - 2009-04-06 02:09:34 delta t: 2
heart - DEBUG chopped trace: Trace (GE, KBS, filt, Z) timerange: 2009-04-06 01:39:46 - 2009-04-06 01:40:34 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1598 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982153.0 b: 1238982158.0 c: 1238982208.0 d: 1238982223.0
heart - DEBUG trace: Trace (IU, LSZ, filt, Z) timerange: 2009-04-06 01:32:28 - 2009-04-06 02:25:42 delta t: 2
heart - DEBUG chopped trace: Trace (IU, LSZ, filt, Z) timerange: 2009-04-06 01:42:38 - 2009-04-06 01:43:26 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1586 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982149.0 b: 1238982154.0 c: 1238982204.0 d: 1238982219.0
heart - DEBUG trace: Trace (II, TLY, filt, Z) timerange: 2009-04-06 01:32:24 - 2009-04-06 02:25:14 delta t: 2
heart - DEBUG chopped trace: Trace (II, TLY, filt, Z) timerange: 2009-04-06 01:42:34 - 2009-04-06 01:43:22 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1984 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982283.0 b: 1238982288.0 c: 1238982338.0 d: 1238982353.0
heart - DEBUG trace: Trace (IU, INCN, filt, Z) timerange: 2009-04-06 01:34:38 - 2009-04-06 02:40:44 delta t: 2
heart - DEBUG chopped trace: Trace (IU, INCN, filt, Z) timerange: 2009-04-06 01:44:48 - 2009-04-06 01:45:36 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1888 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982255.0 b: 1238982260.0 c: 1238982310.0 d: 1238982325.0
heart - DEBUG trace: Trace (IU, DWPF, filt, Z) timerange: 2009-04-06 01:34:08 - 2009-04-06 02:37:02 delta t: 2
heart - DEBUG chopped trace: Trace (IU, DWPF, filt, Z) timerange: 2009-04-06 01:44:20 - 2009-04-06 01:45:08 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1481 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982107.0 b: 1238982112.0 c: 1238982162.0 d: 1238982177.0
heart - DEBUG trace: Trace (IC, WMQ, filt, Z) timerange: 2009-04-06 01:31:42 - 2009-04-06 02:21:02 delta t: 2
heart - DEBUG chopped trace: Trace (IC, WMQ, filt, Z) timerange: 2009-04-06 01:41:52 - 2009-04-06 01:42:40 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1334 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982043.0 b: 1238982048.0 c: 1238982098.0 d: 1238982113.0
heart - DEBUG trace: Trace (II, AAK, filt, Z) timerange: 2009-04-06 01:30:36 - 2009-04-06 02:15:02 delta t: 2
heart - DEBUG chopped trace: Trace (II, AAK, filt, Z) timerange: 2009-04-06 01:40:48 - 2009-04-06 01:41:36 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1827 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982235.0 b: 1238982240.0 c: 1238982290.0 d: 1238982305.0
heart - DEBUG trace: Trace (IC, XAN, filt, Z) timerange: 2009-04-06 01:33:50 - 2009-04-06 02:34:42 delta t: 2
heart - DEBUG chopped trace: Trace (IC, XAN, filt, Z) timerange: 2009-04-06 01:44:00 - 2009-04-06 01:44:48 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1408 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982075.0 b: 1238982080.0 c: 1238982130.0 d: 1238982145.0
heart - DEBUG trace: Trace (IU, KMBO, filt, Z) timerange: 2009-04-06 01:31:10 - 2009-04-06 02:18:04 delta t: 2
heart - DEBUG chopped trace: Trace (IU, KMBO, filt, Z) timerange: 2009-04-06 01:41:20 - 2009-04-06 01:42:08 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1335 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982045.0 b: 1238982050.0 c: 1238982100.0 d: 1238982115.0
heart - DEBUG trace: Trace (II, KURK, filt, Z) timerange: 2009-04-06 01:30:38 - 2009-04-06 02:15:06 delta t: 2
heart - DEBUG chopped trace: Trace (II, KURK, filt, Z) timerange: 2009-04-06 01:40:50 - 2009-04-06 01:41:38 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1195 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981977.0 b: 1238981982.0 c: 1238982032.0 d: 1238982047.0
heart - DEBUG trace: Trace (ZF, ADYE, filt, Z) timerange: 2009-04-06 01:29:32 - 2009-04-06 02:09:20 delta t: 2
heart - DEBUG chopped trace: Trace (ZF, ADYE, filt, Z) timerange: 2009-04-06 01:39:42 - 2009-04-06 01:40:30 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1978 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982281.0 b: 1238982286.0 c: 1238982336.0 d: 1238982351.0
heart - DEBUG trace: Trace (IU, SDV, filt, Z) timerange: 2009-04-06 01:34:36 - 2009-04-06 02:40:30 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SDV, filt, Z) timerange: 2009-04-06 01:44:46 - 2009-04-06 01:45:34 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1120 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981941.0 b: 1238981946.0 c: 1238981996.0 d: 1238982011.0
heart - DEBUG trace: Trace (II, ARU, filt, Z) timerange: 2009-04-06 01:28:54 - 2009-04-06 02:06:12 delta t: 2
heart - DEBUG chopped trace: Trace (II, ARU, filt, Z) timerange: 2009-04-06 01:39:06 - 2009-04-06 01:39:54 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2105 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982315.0 b: 1238982320.0 c: 1238982370.0 d: 1238982385.0
heart - DEBUG trace: Trace (IU, SAML, filt, Z) timerange: 2009-04-06 01:35:10 - 2009-04-06 02:45:18 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SAML, filt, Z) timerange: 2009-04-06 01:45:20 - 2009-04-06 01:46:08 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1370 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982061.0 b: 1238982066.0 c: 1238982116.0 d: 1238982131.0
heart - DEBUG trace: Trace (II, ALE, filt, Z) timerange: 2009-04-06 01:30:54 - 2009-04-06 02:16:32 delta t: 2
heart - DEBUG chopped trace: Trace (II, ALE, filt, Z) timerange: 2009-04-06 01:41:06 - 2009-04-06 01:41:54 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1159 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981959.0 b: 1238981964.0 c: 1238982014.0 d: 1238982029.0
heart - DEBUG trace: Trace (II, ABKT, filt, Z) timerange: 2009-04-06 01:29:14 - 2009-04-06 02:07:50 delta t: 2
heart - DEBUG chopped trace: Trace (II, ABKT, filt, Z) timerange: 2009-04-06 01:39:24 - 2009-04-06 01:40:12 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1110 heart - DEBUG taper times: --- !pf.CosTaper a: 1238981935.0 b: 1238981940.0 c: 1238981990.0 d: 1238982005.0
heart - DEBUG trace: Trace (KZ, AKTO, filt, Z) timerange: 2009-04-06 01:28:50 - 2009-04-06 02:05:48 delta t: 2
heart - DEBUG chopped trace: Trace (KZ, AKTO, filt, Z) timerange: 2009-04-06 01:39:00 - 2009-04-06 01:39:48 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1838 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982239.0 b: 1238982244.0 c: 1238982294.0 d: 1238982309.0
heart - DEBUG trace: Trace (IC, BJT, filt, Z) timerange: 2009-04-06 01:33:54 - 2009-04-06 02:35:08 delta t: 2
heart - DEBUG chopped trace: Trace (IC, BJT, filt, Z) timerange: 2009-04-06 01:44:04 - 2009-04-06 01:44:52 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2012 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982291.0 b: 1238982296.0 c: 1238982346.0 d: 1238982361.0
heart - DEBUG trace: Trace (IC, SSE, filt, Z) timerange: 2009-04-06 01:34:44 - 2009-04-06 02:41:46 delta t: 2
heart - DEBUG chopped trace: Trace (IC, SSE, filt, Z) timerange: 2009-04-06 01:44:56 - 2009-04-06 01:45:44 delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1879 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982251.0 b: 1238982256.0 c: 1238982306.0 d: 1238982321.0
heart - DEBUG trace: Trace (IU, CHTO, filt, Z) timerange: 2009-04-06 01:34:06 - 2009-04-06 02:36:42 delta t: 2
heart - DEBUG chopped trace: Trace (IU, CHTO, filt, Z) timerange: 2009-04-06 01:44:16 - 2009-04-06 01:45:04 delta t: 2
heart - DEBUG Returning chopped traces ... seismic - INFO Initializing synthesizer for "any_P_0" heart - DEBUG Synthetics generation time: 0.017995 heart - DEBUG taper times: --- !pf.CosTaper a: 1238982169.0 b: 1238982174.0 c: 1238982224.0 d: 1238982239.0
heart - DEBUG trace: Trace (IU, TSUM, 0, Z) timerange: 2009-04-06 01:42:38 - 2009-04-06 01:44:18 delta t: 2
heart - DEBUG chopped trace: Trace (IU, TSUM, 0, Z) timerange: 2009-04-06 01:42:54 - 2009-04-06 01:43:42 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982197.0 b: 1238982202.0 c: 1238982252.0 d: 1238982267.0
heart - DEBUG trace: Trace (IU, RCBR, 0, Z) timerange: 2009-04-06 01:43:06 - 2009-04-06 01:44:46 delta t: 2
heart - DEBUG chopped trace: Trace (IU, RCBR, 0, Z) timerange: 2009-04-06 01:43:22 - 2009-04-06 01:44:10 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982119.0 b: 1238982124.0 c: 1238982174.0 d: 1238982189.0
heart - DEBUG trace: Trace (CN, RES, 0, Z) timerange: 2009-04-06 01:41:48 - 2009-04-06 01:43:28 delta t: 2
heart - DEBUG chopped trace: Trace (CN, RES, 0, Z) timerange: 2009-04-06 01:42:04 - 2009-04-06 01:42:52 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982245.0 b: 1238982250.0 c: 1238982300.0 d: 1238982315.0
heart - DEBUG trace: Trace (IC, KMI, 0, Z) timerange: 2009-04-06 01:43:54 - 2009-04-06 01:45:34 delta t: 2
heart - DEBUG chopped trace: Trace (IC, KMI, 0, Z) timerange: 2009-04-06 01:44:10 - 2009-04-06 01:44:58 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982193.0 b: 1238982198.0 c: 1238982248.0 d: 1238982263.0
heart - DEBUG trace: Trace (IU, SSPA, 0, Z) timerange: 2009-04-06 01:43:02 - 2009-04-06 01:44:42 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SSPA, 0, Z) timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:06 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982331.0 b: 1238982336.0 c: 1238982386.0 d: 1238982401.0
heart - DEBUG trace: Trace (MS, BTDF, 0, Z) timerange: 2009-04-06 01:45:20 - 2009-04-06 01:47:00 delta t: 2
heart - DEBUG chopped trace: Trace (MS, BTDF, 0, Z) timerange: 2009-04-06 01:45:36 - 2009-04-06 01:46:24 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982209.0 b: 1238982214.0 c: 1238982264.0 d: 1238982279.0
heart - DEBUG trace: Trace (CU, ANWB, 0, Z) timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:58 delta t: 2
heart - DEBUG chopped trace: Trace (CU, ANWB, 0, Z) timerange: 2009-04-06 01:43:34 - 2009-04-06 01:44:22 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981993.0 b: 1238981998.0 c: 1238982048.0 d: 1238982063.0
heart - DEBUG trace: Trace (G, MBO, 0, Z) timerange: 2009-04-06 01:39:42 - 2009-04-06 01:41:22 delta t: 2
heart - DEBUG chopped trace: Trace (G, MBO, 0, Z) timerange: 2009-04-06 01:39:58 - 2009-04-06 01:40:46 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982309.0 b: 1238982314.0 c: 1238982364.0 d: 1238982379.0
heart - DEBUG trace: Trace (IU, TEIG, 0, Z) timerange: 2009-04-06 01:44:58 - 2009-04-06 01:46:38 delta t: 2
heart - DEBUG chopped trace: Trace (IU, TEIG, 0, Z) timerange: 2009-04-06 01:45:14 - 2009-04-06 01:46:02 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982039.0 b: 1238982044.0 c: 1238982094.0 d: 1238982109.0
heart - DEBUG trace: Trace (IU, KBL, 0, Z) timerange: 2009-04-06 01:40:28 - 2009-04-06 01:42:08 delta t: 2
heart - DEBUG chopped trace: Trace (IU, KBL, 0, Z) timerange: 2009-04-06 01:40:44 - 2009-04-06 01:41:32 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982163.0 b: 1238982168.0 c: 1238982218.0 d: 1238982233.0
heart - DEBUG trace: Trace (II, MSEY, 0, Z) timerange: 2009-04-06 01:42:32 - 2009-04-06 01:44:12 delta t: 2
heart - DEBUG chopped trace: Trace (II, MSEY, 0, Z) timerange: 2009-04-06 01:42:48 - 2009-04-06 01:43:36 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982001.0 b: 1238982006.0 c: 1238982056.0 d: 1238982071.0
heart - DEBUG trace: Trace (GT, DBIC, 0, Z) timerange: 2009-04-06 01:39:50 - 2009-04-06 01:41:30 delta t: 2
heart - DEBUG chopped trace: Trace (GT, DBIC, 0, Z) timerange: 2009-04-06 01:40:06 - 2009-04-06 01:40:54 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982215.0 b: 1238982220.0 c: 1238982270.0 d: 1238982285.0
heart - DEBUG trace: Trace (IC, HIA, 0, Z) timerange: 2009-04-06 01:43:24 - 2009-04-06 01:45:04 delta t: 2
heart - DEBUG chopped trace: Trace (IC, HIA, 0, Z) timerange: 2009-04-06 01:43:40 - 2009-04-06 01:44:28 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982187.0 b: 1238982192.0 c: 1238982242.0 d: 1238982257.0
heart - DEBUG trace: Trace (IU, YAK, 0, Z) timerange: 2009-04-06 01:42:56 - 2009-04-06 01:44:36 delta t: 2
heart - DEBUG chopped trace: Trace (IU, YAK, 0, Z) timerange: 2009-04-06 01:43:12 - 2009-04-06 01:44:00 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982295.0 b: 1238982300.0 c: 1238982350.0 d: 1238982365.0
heart - DEBUG trace: Trace (IC, QIZ, 0, Z) timerange: 2009-04-06 01:44:44 - 2009-04-06 01:46:24 delta t: 2
heart - DEBUG chopped trace: Trace (IC, QIZ, 0, Z) timerange: 2009-04-06 01:45:00 - 2009-04-06 01:45:48 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981981.0 b: 1238981986.0 c: 1238982036.0 d: 1238982051.0
heart - DEBUG trace: Trace (GE, KBS, 0, Z) timerange: 2009-04-06 01:39:30 - 2009-04-06 01:41:10 delta t: 2
heart - DEBUG chopped trace: Trace (GE, KBS, 0, Z) timerange: 2009-04-06 01:39:46 - 2009-04-06 01:40:34 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982153.0 b: 1238982158.0 c: 1238982208.0 d: 1238982223.0
heart - DEBUG trace: Trace (IU, LSZ, 0, Z) timerange: 2009-04-06 01:42:22 - 2009-04-06 01:44:02 delta t: 2
heart - DEBUG chopped trace: Trace (IU, LSZ, 0, Z) timerange: 2009-04-06 01:42:38 - 2009-04-06 01:43:26 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982149.0 b: 1238982154.0 c: 1238982204.0 d: 1238982219.0
heart - DEBUG trace: Trace (II, TLY, 0, Z) timerange: 2009-04-06 01:42:18 - 2009-04-06 01:43:58 delta t: 2
heart - DEBUG chopped trace: Trace (II, TLY, 0, Z) timerange: 2009-04-06 01:42:34 - 2009-04-06 01:43:22 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982283.0 b: 1238982288.0 c: 1238982338.0 d: 1238982353.0
heart - DEBUG trace: Trace (IU, INCN, 0, Z) timerange: 2009-04-06 01:44:32 - 2009-04-06 01:46:12 delta t: 2
heart - DEBUG chopped trace: Trace (IU, INCN, 0, Z) timerange: 2009-04-06 01:44:48 - 2009-04-06 01:45:36 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982255.0 b: 1238982260.0 c: 1238982310.0 d: 1238982325.0
heart - DEBUG trace: Trace (IU, DWPF, 0, Z) timerange: 2009-04-06 01:44:04 - 2009-04-06 01:45:44 delta t: 2
heart - DEBUG chopped trace: Trace (IU, DWPF, 0, Z) timerange: 2009-04-06 01:44:20 - 2009-04-06 01:45:08 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982107.0 b: 1238982112.0 c: 1238982162.0 d: 1238982177.0
heart - DEBUG trace: Trace (IC, WMQ, 0, Z) timerange: 2009-04-06 01:41:36 - 2009-04-06 01:43:16 delta t: 2
heart - DEBUG chopped trace: Trace (IC, WMQ, 0, Z) timerange: 2009-04-06 01:41:52 - 2009-04-06 01:42:40 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982043.0 b: 1238982048.0 c: 1238982098.0 d: 1238982113.0
heart - DEBUG trace: Trace (II, AAK, 0, Z) timerange: 2009-04-06 01:40:32 - 2009-04-06 01:42:12 delta t: 2
heart - DEBUG chopped trace: Trace (II, AAK, 0, Z) timerange: 2009-04-06 01:40:48 - 2009-04-06 01:41:36 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982235.0 b: 1238982240.0 c: 1238982290.0 d: 1238982305.0
heart - DEBUG trace: Trace (IC, XAN, 0, Z) timerange: 2009-04-06 01:43:44 - 2009-04-06 01:45:24 delta t: 2
heart - DEBUG chopped trace: Trace (IC, XAN, 0, Z) timerange: 2009-04-06 01:44:00 - 2009-04-06 01:44:48 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982075.0 b: 1238982080.0 c: 1238982130.0 d: 1238982145.0
heart - DEBUG trace: Trace (IU, KMBO, 0, Z) timerange: 2009-04-06 01:41:04 - 2009-04-06 01:42:44 delta t: 2
heart - DEBUG chopped trace: Trace (IU, KMBO, 0, Z) timerange: 2009-04-06 01:41:20 - 2009-04-06 01:42:08 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982045.0 b: 1238982050.0 c: 1238982100.0 d: 1238982115.0
heart - DEBUG trace: Trace (II, KURK, 0, Z) timerange: 2009-04-06 01:40:34 - 2009-04-06 01:42:14 delta t: 2
heart - DEBUG chopped trace: Trace (II, KURK, 0, Z) timerange: 2009-04-06 01:40:50 - 2009-04-06 01:41:38 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981977.0 b: 1238981982.0 c: 1238982032.0 d: 1238982047.0
heart - DEBUG trace: Trace (ZF, ADYE, 0, Z) timerange: 2009-04-06 01:39:26 - 2009-04-06 01:41:06 delta t: 2
heart - DEBUG chopped trace: Trace (ZF, ADYE, 0, Z) timerange: 2009-04-06 01:39:42 - 2009-04-06 01:40:30 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982281.0 b: 1238982286.0 c: 1238982336.0 d: 1238982351.0
heart - DEBUG trace: Trace (IU, SDV, 0, Z) timerange: 2009-04-06 01:44:30 - 2009-04-06 01:46:10 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SDV, 0, Z) timerange: 2009-04-06 01:44:46 - 2009-04-06 01:45:34 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981941.0 b: 1238981946.0 c: 1238981996.0 d: 1238982011.0
heart - DEBUG trace: Trace (II, ARU, 0, Z) timerange: 2009-04-06 01:38:50 - 2009-04-06 01:40:30 delta t: 2
heart - DEBUG chopped trace: Trace (II, ARU, 0, Z) timerange: 2009-04-06 01:39:06 - 2009-04-06 01:39:54 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982315.0 b: 1238982320.0 c: 1238982370.0 d: 1238982385.0
heart - DEBUG trace: Trace (IU, SAML, 0, Z) timerange: 2009-04-06 01:45:04 - 2009-04-06 01:46:44 delta t: 2
heart - DEBUG chopped trace: Trace (IU, SAML, 0, Z) timerange: 2009-04-06 01:45:20 - 2009-04-06 01:46:08 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982061.0 b: 1238982066.0 c: 1238982116.0 d: 1238982131.0
heart - DEBUG trace: Trace (II, ALE, 0, Z) timerange: 2009-04-06 01:40:50 - 2009-04-06 01:42:30 delta t: 2
heart - DEBUG chopped trace: Trace (II, ALE, 0, Z) timerange: 2009-04-06 01:41:06 - 2009-04-06 01:41:54 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981959.0 b: 1238981964.0 c: 1238982014.0 d: 1238982029.0
heart - DEBUG trace: Trace (II, ABKT, 0, Z) timerange: 2009-04-06 01:39:08 - 2009-04-06 01:40:48 delta t: 2
heart - DEBUG chopped trace: Trace (II, ABKT, 0, Z) timerange: 2009-04-06 01:39:24 - 2009-04-06 01:40:12 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238981935.0 b: 1238981940.0 c: 1238981990.0 d: 1238982005.0
heart - DEBUG trace: Trace (KZ, AKTO, 0, Z) timerange: 2009-04-06 01:38:44 - 2009-04-06 01:40:24 delta t: 2
heart - DEBUG chopped trace: Trace (KZ, AKTO, 0, Z) timerange: 2009-04-06 01:39:00 - 2009-04-06 01:39:48 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982239.0 b: 1238982244.0 c: 1238982294.0 d: 1238982309.0
heart - DEBUG trace: Trace (IC, BJT, 0, Z) timerange: 2009-04-06 01:43:48 - 2009-04-06 01:45:28 delta t: 2
heart - DEBUG chopped trace: Trace (IC, BJT, 0, Z) timerange: 2009-04-06 01:44:04 - 2009-04-06 01:44:52 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982291.0 b: 1238982296.0 c: 1238982346.0 d: 1238982361.0
heart - DEBUG trace: Trace (IC, SSE, 0, Z) timerange: 2009-04-06 01:44:40 - 2009-04-06 01:46:20 delta t: 2
heart - DEBUG chopped trace: Trace (IC, SSE, 0, Z) timerange: 2009-04-06 01:44:56 - 2009-04-06 01:45:44 delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper a: 1238982251.0 b: 1238982256.0 c: 1238982306.0 d: 1238982321.0
heart - DEBUG trace: Trace (IU, CHTO, 0, Z) timerange: 2009-04-06 01:44:00 - 2009-04-06 01:45:40 delta t: 2
heart - DEBUG chopped trace: Trace (IU, CHTO, 0, Z) timerange: 2009-04-06 01:44:16 - 2009-04-06 01:45:04 delta t: 2
heart - DEBUG Post-process time 0.023910 heart - DEBUG Stack traces time 0.000000 heart - DEBUG Returning... seismic - DEBUG Teleseismic forward model on test model takes: 0.840477 models - INFO Model building was successful!
models - INFO Using "bin" backend to store samples! models - INFO ... Initiate Metropolis for Parallel Tempering... proposal_distribution: Normal, tune_interval=10, n_chains=5
models.base - INFO ... Starting Parallel Tempering ...
distributed - INFO MPI is installed! distributed - INFO Detected python3 distributed - INFO Done initialising mpi runner distributed - INFO sampler directory: /home/lab/bayesiano/beat/beat/sampler/pt.py distributed - DEBUG Running mpi with arguments: -n 6 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc Traceback (most recent call last): File "/home/lab/anaconda3/envs/beat/bin/beat", line 11, in
load_entry_point('beat==1.0', 'console_scripts', 'beat')()
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x8664.egg/beat/apps/beat.py", line 1686, in main
globals()['command' + command](args)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 775, in command_sample
sample(step, problem)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/base.py", line 196, in sample
record_worker_chains=pa.record_worker_chains)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/pt.py", line 836, in pt_sample
project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 226, in run_mpi_sampler
samplerdir, n_jobs=n_jobs, loglevel=loglevel, project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 158, in run
in the directory %s'''.lstrip() % (program, self.tempdir))
beat.sampler.distributed.MPIError: mpiexec has been invoked as "mpiexec"
in the directory /tmp/mpiexec-173lp6ia
distributed - WARNING not removing temporary directory: /tmp/mpiexec-173lp6ia
------------------------------------------------------------*-
6) Now running the line:
mpiexec -n 6 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
in the temporary directory: /tmp/mpiexec-173lp6ia
I did not get any error and mpiexec started to execute the process.
------------------------------------------------------------*-
NOTE: I am working in a CPU with 6 cores and 12 threads, the mpi version is 4.0.3
Thanks