hvasbath
commented
5 years ago Hi again! Thanks for reporting and not giving up hope ;) ! Your processes timed out and got killed by the process manager. Do you follow exactly the Tutorial-or did you change some values? May I ask for a list of your system hardware? The forward calculation is very slow in your case- 19s, it should rather be around 1-2s . You may also try turning off processbars-please set the processbar flag to false in the SMC config. Also please try running: THEANO_FLAGS=optimizer=fast_compile beat sample Laquila (instead of beat sample Laquila) will update the FAQ with this respect.
vic-chou
nanfang0422
Hi, when I run the command "beat sample Laquila", I meet a error as follows beat sample Laquila config - INFO All parameter-priors ok! config - INFO All hyper-parameters ok! config - INFO All hierarchical-parameters ok! models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ... models - INFO ---------------------
/usr/local/python35/lib/python3.5/site-packages/scipy/signal/signaltools.py:1341: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use
arr[tuple(seq)]instead ofarr[seq]. In the future this will be interpreted as an array index,arr[np.array(seq)], which will result either in an error or a different result. out_full[ind] += zi /usr/local/python35/lib/python3.5/site-packages/scipy/signal/signaltools.py:1344: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; usearr[tuple(seq)]instead ofarr[seq]. In the future this will be interpreted as an array index,arr[np.array(seq)], which will result either in an error or a different result. out = out_full[ind] /usr/local/python35/lib/python3.5/site-packages/scipy/signal/signaltools.py:1350: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; usearr[tuple(seq)]instead ofarr[seq]. In the future this will be interpreted as an array index,arr[np.array(seq)], which will result either in an error or a different result. zf = out_full[ind] heart - INFO Consistent number of datasets and targets in any_P_0 wavemap! heart - INFO Number of seismic datasets for wavemap: any_P_0: 35 geodetic - INFO Number of geodetic datasets: 2 geodetic - INFO Number of geodetic data points: 419 geodetic - WARNING Covariance estimation not implemented (yet)! Using imported covariances! geodetic - INFO Fit residual ramp selected! models - INFO ... Building model ...models - INFO Optimization for 3 hyperparameters in total! seismic - INFO Seismic optimization on: north_shift, nucleation_y, east_shift, width, nucleation_x, strike, time, slip, length, dip, rake, depth seismic - INFO Retrieving seismic data-covariances with structure "non-toeplitz" for any_P_0 ... seismic - INFO Preparing data of "any_P_0" for optimization seismic - INFO Initializing synthesizer for "any_P_0" geodetic - INFO Geodetic optimization on: north_shift, east_shift, width, strike, slip, length, dip, rake, depth geodetic - INFO Estimating ramp for each dataset... geodetic - INFO Initialized 4 hierarchical parameters (ramps). models - INFO Model building was successful!
models - INFO ... Initiate Sequential Monte Carlo ... n_chains=1000, tune_interval=10, n_jobs=4, proposal_distribution: MultivariateCauchy,
models - INFO Compilation time: 1598.274327 models.base - INFO ... Starting SMC ...
backend - INFO Reloading existing results ... backend - INFO Loading multitrace from /test_beat/Laquila/geometry/stage_0 backend - INFO Checking for corrupted files ... smc - INFO Sample initial stage: ... smc - INFO Beta: 0.000000 Stage: 0 backend - INFO Loading multitrace from /test_beat/Laquila/geometry/stage_0 smc - INFO Updating Covariances ... seismic - INFO Updating data-covariances ... seismic - INFO Retrieving seismic data-covariances with structure "non-toeplitz" for any_P_0 ... seismic - INFO Not updating seismic velocity model-covariances because number of model variations is too low! < 5 geodetic - INFO Not updating geodetic velocity model-covariances because number of model variations is too low! < 5 sampler - INFO Updating last samples ... sampler - INFO in /test_beat/Laquila/geometry/trans_stage_0 sampler - INFO Initialising 1000 chain traces ... sampler - INFO Serial time per sample: 19.111001 sampler - INFO Data to be memory shared: sampler - INFO No data to be memshared! sampler - INFO Sampling ... parallel - INFO Worker timeout after 90 second(s) parallel - INFO Overseer timeout after 7500 second(s) parallel - INFO Chunksize: 250 multiprocessing.pool.RemoteTraceback: """
Traceback (most recent call last): | 0/1 [00:00<?, ?it/s] File "/usr/local/python35/lib/python3.5/site-packages/theano/compile/function_module.py", line 903, in call self.fn() if output_subset is None else\ File "/usr/local/python35/lib/python3.5/site-packages/theano/gof/op.py", line 892, in rval r = p(n, [x[0] for x in i], o) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/theanof.py", line 400, in perform arrival_times=arrival_times) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/heart.py", line 2574, in seis_synthetics targets=targets, nprocs=nprocs) File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 3146, in process nthreads=nthreads): File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 2692, in process_dynamic source, target, components, dsource_cache, nthreads) File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 2989, in base_seismogram nthreads=nthreads) File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/store.py", line 1907, in seismogram 'c', optimization) File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/store.py", line 856, in _sum int(itmin), int(nsamples))) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 116, in timeout_handler raise TimeoutException(traceback.format_stack()) beat.parallel.TimeoutException
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/base.py", line 279, in _sample for strace in sampling: File "/usr/local/python35/lib/python3.5/site-packages/tqdm-4.28.1-py3.5.egg/tqdm/_tqdm.py", line 979, in iter for obj in iterable: File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/base.py", line 333, in _iter_sample point, out_list = step.step(point) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/backend.py", line 93, in step apoint, alist = self.astep(self.bij.map(point)) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/metropolis.py", line 207, in astep l_new = self.logp_forw(q0) File "/usr/local/python35/lib/python3.5/site-packages/theano/compile/function_module.py", line 917, in call storage_map=getattr(self.fn, 'storage_map', None)) File "/usr/local/python35/lib/python3.5/site-packages/theano/gof/link.py", line 325, in raise_with_op reraise(exc_type, exc_value, exc_trace) File "/usr/local/python35/lib/python3.5/site-packages/six.py", line 692, in reraise raise value.with_traceback(tb) File "/usr/local/python35/lib/python3.5/site-packages/theano/compile/function_module.py", line 903, in call self.fn() if output_subset is None else\ File "/usr/local/python35/lib/python3.5/site-packages/theano/gof/op.py", line 892, in rval r = p(n, [x[0] for x in i], o) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/theanof.py", line 400, in perform arrival_times=arrival_times) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/heart.py", line 2574, in seis_synthetics targets=targets, nprocs=nprocs) File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 3146, in process nthreads=nthreads): File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 2692, in process_dynamic source, target, components, dsource_cache, nthreads) File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 2989, in base_seismogram nthreads=nthreads) File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/store.py", line 1907, in seismogram 'c', optimization) File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/store.py", line 856, in _sum int(itmin), int(nsamples))) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 116, in timeout_handler raise TimeoutException(traceback.format_stack()) beat.parallel.TimeoutException
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/python35/lib/python3.5/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, kwds)) File "/usr/local/beat/python35/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar return list(map(args)) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 175, in _pay_worker return overseer(worker.timeout)(worker.run)() File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 123, in wrapped_f result = func(args, kwargs) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 164, in run return self.function(*self.work) File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/base.py", line 291, in _sample strace.record_buffer() UnboundLocalError: local variable 'strace' referenced before assignment """
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/usr/local/python35/bin/beat", line 11, in
load_entry_point('beat==1.0rc1', 'console_scripts', 'beat')()
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x8664.egg/beat/apps/beat.py", line 1588, in main
globals()['command' + command](args)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/apps/beat.py", line 755, in command_sample
sample(step, problem)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/models/base.py", line 174, in sample
rm_flag=pa.rm_flag)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/smc.py", line 471, in smc_sample
homepath, step, progressbar, model, n_jobs, rm_flag)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/base.py", line 586, in update_last_samples
mtrace = iter_parallel_chains(**sample_args)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/base.py", line 483, in iter_parallel_chains
for res in p:
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 252, in paripool
chunksize=chunksize, callback=callback).get(pool_timeout)
File "/usr/local/python35/lib/python3.5/multiprocessing/pool.py", line 644, in get
raise self._value
UnboundLocalError: local variable 'strace' referenced before assignment
Can you help me solve this problem? Thank you!