search

radical-collaboration / hpc-workflows

NSF16514 EarthCube Project - Award Number:1639694
5 stars 0 forks source link

Non-fatal error while processing observed data (Stampede) #18

Closed vivek-bala closed 7 years ago

vivek-bala commented 7 years ago

I process the obsd data using the pypaw-process_asdf executable as follows:

mpirun_rsh -np 16 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 c559-801 LMOD_CMD=$LMOD_CMD LMOD_SYSTEM_DEFAULT_MODULES=$LMOD_SYSTEM_DEFAULT_MODULES /work/02734/vivek91/modules/miniconda2/envs/pypaw_env/bin/pypaw-process_asdf "-f" "/work/02734/vivek91/modules/simpy/examples/titan_global_inv/paths/ProcessObserved/C201002060444A.proc_obsd_17_40.path.json" "-p" "/work/02734/vivek91/modules/simpy/examples/titan_global_inv/params/ProcessObserved/proc_obsd.17_40.param.yml"

I get a non-fatal error in stdout:

Error during the processing of station 'AC.BCI' and tag 'raw_observed' on rank 4:
Traceback (At max 3 levels - most recent call last):
  File "/work/02734/vivek91/modules/pyasdf/pyasdf/asdf_data_set.py", line 1689, in process
    traceback_limit=traceback_limit)
  File "/work/02734/vivek91/modules/pyasdf/pyasdf/asdf_data_set.py", line 1706, in _dispatch_processing_mpi
    traceback_limit=traceback_limit)
  File "/work/02734/vivek91/modules/pyasdf/pyasdf/asdf_data_set.py", line 1888, in _dispatch_processing_mpi_worker_node
    stream = process_function(stream, inv)
  File "/work/02734/vivek91/modules/pypaw/pypaw/process.py", line 47, in process_wrapper
    return process_stream(stream, **param)
  File "/work/02734/vivek91/modules/pytomo3d/pytomo3d/signal/process.py", line 227, in process_stream
    st = flex_cut_stream(st, starttime, endtime, dynamic_npts=10)
  File "/work/02734/vivek91/modules/pytomo3d/pytomo3d/signal/process.py", line 83, in flex_cut_stream
    raise ValueError("None of traces in Stream satisfy the "

ValueError: None of traces in Stream satisfy the cut time length

Entire log: https://gist.github.com/vivek-bala/e173c315222b27c503546fee4f3fb5ff

"Non-fatal" since I do see a 1.2G hdf5 file in the expected output location (/work/02734/vivek91/DATA_RADICAL/proc_obsd).

All the data locations have public permissions, so should be accessible.

Is this expected? Is this to be ignored/addressed?

wjlei1990 commented 7 years ago

Hi Vivek,

It is normal if you have such error for some of the traces in the asdf file. It is because for real observed data, some instrument may not be recording during the time period. For example, you are asking to 1 minutes data but the real observed data only contains 30 seconds. So when cut such seismograms, it may raise such errors. No big deal.

vivek-bala commented 7 years ago

Great, thanks! That is good to know.