unable to wrangle events

[georgeha]

I am trying to wrangle events from these sessions.

from mpi-exec only 3 sessins are wrangled (as I can see from the sessions.csv) https://github.com/radical-experiments/rp_launch_methods/tree/master/analysis/data/exp-1-orte-mpi-exec

from the orte no sessions are wrangled. I am trying to reproduce the messages from the wrangler but I can't, because no new sessions are found.

Any ideas? Thanks

[andre-merzky]

Can you please send the output of radical-stack, the wrangler command line, and any output you see? Thanks!

[georgeha]

stack:

  python               : 2.7.14
  pythonpath           :
  virtualenv           : /home/george/ve

  radical.analytics    : v0.45.2-101-g8358b08@devel
  radical.pilot        : 0.47-0.47-14-gb1dff8bd@devel
  radical.utils        : 0.47.4
  saga                 : 0.47-v0.46-48-g5825f1b0@devel

$ cd rp_launch_methods/analysis/data/exp-1-orte-mpi-exec/mpi-exec 
$ radical-analytics-wrangler.py -d exp-mpi-exec/ -t exp
Get raw sessions:
Done.
Marking sessions for wrangling:
No new sessions to wrangle found.

$ cd rp_launch_methods/analysis/data/exp-1-orte-mpi-exec/orte
$ radical-analytics-wrangler.py -d exp-orte -t exp
...
same ouput: no new sessions found

[andre-merzky]

Thanks, that helps. I won't manage to look at it today I'm afraid, but will ping back tomorrow evening.

[georgeha]

Sure. I manage to re-wrangle. I am posting the output:

Marking sessions for wrangling:

expe-orte -- rp.session.one.george.017619.0000 -- Loading pilots:

expe-orte --- rp.session.one.george.017619.0000
pilot.0000:
 WARNING: Failed to get timestamp for state CANCELED
 WARNING: Failed to get timestamp for state DONE
 P_PMGR_QUEUING util_p_uexec P_LRMS_RUNNING util_p_orte util_p_total util_p_setup_1 util_p_setup_2 P_LRMS_SUBMITTING P_LRMS_QUEUING util_p_term util_p_boot P_PMGR_SCHEDULING
stored in ./pilots.csv.

expe-orte -- rp.session.one.george.017619.0000 -- Loading units:

expe-orte --- rp.session.one.george.017619.0000
unit.000000:
 util_u_exec_rp U_AGENT_QUEUING
WARNING: Failed to calculate duration util_u_unschedule

WARNING: Failed to calculate duration util_u_exec_orte
 util_u_eprep util_u_exec_cu U_AGENT_SCHEDULING U_UMGR_SCHEDULING U_UMGR_BINDING util_u_total U_AGENT_QUEUING_EXEC U_AGENT_EXECUTING
WARNING: Failed to calculate duration util_u_exec_app
 util_u_equeue
stored in ./units.csv.

expe-orte -- rp.session.one.george.017619.0000 -- Loading session:

expe-orte --- rp.session.one.george.017619.0000
stored in ./sessions.csv
no rel sync  ./expe-orte/rp.session.one.george.017625.0000/pilot.0000/bootstrap_1.prof
no sync_abs event: [0.0026, 'bootstrap_1_start', 'bootstrap_1', 'MainThread', 'pilot.0000', 'PMGR_ACTIVE_PENDING', '', 'pilot']

expe-orte -- rp.session.one.george.017625.0000 -- Loading pilots:
no rel sync  ./expe-orte/rp.session.one.george.017625.0000/pilot.0000/bootstrap_1.prof
no sync_abs event: [0.0026, 'bootstrap_1_start', 'bootstrap_1', 'MainThread', 'pilot.0000', 'PMGR_ACTIVE_PENDING', '', 'pilot']

expe-orte --- rp.session.one.george.017625.0000

expe-orte -- rp.session.one.george.017625.0000 -- Loading units:
no rel sync  ./expe-orte/rp.session.one.george.017625.0000/pilot.0000/bootstrap_1.prof
no sync_abs event: [0.0026, 'bootstrap_1_start', 'bootstrap_1', 'MainThread', 'pilot.0000', 'PMGR_ACTIVE_PENDING', '', 'pilot']

expe-orte --- rp.session.one.george.017625.0000

expe-orte -- rp.session.one.george.017625.0000 -- Loading session:

expe-orte --- rp.session.one.george.017625.0000Traceback (most recent call last):
  File "/home/george/teest_ve/bin/radical-analytics-wrangler.py", line 846, in <module>
    wrangle_session(sdir, sid)
  File "/home/george/teest_ve/bin/radical-analytics-wrangler.py", line 689, in wrangle_session
    sdm, pdm, udm, pilots, units, sts)
  File "/home/george/teest_ve/bin/radical-analytics-wrangler.py", line 528, in load_session
    s['NEW'].append(min(sra_pilots.timestamps(state='NEW')))
ValueError: min() arg is an empty sequence

[andre-merzky]

Some of your sessions don't have profiles, so analytics cannot find timestamps for events, and the error above is triggered for the first event in a pilot's life (getting into NEW state).:

 …/exp-1-orte-mpi-exec/mpi-exec/exp-mpi-exec  [master] $ for d in rp*; do ls -l $d/*.prof > /dev/null; done
ls: cannot access 'rp.session.one.george.017624.0007/*.prof': No such file or directory
ls: cannot access 'rp.session.one.george.017624.0008/*.prof': No such file or directory
ls: cannot access 'rp.session.one.george.017624.0009/*.prof': No such file or directory
ls: cannot access 'rp.session.one.george.017624.0010/*.prof': No such file or directory
ls: cannot access 'rp.session.one.george.017624.0011/*.prof': No such file or directory
ls: cannot access 'rp.session.one.george.017624.0012/*.prof': No such file or directory