Closed georgeha closed 4 years ago
Can you please send the output of radical-stack
, the wrangler command line, and any output you see? Thanks!
stack:
python : 2.7.14
pythonpath :
virtualenv : /home/george/ve
radical.analytics : v0.45.2-101-g8358b08@devel
radical.pilot : 0.47-0.47-14-gb1dff8bd@devel
radical.utils : 0.47.4
saga : 0.47-v0.46-48-g5825f1b0@devel
$ cd rp_launch_methods/analysis/data/exp-1-orte-mpi-exec/mpi-exec
$ radical-analytics-wrangler.py -d exp-mpi-exec/ -t exp
Get raw sessions:
Done.
Marking sessions for wrangling:
No new sessions to wrangle found.
$ cd rp_launch_methods/analysis/data/exp-1-orte-mpi-exec/orte
$ radical-analytics-wrangler.py -d exp-orte -t exp
...
same ouput: no new sessions found
Thanks, that helps. I won't manage to look at it today I'm afraid, but will ping back tomorrow evening.
Sure. I manage to re-wrangle. I am posting the output:
Marking sessions for wrangling:
expe-orte -- rp.session.one.george.017619.0000 -- Loading pilots:
expe-orte --- rp.session.one.george.017619.0000
pilot.0000:
WARNING: Failed to get timestamp for state CANCELED
WARNING: Failed to get timestamp for state DONE
P_PMGR_QUEUING util_p_uexec P_LRMS_RUNNING util_p_orte util_p_total util_p_setup_1 util_p_setup_2 P_LRMS_SUBMITTING P_LRMS_QUEUING util_p_term util_p_boot P_PMGR_SCHEDULING
stored in ./pilots.csv.
expe-orte -- rp.session.one.george.017619.0000 -- Loading units:
expe-orte --- rp.session.one.george.017619.0000
unit.000000:
util_u_exec_rp U_AGENT_QUEUING
WARNING: Failed to calculate duration util_u_unschedule
WARNING: Failed to calculate duration util_u_exec_orte
util_u_eprep util_u_exec_cu U_AGENT_SCHEDULING U_UMGR_SCHEDULING U_UMGR_BINDING util_u_total U_AGENT_QUEUING_EXEC U_AGENT_EXECUTING
WARNING: Failed to calculate duration util_u_exec_app
util_u_equeue
stored in ./units.csv.
expe-orte -- rp.session.one.george.017619.0000 -- Loading session:
expe-orte --- rp.session.one.george.017619.0000
stored in ./sessions.csv
no rel sync ./expe-orte/rp.session.one.george.017625.0000/pilot.0000/bootstrap_1.prof
no sync_abs event: [0.0026, 'bootstrap_1_start', 'bootstrap_1', 'MainThread', 'pilot.0000', 'PMGR_ACTIVE_PENDING', '', 'pilot']
expe-orte -- rp.session.one.george.017625.0000 -- Loading pilots:
no rel sync ./expe-orte/rp.session.one.george.017625.0000/pilot.0000/bootstrap_1.prof
no sync_abs event: [0.0026, 'bootstrap_1_start', 'bootstrap_1', 'MainThread', 'pilot.0000', 'PMGR_ACTIVE_PENDING', '', 'pilot']
expe-orte --- rp.session.one.george.017625.0000
expe-orte -- rp.session.one.george.017625.0000 -- Loading units:
no rel sync ./expe-orte/rp.session.one.george.017625.0000/pilot.0000/bootstrap_1.prof
no sync_abs event: [0.0026, 'bootstrap_1_start', 'bootstrap_1', 'MainThread', 'pilot.0000', 'PMGR_ACTIVE_PENDING', '', 'pilot']
expe-orte --- rp.session.one.george.017625.0000
expe-orte -- rp.session.one.george.017625.0000 -- Loading session:
expe-orte --- rp.session.one.george.017625.0000Traceback (most recent call last):
File "/home/george/teest_ve/bin/radical-analytics-wrangler.py", line 846, in <module>
wrangle_session(sdir, sid)
File "/home/george/teest_ve/bin/radical-analytics-wrangler.py", line 689, in wrangle_session
sdm, pdm, udm, pilots, units, sts)
File "/home/george/teest_ve/bin/radical-analytics-wrangler.py", line 528, in load_session
s['NEW'].append(min(sra_pilots.timestamps(state='NEW')))
ValueError: min() arg is an empty sequence
Some of your sessions don't have profiles, so analytics cannot find timestamps for events, and the error above is triggered for the first event in a pilot's life (getting into NEW
state).:
…/exp-1-orte-mpi-exec/mpi-exec/exp-mpi-exec [master] $ for d in rp*; do ls -l $d/*.prof > /dev/null; done
ls: cannot access 'rp.session.one.george.017624.0007/*.prof': No such file or directory
ls: cannot access 'rp.session.one.george.017624.0008/*.prof': No such file or directory
ls: cannot access 'rp.session.one.george.017624.0009/*.prof': No such file or directory
ls: cannot access 'rp.session.one.george.017624.0010/*.prof': No such file or directory
ls: cannot access 'rp.session.one.george.017624.0011/*.prof': No such file or directory
ls: cannot access 'rp.session.one.george.017624.0012/*.prof': No such file or directory
I am trying to wrangle events from these sessions.
from mpi-exec only 3 sessins are wrangled (as I can see from the sessions.csv) https://github.com/radical-experiments/rp_launch_methods/tree/master/analysis/data/exp-1-orte-mpi-exec
from the orte no sessions are wrangled. I am trying to reproduce the messages from the wrangler but I can't, because no new sessions are found.
Any ideas? Thanks