search

radical-cybertools / radical.pilot

RADICAL-Pilot
http://radical-cybertools.github.io/radical-pilot/index.html
Other
54 stars 23 forks source link

Missing states in the unit dataframe #550

Closed mturilli closed 9 years ago

mturilli commented 9 years ago

Currently:

db = 'mongodb://54.221.194.147:24242/mturilli_aimesexperiments'
session = '5512d11123769c2c6788a5db'
s_frame, p_frame, u_frame = rpu.get_session_frames(db,session,cachedir)
u_frame
   Canceled      Done  Executing Failed     New Pending  PendingExecution  PendingInputStaging PendingOutputStaging  Scheduling  StagingInput  StagingOutput  Unscheduled  cores  finished                       pid                       sid   started                       uid
0      None  5868.715   4949.973   None  71.869    None          4949.745             4931.404                 None    4949.831      4931.881       5850.696       72.191      4  5850.696  5512d14323769c2c6788a5de  5512d11123769c2c6788a5db  4949.973  5512d15623769c2c6788a61c
1      None  5872.830   4955.384   None  71.928    None          4954.072             4931.418                 None    4955.242      4931.922       5864.667       72.191      4  5855.885  5512d14323769c2c6788a5de  5512d11123769c2c6788a5db  4955.384  5512d15723769c2c6788a61e
2      None  5874.412   4952.678   None  71.968    None          4952.192             4931.426                 None    4952.537      4949.758       5853.399       72.191      4  5853.240  5512d14323769c2c6788a5de  5512d11123769c2c6788a5db  4952.678  5512d15723769c2c6788a61f
...

Related to #548 #538 #537 possibly #547 #540

mturilli commented 9 years ago

RP 0.28

Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import radical.utils as ru
>>> import radical.pilot.utils as rpu
>>> import pandas as pd
>>> pd.set_option('display.width', 1000)
>>> cachedir = os.getcwd()
>>> dburl = 'mongodb://54.221.194.147:24242/mturilli_aimesexperiments'
>>> session = 'rp.session.ip-10-184-31-85.mturilli.016547.0001'
>>> mongo, db, dbname, cname, pname = ru.mongodb_connect (str(dburl))
>>> s_frame, p_frame, u_frame = rpu.get_session_frames(db,session,cachedir)
>>> s_frame
   finished  n_pilots  n_units                                              sid       started
0  1917.653         3        8  rp.session.ip-10-184-31-85.mturilli.016547.0001  1.429710e+09
>>> p_frame
   Active  Canceled  Done Failed  Launching   New  PendingActive  PendingLaunch  cores  finished  n_units         pid          resource  runtime                                              sid  started
0     NaN       NaN  None   None      3.542  None         15.643          2.579     20       NaN        0  pilot.0000  xsede.blacklight       56  rp.session.ip-10-184-31-85.mturilli.016547.0001      NaN
1     NaN       NaN  None   None     15.645  None         33.529          7.083     20       NaN        0  pilot.0001    xsede.stampede       56  rp.session.ip-10-184-31-85.mturilli.016547.0001      NaN
2  91.419  1917.653  None   None     33.531  None         42.300          9.673     20  1917.653        8  pilot.0002      xsede.gordon       56  rp.session.ip-10-184-31-85.mturilli.016547.0001   91.419
>>> u_frame
   Allocating Canceled      Done  Executing Failed   New  PendingExecution PendingInputStaging PendingOutputStaging  Scheduling  StagingInput  StagingOutput Unscheduled  cores  finished         pid                                              sid   started          uid
0      98.422     None  1005.850     98.472   None  None            98.079                None                 None       9.922        92.044        999.910        None      8   998.953  pilot.0002  rp.session.ip-10-184-31-85.mturilli.016547.0001    98.472  unit.000001
1      97.258     None  1003.577     97.318   None  None            96.788                None                 None       9.942        91.904        999.149        None      4   997.946  pilot.0002  rp.session.ip-10-184-31-85.mturilli.016547.0001    97.318  unit.000006
2      99.811     None  1006.855     99.876   None  None            99.335                None                 None       9.928        95.637       1003.095        None     16  1000.358  pilot.0002  rp.session.ip-10-184-31-85.mturilli.016547.0001    99.876  unit.000002
3     101.160     None  1004.846    101.210   None  None           100.697                None                 None       9.949        96.789       1003.579        None      4  1001.664  pilot.0002  rp.session.ip-10-184-31-85.mturilli.016547.0001   101.210  unit.000007
4     103.134     None  1904.894    998.001   None  None           102.706                None                 None       9.940        98.081       1900.466        None      2  1898.497  pilot.0002  rp.session.ip-10-184-31-85.mturilli.016547.0001   998.001  unit.000005
5     102.035     None  1907.162   1000.424   None  None           101.701                None                 None       9.933        96.947       1904.396        None     16  1900.911  pilot.0002  rp.session.ip-10-184-31-85.mturilli.016547.0001  1000.424  unit.000003
6     105.108     None  1908.168    999.014   None  None           104.715                None                 None       9.921        99.336       1904.895        None      8  1899.506  pilot.0002  rp.session.ip-10-184-31-85.mturilli.016547.0001   999.014  unit.000000
7     104.255     None  1906.158    998.055   None  None           103.711                None                 None       9.934        98.229       1900.674        None      2  1898.500  pilot.0002  rp.session.ip-10-184-31-85.mturilli.016547.0001   998.055  unit.000004

Missing states

Incorrect states

andre-merzky commented 9 years ago

This has been fixed, frames seem to be working for Matteo's analysis now.