Save session with timings to mongoDB

[mturilli]

This feature is required by AIMES and a requirement for a milestone. Happy to code it into TROY myself if other developers are too busy. Just let know.

[andre-merzky]

Hi Matteo,

I pushed the current state to a separate branch (feature/timing). This is probably incomplete and unstable, but I would like to get feedback at this point if this is what you need.

You will need radical.utils devel branch, as I am reusing the mongodb setup from the dumper utility.

You should be able to try the following:

python examples/phase1_demo_2.py

This should print, near the end, something like this:

    event    timed_create ['session.2014-02-05.13:16:36.0001']
    duration saga.Session ['init']
    troy.Task
      t.0004
      t.0005
      t.0006
      t.0007
      t.0001
      t.0002
      t.0003
      t.0008
      t.0009
    troy.Unit
      cu.0009
      cu.0008
      cu.0005
      cu.0004
      cu.0007
      cu.0006
      cu.0001
      cu.0003
      cu.0002
    troy.Workload
      wl.0002
      wl.0003
      wl.0001
    troy.Overlay
      ol.0001
    troy.OverlayManager
      olm.0001
    troy.Planner
      planner.0001
    troy.WorkloadManager
      wlm.0001
    troy.Pilot
      p.0001
      p.0002

which shows the structure of the object tree. Copy the session ID from above (session.2014-02-05.13:16:36.0001), and you can inspect the data stored in mongodb via:

dump_mongodb.py dump mongodb://localhost/timing/session.2014-02-05.13:16:36.0001

Again, this is beta, so please let me know if that works, where not, and if this is close to what you need.

[andre-merzky]

PS.: some events are logged with timestamp 0, i.e. January first 1970, i.e. the beginning of time. For those events I do not have a valid time stamp, only know that the event occured from Sinon's state details.

PPS.: I'll add the same data from BigJob as soon as you let me know that this is the right approach.

[mturilli]

Thank you Andre,

We may need to add the dependency on pymongo "ImportError: No module named pymongo" at the first attempt. More to follow :)

[andre-merzky]

I added pymongo to radical.utils dependencies now, this should solve it. Sorry for the oversight...

FWIW, here are a number of commands to dig information out of the database:

• list timed sessions:

dump_mongodb.py list mongodb://localhost/timing/

• list elements timed in the session (session ID copied from output above):

dump_mongodb.py list mongodb://localhost/timing/session.2014-02-07.22:32:10.0001

• list timings for workload wl.0001 -- this dump will also list all task IDs of that workload:

dump_mongodb.py dump mongodb://localhost/timing/session.2014-02-07.22:32:10.0001/wl.0001

• list all timings for task t.0004 listed above -- this will also list CU IDs of that task:

dump_mongodb.py dump mongodb://localhost/timing/session.2014-02-07.22:32:10.0001/t.0004

• list timings for CU cu.0006 listed above:

dump_mongodb.py dump mongodb://localhost/timing/session.2014-02-07.22:32:10.0001/cu.0006

• dump the complete session:

dump_mongodb.py dump mongodb://localhost/timing/session.2014-02-07.22:32:10.0001/

I just realized that the components field of the overlays is empty -- that should contain the pilot IDs for that overlay. The pilots themself are timed and included, but that relation is missing. Will add in a minute -- but the same mechanism works in principle on the pilot side, too.

[andre-merzky]

The problem listed at the end of the last comment is now fixed, in 3996ba5.

[mturilli]

examples/phase1_demo_2.py fails with:

Traceback (most recent call last):
  File "examples/phase1_demo_2.py", line 86, in <module>
    overlay_descr = planner.derive_overlay (workload_1.id)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/planner/planner.py", line 143, in derive_overlay
    self._planner.derive_overlay, [workload])
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/utils/time_it.py", line 249, in timed_method
    return method (*args, **kwargs)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/plugins/planner/plugin_planner_concurrent.py", line 87, in derive_overlay
    s_partition = troy.Workload ()
TypeError: __init__() takes at least 2 arguments (1 given)

Please note:

• While trying to debug the error, I noticed that I was not able to specify an empty session session = {} to pass to troy.Planner/WorkloadManager/OverlayManager.
• When instantiating troy.Planner/WorkloadManager/OverlayManager without session, I get the following error:

    Traceback (most recent call last):
      File "examples/phase1_demo_2.py", line 64, in <module>
        planner     = PLUGIN_PLANNER            )
    TypeError: __init__() takes at least 2 arguments (2 given)

Type error in 2 arguments when I pass 2 arguments?

Please let me know whether you need the full trace.

[andre-merzky]

I pushed a fix which allows again to use default sessions.

Thanks, Andre.

[mturilli]

Thanks Andre. I still see the original error though:

$ python examples/phase1_demo_2.py
2014:02:08 07:56:06 MainThread   troy.logger           : [CRITICAL] session id: session.2014-02-08.12:56:06.0001
File "examples/phase1_demo_2.py", line 60, in <module>
    session = troy.Session ({'concurrent_planner' : {'concurrency' : '100'}})
File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/session.py", line 54, in __init__
    for line in traceback.format_stack():
Exception saga.exceptions.DoesNotExist: attribute key is invalid: state (/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/saga/attributes.py +401 (_attributes_t_init)  :  raise se.DoesNotExist ("attribute key is invalid: %s"  %  (key)))
  File "examples/phase1_demo_2.py", line 86, in <module>
    overlay_descr = planner.derive_overlay (workload_1.id)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/planner/planner.py", line 145, in derive_overlay
    self._planner.derive_overlay, [workload])
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/utils/time_it.py", line 249, in timed_method
    return method (*args, **kwargs)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/plugins/planner/plugin_planner_concurrent.py", line 87, in derive_overlay
    s_partition = troy.Workload ()
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/workload/workload.py", line 184, in __del__
    self.cancel ()
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/workload/workload.py", line 195, in cancel
    if  self.state in [DISPATCHED] :
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/radical/utils/signatures.py", line 446, in takes_invocation_proxy
    return method(*pargs, **pkwargs)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/radical/utils/signatures.py", line 477, in returns_invocation_proxy
    result = method (*args, **kwargs)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/saga/attributes.py", line 2628, in __getattr__
    return self._attributes_i_get      (key, flow=self._DOWN)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/saga/attributes.py", line 1140, in _attributes_i_get
    d = self._attributes_t_init (key)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/radical/utils/signatures.py", line 446, in takes_invocation_proxy
    return method(*pargs, **pkwargs)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/radical/utils/signatures.py", line 477, in returns_invocation_proxy
    result = method (*args, **kwargs)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/saga/attributes.py", line 401, in _attributes_t_init
    raise se.DoesNotExist ("attribute key is invalid: %s"  %  (key))
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/saga/exceptions.py", line 425, in __init__
    SagaException.__init__ (self, msg, parent, api_object, from_log)
 in  ignored
Traceback (most recent call last):
  File "examples/phase1_demo_2.py", line 86, in <module>
    overlay_descr = planner.derive_overlay (workload_1.id)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/planner/planner.py", line 145, in derive_overlay
    self._planner.derive_overlay, [workload])
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/utils/time_it.py", line 249, in timed_method
    return method (*args, **kwargs)
  File "/Users/mturilli/Virtualenvs/TROY_timings/lib/python2.7/site-packages/troy/plugins/planner/plugin_planner_concurrent.py", line 87, in derive_overlay
    s_partition = troy.Workload ()
TypeError: __init__() takes at least 2 arguments (1 given)

[andre-merzky]

Ah, sorry, I misread. Yes, please pass a session to the workload. Otherwise its somewhat difficult to associate WL timings with a session.

This will probably change / simplify in the future, but that will need some thinking. At the moment it is possible to pass different sessions to workload_manager, overlay_manager and planner, as we wanted to keep them fully independent, but it is not clear (to me) what it means semantically if they use different sessions.

Thanks, Andre.

[mturilli]

Hi Andre,

Apologies, my bad. The session thing at the moment is a second-order concern. I reported it for completeness. That error is produced by the default version of examples/phase1_demo_2.py. The only change is WORKDIR.

PLUGIN_PLANNER              = 'concurrent'
PLUGIN_OVERLAY_SCHEDULER    = 'round_robin'
PLUGIN_OVERLAY_TRANSLATOR   = troy.AUTOMATIC
PLUGIN_OVERLAY_PROVISIONER  = 'sinon'
PLUGIN_WORKLOAD_SCHEDULER   = troy.AUTOMATIC
PLUGIN_WORKLOAD_DISPATCHER  = 'sinon' # troy.AUTOMATIC # 'sinon'

#WORKDIR                     = '/N/u/mturilli/troy_demo/'
WORKDIR                     = '/Users/mturilli/Projects/RADICAL/testing/AIMES/mongodb'

    # create a session with custom config options
    session = troy.Session ({'concurrent_planner' : {'concurrency' : '100'}})

    # create planner, overlay and workload manager, with plugins as configured
    planner      = troy.Planner         (session     = session                   ,
                                         planner     = PLUGIN_PLANNER            )
    workload_mgr = troy.WorkloadManager (session     = session                   ,
                                         scheduler   = PLUGIN_WORKLOAD_SCHEDULER ,
                                         dispatcher  = PLUGIN_WORKLOAD_DISPATCHER)
    overlay_mgr  = troy.OverlayManager  (session     = session                   ,
                                         scheduler   = PLUGIN_OVERLAY_SCHEDULER  ,
                                         translator  = PLUGIN_OVERLAY_TRANSLATOR ,
                                         provisioner = PLUGIN_OVERLAY_PROVISIONER)

[andre-merzky]

NP for reporting! :)

But in that example, I see:

workload_1 = troy.Workload (session=session)

so that error should not occur (in this example -- I did not make the change in other examples). Hmmm...

[mturilli]

I am basically running the example out of a fresh clone + ve. How can I help with the debugging?