Feature/session plots

[andre-merzky]

This moves the core logic of core placement / utilization plots into a session method, and adds a simplified example how to create utilization plots out of the returned data. The example will calculate overall utilization like:

$ ./examples/09_core_allocation_plot.py data/raw/exp4/rp.session.login4.mturilli1.018128.0000
alloc: 26643624.72 [100.0%]
block: 19103353.02 [ 71.7%]
use  : 14749665.54 [ 55.4%]

and create plots like the one attached.

[mturilli]

@andre-merzky can you help me to interpret the plot? I am not able to make sense of it.

[andre-merzky]

axis labels and legend are missing I realize ... :-/ labels:

• x axis: time
• y axis: slot index (0... number of pilot cores)

legend:

• yellow - allocated: cores are allocated by pilot (full pilot runtime)
• light orange - blocked: cores are blocked from use by RP (they are assigned to a task, but the task did not yet start / already finished)
• orange - used: cores are used by the actual application workload

So, this basically shows where and when tasks we placed and executed.

[mturilli]

Thanks, now I get it. Does the white space between 0 and ~1300 represent queue time?

[andre-merzky]

Does the white space between 0 and ~1300 represent queue time?

Well, basically everything before pilot startup: queue time, pilot submission time, whatever.

[mturilli]

Thanks, then yes, I would eliminate that 'everything' and consider this plot the visual representation of how available resources have been used (or not)

[mturilli]

One question: I am assuming we can use as many colors as we want, one for each couple of events we care to represent. For example, it would be useful to show whether the scheduling time was constant in relation to the light orange portion?

[andre-merzky]

One question: I am assuming we can use as many colors as we want, one for each couple of events we care to represent. For example, it would be useful to show whether the scheduling time was constant in relation to the light orange portion?

At the moment, on;y three categories are defined (allocated, blocked, used), so there are only 3 colors to show (one per category).

I am not sure I understand what you mean with to show whether the scheduling time was constant: the light orange is not really scheduling time, but the time from when scheduling succeeded to time when application started. Does that help / answer?

[andre-merzky]

Attaching another sample with a small heterogeneous run - tasks and gaps are easier to identify here.

[mturilli]

Thanks. The light orange should also include the time from when the scheduling started to when the scheduling finished and, going backwards, the time from when the unit was received to when it was queued for scheduling. This is because, during 'all' that time, the resource are already available. Or is available to the pilot different from allocated by the pilot?

[andre-merzky]

what you call 'available' is what I mean with allocated - the pilot owns the core and can, in principle, use it. That is the yellow color. light orange is when the core is assigned to a specific unit - it is the blocked for anything else. Orange is when the workload actually uses the core.

We can't extend backwards to when the unit was received, or when scheduling started - at that point we do not know what core the unit will land on, and in fact the core might be used by a different unit at that point, which would not help understanding the plot ;-)

[mturilli]

Great, thank you! Now I am confident I understand the plot and I am looking forward to using it :D

[andre-merzky]

:-)