Prof utils fails with KeyError: 'index'

[AymenFJA]

This is related to the IMPRESS paper.

  python               : /home/aymen/ve/rp_debug/bin/python3
  pythonpath           : 
  version              : 3.10.12
  virtualenv           : /home/aymen/ve/rp_debug

  radical.analytics    : 1.81.0
  radical.entk         : 1.81.0
  radical.gtod         : 1.81.0
  radical.pilot        : 1.83.0
  radical.utils        : 1.81.0

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[10], line 1
----> 1 plot_utilization(data, resources=['cpu', 'gpu'], save=True, with_metrics=False)

Cell In[9], line 86, in plot_utilization(data, resources, save, with_metrics)
     83 for idx, (p_uid, task_uids) in enumerate(p_tasks.items()):
     84     p_session = data['session'].filter(uid=task_uids + [pid], inplace=False)
---> 86     consumed = rp.utils.get_consumed_resources(
     87         p_session, rtype, {'consume': DURATIONS})
     89     METRICS[-1][-1] = M_COLOR_RUNNING[idx]
     90     # generate the subplot with labels

File ~/ve/rp_debug/lib/python3.10/site-packages/radical/pilot/utils/prof_utils.py:937, in get_consumed_resources(session, rtype, tdurations)
    934 consumed = dict()
    935 for e in session.get(etype=['pilot', 'task']):
--> 937     if   e.etype == 'pilot': data = _get_pilot_consumption(e, rtype)
    938     elif e.etype == 'task' : data = _get_task_consumption(session, e, rtype,
    939                                                           tdurations)
    941     for metric in data:

File ~/ve/rp_debug/lib/python3.10/site-packages/radical/pilot/utils/prof_utils.py:1176, in _get_pilot_consumption(pilot, rtype)
   1174 if t0 is not None:
   1175     for node in pnodes:
-> 1176         r0, r1 = get_node_index(nodes, node['index'], pn)
   1177         boxes.append([t0, t1, r0, r1])
   1179 ret[metric] = {pid: boxes}

KeyError: 'index'

[andre-merzky]

Thanks @AymenFJA - can you please attach the session / profiles? Or do you know a way to reproduce that issue?

[andre-merzky]

FWIW - please make sure that you use the same radical stack (and here specifically RP version) when producing the profile and when running analytics.