LLNL / maestrowf

A tool to easily orchestrate general computational workflows both locally and on supercomputers
https://maestrowf.readthedocs.io
MIT License
133 stars 43 forks source link

Maestro Status Viz #282

Closed doutriaux1 closed 4 years ago

doutriaux1 commented 4 years ago

There's probably a better way to this, but the following script helps me visualize better Maestro status output by converting it to a panda dataframe that I then sort and viz in a notebook.

Just leaving it here in case someone finds it useful

from subprocess import Popen, PIPE
import pandas as pd
import shlex

study = "/p/lustre1/cdoutrix/ALE/Hohlraum/generate_hohlraum_20200605-072600"
maestro = "/g/g19/cdoutrix/miniconda3/envs/kosh/bin/maestro"
cmd = f"{maestro} status {study}"
p = Popen(shlex.split(cmd), stderr=PIPE, stdout=PIPE)
status,e = p.communicate()

def split_row(row, cols):
    start = 0
    values = []
    for c in cols:
        length = len(c)+2
        values.append(row[start:start+length].strip())
        start += length
    return values

def parse_status(status):
    lines = status.split("\n")
    names = lines[0]
    delimiters_line = lines[1].split()
    columns = split_row(names, delimiters_line)
    data = []
    for ln in lines[2:-1]:
        data.append(split_row(ln, delimiters_line))
    return columns, data

columns, data = parse_status(status.decode())

df = pd.DataFrame(data, columns=columns)
df.sort_values("State")
FrankD412 commented 4 years ago

@doutriaux1 -- Thanks for the info. Do you mind taking a look at the sister repository for examples? Could you make a PR for this there? https://github.com/LLNL/maestro_sheetmusic

doutriaux1 commented 4 years ago

@FrankD412 I wasn't aware of this repo. Will do. Thanks!

FrankD412 commented 4 years ago

@FrankD412 I wasn't aware of this repo. Will do. Thanks!

No worries -- it's new. I just started it yesterday haha Thanks for the contribution. :)