Track job resource usage on Flux

[pjvandehaar]

• [x] Fetch job results from a dedicated email account.
• [ ] Use jobnames or timing to select jobs from the current make.
• [ ] Summarize resources_used.cput, resources_used.mem, and resources_used.walltime.
• [ ] Assert Exit_status=0.
• [ ] Classify jobs by task before summaries. Use jobnames?

[pjvandehaar]

Here's python to retrieve emails and when they were sent:

from __future__ import print_function
import imaplib, getpass, email, dateutil.parser, sys
m = imaplib.IMAP4_SSL('imap.gmail.com')
m.login('<your_email_here>@gmail.com', getpass.getpass())
# use m.list() to find the right mailbox
m.select('"[Gmail]/All Mail"') # Beware the awkward doubled quotes
rv = m.search(None, 
     '(SUBJECT "PBS JOB") (SUBJECT "Execution terminated") (FROM "hpc-support")')
message_ids = rv[1][0].split()
rv = m.fetch(message_ids[0], '(RFC822)')
if sys.version_info.major > 2:
    msg = email.message_from_bytes(rv[1][0][1])
else:
    msg = email.message_from_string(rv[1][0][1])
print(dateutil.parser.parse(msg['Date']))
for line in msg.get_payload().split('\n'):
    line = line.strip() # I hate "\r"
    for line_start in ['resources_used.cput=', 'resources_used.mem=', 
            'resources_used.walltime=', 'Job Name:', 'PBS Job Id:']:
        if line.startswith(line_start):
            print(line_start, 'is', line.lstrip(line_start).strip())

[pjvandehaar]

For an MVP:

• [ ] Starting at the largest message_id, walk backwards until there's a 24-hour gap.
• [ ] Plot a metric vs time graph for each of cput, walltime, mem.

ascii plotting libraries:

• I can use gnuplot manually like this or with the terribly old gnuplot.py interface. /usr/bin/gnuplot is available on CSG and Flux.
• sympy.plotting.textplot works, but only supports sage-style symbolic magic.
• scitools.aplotter is hard to install.
• bashplotlib is uninstallable.

Raw gnuplot works:

import subprocess, random
rows, columns = (int(r) for r in subprocess.check_output([b'stty', b'size']).split())

data1 = [(x*0.1, random.gauss(40,3) - 30*(x/100.)**2) for x in range(0,100)]
data2 = [(x*0.1, random.gauss(20,3) + 30*(x/100.)**2) for x in range(0,100)]

gnuplot = subprocess.Popen("/usr/bin/gnuplot", stdin=subprocess.PIPE)
gnuplot.stdin.write("set term dumb {} {}\n".format(columns - 3, rows - 3).encode())
gnuplot.stdin.write(b"set rmargin 0\nset tmargin 0\n")
gnuplot.stdin.write(b"plot '-' title 'foo', '-' title 'bar'\n")
for data in (data1, data2):
    for x,y in data:
        gnuplot.stdin.write("{} {}\n".format(x,y).encode())
    gnuplot.stdin.write(b"e\n")
gnuplot.stdin.write(b"exit\n")
gnuplot.wait()

[pjvandehaar]

Once I have something decent, I will post it to http://genome.sph.umich.edu/wiki/Using_Gotcloud_on_Flux and make links to that page from others.

[pjvandehaar]

Oh well, nevermind.