search

fasrc / slurmmon

gather and plot data about Slurm scheduling and job statistics
Other
50 stars 14 forks source link

problems slurmmon_whitespace_report #3

Open henkela opened 10 years ago

henkela commented 10 years ago

Running /usr/sbin/slurmmon_whitespace_report shows: Traceback (most recent call last): File "/usr/sbin/slurmmon_whitespace_report", line 230, in live_nodes_cpu=True, File "/usr/sbin/slurmmon_whitespace_report", line 117, in write_report for i, x in enumerate((j['User'], j['JobID'], int(round(j['CPU_Wasted']/(60_60_24))), '%d%%' % int(round(j['CPU_Efficiency']100)), j['NCPUS'], config.syntax_highlight(j['JobScriptPreview']))): TypeError: unsupported operand type(s) for : 'NoneType' and 'int'

Setting completed_jobs_cpu=False, I obtain: * ERROR * unable to parse squeue job text ['']: need more than 0 values to unpack

Any advice? My current guess would be that you're using a different resource management in slurm. Maybe you could provide your settings in slurm.conf or add the configuration prerequisites for using slurmmon?

jabrcx commented 10 years ago

Thanks for the report.

Both exceptions can be explained by slurmmon not accounting for edge cases -- bugs which I've just patched.

1) TypeError with * operation

Could've been caused by a COMPLETED job that had zero TotalCPU (something we apparently have not seen), which poorly resulted in a CPU_Efficiency of None. Commit 3d151c1 should have that working properly now.

(Side note: mixing None and KeyError was premature optimization that I've been backing out of in a refactoring (wip).)

2) ValueError parsing squeue output

Could've happened if there were no jobs running on one of the hosts that's "wasting" the most cpu (which will happen if CPUAlloc~=0 everywhere, i.e. no jobs to be wasteful). Commit 6002e14 should have fix that now.

Also, in both cases it was not nice that one thing (job or node) could kill the whole loop. Instead it now prints errors to the terminal (note: NOT the html report yet) for that thing and moves on to the next thing (and exits with non-zero exit status).

Of course, I don't really know if these were the reasons you were getting these errors or not. Could you try the latest RPMs (version 0.0.2-fasrc04) or files from the repo and see if things are better?

If not, I'm happy to dig deeper, share config (to answer your question, I haven't identified any assumptions/requirements), etc.

henkela commented 10 years ago

Thanks for your fast reply!

I've downloaded new newest files and now slurmmon_whitespace_report runs without errors. It just gives a warning for some jobs: WARNING: unable to process job details for job [<JobID 8480>]: 'NoneType' object has no attribute 'rsplit'

jabrcx commented 10 years ago

Okay, looks like some more unexpected input it has to account for. (Again, after the current refactor, it'll raise a KeyError naming what parameter it was assuming was present is not there, rather than the unhelpful error from trying to use None, but I digress...)

Are those empty jobs more the exceptional situation or the usual case? Can you think of anything edge-casey about the jobs?

Thanks!

henkela commented 10 years ago

Actually, I'm running a test cluster with a hand full of nodes. I'm checking slurm as potential batch system for our next cluster (instead of LSF). For having some jobs, I wrote a little bash script which submits jobs for some random test users asking for a random amount of cores and sending the command sleep with random time. So, no real jobs there at the moment. Don't know if this causes the warning.