cf --qstat is ridiculously slow when a significant number of jobs is running

[s-andrews]

$ time qstat -u "*" > /dev/null
real    0m0.101s
user    0m0.066s
sys     0m0.010s

$ time cf --qstatall > /dev/null
real    3m43.681s
user    3m42.923s
sys     0m0.467s

======================================================================
 Summary Job Counts
======================================================================
 Cluster Flow Pipelines:      9
 Running Jobs:              534
 Queued (Resources):       3492
 Jobs being deleted:         16

[s-andrews]

It looks like this is due to how crappily slow XML::Simple is at building the complete data structure from the XML document.

[s-andrews]

So apparently the problems with XML::Simple are well known:

http://www.perlmonks.org/?node_id=409517

They can be improved a lot by changing the back end parser used. XML::Parser seems to the one you want. If I switch the parsing to this then:

$ time cf --qstatall > /dev/null

real    0m19.179s
user    0m18.815s
sys     0m0.279s

..which is still a bit slower than I'd like but it's a lot better than before. The proper answer is probably to use a SAX interface and parse it on the fly using a state based parser, but that's a bit further off.

I'll submit a patch.

[ewels]

Wow, that is crappy. I was going to say that it works fine for me, but the slurm parser doesn't use XML so that's probably why.

I wrote this code aages ago though, so not sure why this would be a new problem for you guys?

[FelixKrueger]

The problem isn't new, we simply chose to not complain about it. Now that we are doing the configuration of the new version we might as well whine about it ...

[s-andrews]

Pull request submitted.

To be honest I've never used cf --qstat but we've hit another problem with the new version which that might have been the answer to - I'll be in touch about that one!

[ewels]

Brilliant, thanks! Yes I'm not surprised that you don't usually use it if it takes 3 minutes to return anything :) I use it all the time, even to monitor non-cluster flow jobs, I much prefer it to the default queue monitor output (it has pretty colours 😍 )

[ewels]

ps. Another tip to make cf --qstat more usable:

alias qs='cf --qstat'

[s-andrews]

There's another bug report coming about that :-)

On our system the job ids come out as white on a light yellow background...

[ewels]

Yeah, it's always done that - you disable it by running cf --setup and setting @colourful 0 in the config file. You guys just did it so long ago that you've forgotten 😉

[ewels]

@FelixKrueger suggested that colourful output should be opt-in instead of opt-out, and I'm inclined to agree.

[s-andrews]

Surely there must be a less controversial colour code to use? All of the other colours come out OK - just that one sucks.

[ewels]

* tries to defend colour choices.. *

Ok yeah, it does suck a bit. It is at least readable on my colour scheme and stands out (which was the desired effect):

I think it looked better with my previous light colour scheme when working at the BI. But maybe just using coloured text without the background would be less sucky.

How about... Green text?

Phil

[ewels]

See #102