search

OSC / ood_core

Open OnDemand core library
https://osc.github.io/ood_core/
MIT License
9 stars 26 forks source link

LSF Adapter: Address performance issues #40

Open ericfranz opened 7 years ago

ericfranz commented 7 years ago

Need to do a little performance analysis. A better algorithm might do. However, there is a lot of parsing and string manipulation going on. Using the C library might make it faster (via Fiddle http://ruby-doc.org/stdlib-2.2.0/libdoc/fiddle/rdoc/Fiddle.html).

It was observed when testing that 30 jobs can take 1-2 seconds and 4000 jobs could take 28 seconds. This is way too long.

Many Fiddle tutorials online. One example: http://blog.honeybadger.io/use-any-c-library-from-ruby-via-fiddle-the-ruby-standard-librarys-best-kept-secret/

See http://publibfp.dhe.ibm.com/epubs/pdf/c2753121.pdf and lsb_openjobinfo() and lsb_readjobinfo() and jobInfoEnt structure. Also, here is some example code that uses the C library: https://github.com/PlatformLSF/lsf-drmaa/blob/025f9c49af48e410dc0ab0b9c611c42935dc09eb/lsf_drmaa/job.c

┆Issue is synchronized with this Asana task by Unito

ericfranz commented 7 years ago
  1. jobInfoEnt structure appears to be identical between 8.3 and 9.1
  2. Fiddle does not provide a time_t value. Try using Fiddle::TYPE_LONG in its place. Or you could try this in Fiddle:
typealias "time_t", "long"