cli client: long startup times

[zenhack]

I've noticed on the box @naved001 just gave me access to that running the command line tool is a bit slow, enough to be slightly annoying:

$ time hil --help
Usage: hil [OPTIONS] COMMAND [ARGS]...

  The HIL Command line.

  Every subcommand supports --help option to see all arguments (positional
  and optional) and additional help for that subcommand.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  headnode           Commands related to headnode
  network            Commands related to network
  networking-action  Commands related to networking-actions
  node               Commands related to node
  port               Commands related to port
  project            Commands related to project
  switch             Commands related to switch
  user               Commands related to user

real    0m1.248s
user    0m1.013s
sys 0m0.237s
$ 

almost a second and a half just to display the help. Not a showstopper, but like I said, annoying, and it really shouldn't take that long to start up. We should figure out what's going on.

[naved001]

yeah, I was poking around at it once before and didn't really come to a conclusion.

The machine I gave you access to is a VM running on a 3 node oVirt setup which uses GlusterFS for the disks of these VMs. The kzn-hil-server VM has 8GB Ram, and 4 vCPUs.

These are the timings on my local machine (i7-3770, samsung evo 860 SSD, 32GB RAM, ubuntu18.04)

naved::~$time hil --help
Usage: hil [OPTIONS] COMMAND [ARGS]...

  The HIL Command line.

  Every subcommand supports --help option to see all arguments (positional
  and optional) and additional help for that subcommand.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  headnode           Commands related to headnode
  network            Commands related to network
  networking-action  Commands related to networking-actions
  node               Commands related to node
  port               Commands related to port
  project            Commands related to project
  switch             Commands related to switch
  user               Commands related to user

real    0m0.381s
user    0m0.331s
sys 0m0.051s

There's one more environment, kumo, where we have hil setup. Again, the hil server is a VM with it's disk on a 12 drive raid 6

[root@kumo-hil ~]# time hil --help
Usage: hil [OPTIONS] COMMAND [ARGS]...

  The HIL Command line.

  Every subcommand supports --help option to see all arguments (positional
  and optional) and additional help for that subcommand.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  headnode           Commands related to headnode
  network            Commands related to network
  networking-action  Commands related to networking-actions
  node               Commands related to node
  port               Commands related to port
  project            Commands related to project
  switch             Commands related to switch
  user               Commands related to user

real    0m0.630s
user    0m0.476s
sys 0m0.110s

So I don't know if it's something to do with the code itself, maybe it's due to slow disks?

[zenhack]

I took two traces of hil --help; this one is from the moc server:

kzn-trace.txt

This one is from my laptop:

local-trace.txt

The latter is taking ~0.4 seconds vs. ~1.3 seconds. There are about 75% as many syscalls being made on my local machine vs. the moc server. My laptop has an SSD for storage, so I suspect that disk access time is a big part of it; there are a lot of open/stat/mmap calls in there.

Python startup times are bad in general; the 0.4 seconds it takes to start up on my local machine is already pretty gross...