Web interface slow due to repeated calling off 'get_bhyve_profiles'

[gizahNL]

Currently the site calls get_bhyve_profiles for every page load, most likely every tiny thing as well as I see many instances running in paralel.

This slows down incredibly, ideally output should be cached and refreshed when needed (if output is static once an hour via cronjob might be ideal)

[gizahNL]

Simple dirty hack to work arround: create the following script:

#!/bin/sh
/usr/jails/modules/get_bhyve_profiles --vm > /tmp/profile_vm
/usr/jails/modules/get_bhyve_profiles --template > /tmp/profile_template

call it from cron every 5 minutes

place in /usr/jails/modules following scripts: /usr/jails/modules/profile_vm

#!/bin/sh
cat /tmp/profile_vm

/usr/jails/modules/profile_template

#!/bin/sh
cat /tmp/profile_template

then modify php/config.php and replace get_bhyve_profiles with the corrospondending scripts.

[gizahNL]

without it page loading of clonos site is at least 40+ seconds, with the workaround using flat file ("cached response") responsiveness is in the milliseconds

[olevole]

the cache for profiles sounds reasonable. Nevertheless, 40+ seconds is very strange. I'll check it out soon.

[gizahNL]

13 seconds for

time /usr/jails/modules/get_bhyve_profiles --template
[{"name": "Linux UbuntuServer LTS 17.10.1", "profile": "ubuntuserver-x86-17.10", "type": "linux"}, {"name": "FreeBSD 11.1", "profile": "FreeBSD-x64-11.1", "type": "freebsd"}]
4.454u 9.097s 0:13.52 100.1%    5+168k 143+0io 0pf+0w

and again 13 seconds for

time /usr/jails/modules/get_bhyve_profiles --vm
[{"name": "Kali Linux 2017.03", "profile": "Kali-2017-amd64", "type": "linux"}, {"name": "Linux Fedora 27 server", "profile": "fedora-server-27-x86_64", "type": "linux"}, {"name": "Linux KUbuntu LTS 17.10.3", "profile": "kubuntu-x86-17.10", "type": "linux"}, {"name": "Linux UbuntuServer LTS 16.04.3", "profile": "ubuntuserver-x86-16.04", "type": "linux"}, {"name": "DragonFly 4.8.1", "profile": "x86-4", "type": "dflybsd"}, {"name": "OpenBSD 6.2-RELEASE x86-64", "profile": "x86-6", "type": "openbsd"}, {"name": "OPNSense 17.7", "profile": "OPNsense-17-RELEASE-amd64", "type": "freebsd"}, {"name": "pfSense 2.4.2", "profile": "pfSense-2-RELEASE-amd64", "type": "freebsd"}, {"name": "Linux UbuntuServer LTS 17.10.1", "profile": "ubuntuserver-x86-17.10", "type": "linux"}, {"name": "Linux KUbuntu LTS 16.04.3", "profile": "kubuntu-x86-16", "type": "linux"}, {"name": "HardenedBSD is a security-enhanced fork of FreeBSD", "profile": "HardenedBSD-x64-10-stable", "type": "freebsd"}, {"name": "FreeBSD 10.4", "profile": "FreeBSD-x64-10.4", "type": "freebsd"}, {"name": "Linux CentOS 7.4", "profile": "CentOS-7.4-x86_64", "type": "linux"}, {"name": "DragonFly 5.0.2", "profile": "x86-5", "type": "dflybsd"}, {"name": "HardenedBSD is a security-enhanced fork of FreeBSD", "profile": "HardenedBSD-x64-11-stable", "type": "freebsd"}, {"name": "Linux Debian 9.3.0", "profile": "Debian-x86-9", "type": "linux"}, {"name": "OPNSense 18.1", "profile": "OPNsense-18-RELEASE-amd64", "type": "freebsd"}, {"name": "Linux OpenSuse 42.3", "profile": "opensuse-x86-42", "type": "linux"}, {"name": "FreeBSD 11.0", "profile": "FreeBSD-x64-11.0", "type": "freebsd"}, {"name": "Linux CentOS 6.9", "profile": "CentOS-6-x86_64", "type": "linux"}, {"name": "FreeBSD 11.1", "profile": "FreeBSD-x64-11.1", "type": "freebsd"}, {"name": "Linux CentOS 7.4", "profile": "CentOS-7.3-x86_64", "type": "linux"}, {"name": "Linux Fedora 26 server", "profile": "fedora-server-26-x86_64", "type": "linux"}, {"name": "Linux Gentoo 20170118", "profile": "gentoo-amd64-2017", "type": "linux"}, {"name": "Microsoft Windows 10", "profile": "10_86x_64x", "type": "windows"}, {"name": "FreeBSD 10.3", "profile": "FreeBSD-x64-10.3", "type": "freebsd"}, {"name": "Linux Debian 8.10.0", "profile": "Debian-x86-8", "type": "linux"}]
4.562u 9.117s 0:13.74 99.4% 6+167k 347+0io 604pf+0w

The problem is also that they both get called more often than needed, my guess is perhaps even for every element, I've seen a couple of those processes run in paralel while I'm only opening 1 new page.

[olevole]

The implementation of the cache is currently blocked by the task of implementing the internal cron-service. I'm thinking about setting like :

[x] Update VM profiles periodically

in fact, it will perform git pull from https://github.com/cbsd/cbsd-vmprofiles ( source for profile list/settings ). In CBSD this happens through ~cbsd/etc/Makefile file: https://github.com/cbsd/cbsd/blob/master/etc/Makefile:

make -C ~cbsd/etc profiles-update

for this operation (or also as periodic), we can hang a cache creation

[olevole]

Fixed in BETA1, thanks!