HHVM Support (experimental work in progress)

[thebennos]

We had a discussion over this, here https://www.drupal.org/node/2160975

Have seen that there are signifcant changes.

BOA Version is now 2.3.3. PHP-FPM is now used with the unix socket. Should be easier to ingrate HHVM now, Maybe a simple switch or a control file like it is now possible to add Solr 4 cores.

[omega8cc]

Yep, we should look into this.

[thebennos]

I tested HHVM on BOA a little bit. After install only some lines need to be adjusted . Normaly "fastcgi_pass unix:/var/run/o1.fpm.socket;" in the nginx_vhost_common.conf need to be replaced with with "fastcgi_pass 127.0.0.1:9000;"

Then HHVM works on BOA.

Installation is simple. If needed I can post a guide here.

[Pls]

Looks interesting. Can you compare speed difference bettween HHVM and standard PHP setup on BOA?

Would love to see guide for installing and testing HHVM.

[thebennos]

Installation is easy. Use a prebuilt package. Boa has full support for Wheezy. So https://github.com/facebook/hhvm/wiki/Prebuilt-Packages-on-Debian-7

HHVM starts automatically and you can access it as mentioned above via "fastcgi_pass 127.0.0.1:9000;"

Look in the nginx conf file and replace existing fastcgi lines.

Performance: I do not tested it really, but you can find good testing with google. Here an example: https://kinsta.com/blog/real-world-wordpress-benchmarks-with-php5-5-php5-6-php-ng-and-hhvm/

Each drupal installation has it's own performance profile, so you need to test it on your own.

[omega8cc]

Using TCP port method is rather too limiting in the BOA context, because it will restrict it to use only single Aegir instance per system. I didn't investigate it yet, but I think it previously required socket based listening mode, and this could make it more useful in the BOA context.

[omega8cc]

It does support sockets, so it should be easy to configure it properly using BOA standard configuration: https://github.com/facebook/hhvm/wiki/FastCGI

[omega8cc]

You just start it with:

hhvm --mode daemon -vServer.Type=fastcgi -vServer.FileSocket=/var/run/o1.fpm.socket

[and1truong]

A nice benchmark http://web-rocker.de/2013/12/drupal-7-hhvm-vs-php-55-zend-opcache

[Pls]

Great presentation about HHVM from Josh Koenig - https://www.youtube.com/watch?v=hl-vGkz74io

[omega8cc]

It works!

[Pls]

Great addition, BOA never stops moving and experimenting, which is awesome!

As I understand after enabling HHVM it switches all sites on octopus to use it (instead of standard PHP), right? Do we know any caveats/bugs HHVM has right now on D7/8? Do you plan to introduce it on omega8.cc for potential switch (just curious)? I am really interested in speed improvoments it achieves, so can't wait to test it out. Let's test!

[omega8cc]

As I understand after enabling HHVM it switches all sites on octopus to use it (instead of standard PHP), right?

Correct, it is Octopus instance-wide change, because it modifies instance specific Nginx configuration include file shared by all sites on the instance, to point them to HHVM socket.

Do we know any caveats/bugs HHVM has right now on D7/8?

It works also for Pressflow 6, because Aegir control panel works just fine. The HHVM is a work in progress, so they keep fixing bugs and compatibility issues pretty quickly. Please do note that it is a custom PHP engine, which doesn't use extensions normally added in BOA for PHP-FPM.

Note also that currently it ignores some security restrictions available in PHP-FPM, like open_basedir and disable_functions, so it is not recommended to use in production yet.

Testing is welcome!

Do you plan to introduce it on omega8.cc for potential switch (just curious)?

We plan to make it available as an option on our high-end Power Engines, where you can use multiple Octopus instances, so it is safe to use both PHP-FPM for production sites and HHVM for testing, at least until it is confirmed that it doesn't cause any issues for the site tested.

Happy New Year!

[omega8cc]

Re-opening, since there are still some known issues to fix.

[omega8cc]

When new site is installed, and the files directories are not yet magically fixed by BOA, so they still are owned by Aegir backend user, typically o1, while HHVM is configured to run as o1.ftp, you will see a broken layout due to missing css and js, along with bunch of errors like:

The specified file temporary://fileGFhQr7 could not be copied, because the destination directory is not properly configured. This may be caused by a problem with file or directory permissions. More information is available in the system log.
The specified file temporary://fileQyzrEf could not be copied, because the destination directory is not properly configured. This may be caused by a problem with file or directory permissions. More information is available in the system log.
The specified file temporary://fileSvd6Rn could not be copied, because the destination directory is not properly configured. This may be caused by a problem with file or directory permissions. More information is available in the system log.

Not that this happens because you can configure the user the HHVM should run as, but you can't configure the group to use, like it is possible in PHP-FPM, so while o1.ftp is a member of www-data, when running with its default group (users), it can't write anything in the files directory, since it is initially created with permissions like this:

drwxrwsr-x 17 o1 www-data 4.0K Jan  3 00:07 ./
drwxr-xr-x  7 o1 users    4.0K Jan  3 00:07 ../
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 advagg_css/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 advagg_js/
drwxrwsr-x  5 o1 www-data 4.0K Jan  3 00:07 civicrm/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 css/
drwxrwsr-x  3 o1 www-data 4.0K Jan  3 00:07 ctools/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 deployment/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 feeds/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 imagecache/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 images/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 js/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 locations/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 pictures/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 styles/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 tmp/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 00:07 xmlsitemap/

[omega8cc]

That said, it seems to be D7 specific problem, since Pressflow 6 works just fine and these permissions are sufficient to write css files:

drwxrwsr-x  2 o1     www-data 4.0K Jan  3 01:04 ./
drwxrwsr-x 17 o1     www-data 4.0K Jan  1 20:48 ../
-rw-rw-r--  1 o1.ftp www-data  37K Jan  3 01:03 css_2fbc6344ba25c88afc9388c92befe58b.css
-rw-rw-r--  1 o1.ftp www-data  38K Jan  3 01:04 css_565ce5430fbd0dd6f9e9eb2a6a67b38c.css

[omega8cc]

Could this be related to the fact that D7 apparently creates temp files with very restricted permissions?

drwxrwsr-x 2 o1     www-data 4.0K Jan  3 01:08 ./
drwxrwsr-x 5 o1     www-data 4.0K Jan  3 00:54 ../
-rw------- 1 o1.ftp www-data 3.0K Jan  3 01:08 fileaH28Zn
-rw------- 1 o1.ftp www-data  94K Jan  3 01:08 fileCtDEhn
-rw------- 1 o1.ftp www-data  489 Jan  3 01:08 fileEooEVR
-rw------- 1 o1.ftp www-data  18K Jan  3 01:08 fileijTGxQ
-rw------- 1 o1.ftp www-data 4.4K Jan  3 01:08 fileKBUieR
-rw------- 1 o1.ftp www-data  28K Jan  3 01:08 fileMcVVzm
-rw------- 1 o1.ftp www-data  48K Jan  3 01:08 fileqTVQDS
-rw------- 1 o1.ftp www-data 2.4K Jan  3 01:08 files2nWSl
-rw------- 1 o1.ftp www-data 7.5K Jan  3 01:08 fileSTAAcl

[omega8cc]

Actually, HHVM seems to respect RUN_AS_GROUP="www-data", because its socket has expected user/group:

-rw-r--r-- 1 o1.ftp www-data 5 Jan  3 01:30 /var/run/hhvm/o1/hhvm.pid

[omega8cc]

Another tests confirms that HHVM runs with default group users and not www-data, because it can write css and js files after switching the group to users like shown below:

drwxrwsr-x 18 o1 www-data 4.0K Jan  3 02:23 ./
drwxr-xr-x  8 o1 users    4.0K Jan  3 02:23 ../
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 advagg_css/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 advagg_js/
drwxrwsr-x  5 o1 www-data 4.0K Jan  3 02:23 civicrm/
drwxrwsr-x  2 o1 users    4.0K Jan  3 02:23 css/
drwxrwsr-x  3 o1 www-data 4.0K Jan  3 02:23 ctools/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 deployment/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 feeds/
-r--r--r--  1 o1 www-data  476 Jan  3 02:23 .htaccess
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 imagecache/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 images/
drwxrwsr-x  2 o1 users    4.0K Jan  3 02:23 js/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 locations/
drwxrwxrwx  4 o1 www-data 4.0K Jan  3 02:23 php/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 pictures/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 styles/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 tmp/
drwxrwsr-x  2 o1 www-data 4.0K Jan  3 02:23 xmlsitemap/

This means that it ignores another layer of protection built in BOA, which allows to limit web server privileges, so it can access only whitelisted binaries etc. HHVM, in contrast to PHP-FPM, ignores the group restrictions, ignores open_basedir and disable_functions, so it theoretically allows to launch very easy attacks, if the attacker will manage to upload something like web shell.

[omega8cc]

Related issues:

https://github.com/facebook/hhvm/issues/2946 https://github.com/facebook/hhvm/issues/2745

[macmladen]

This is very interesting, however from your comments seems pretty far from production strength. There are obvious speed gains although I haven't seen any comments (around net) about memory consumption and I've read about some minor problems.

But it looks promising once all things are worked out.

Is it possible to have HHVM as an option, just like other version of PHP, e.g 5.4, 5.5 and HHVM so it can be chosen on per site / platform basis?

[omega8cc]

Is it possible to have HHVM as an option, just like other version of PHP, e.g 5.4, 5.5 and HHVM

It is an option, but not exactly like other PHP related options, because you can't control HHVM version. It also requires Nginx config modification on the fly. It also requires custom system user created on the fly. That is why it is managed with control file and not just a variable in .cnf, so it is not possible to enable it on the Octopus instance install or upgrade. You still need standard PHP installed, so it could be used in the Aegir and Drush backend, and that is why you can't specify HHVM as an option instead of PHP version.

Furthermore, currently the octopus upgrade will automatically switch the instance back to available PHP-FPM version, so the control file doesn't make HHVM permanent per instance (yet).

Note that to switch to HHVM, once it is installed, you only need to create an empty ~/static/control/hhvm.info control file, while to switch the Octopus instance back to PHP-FPM, you need to delete ~/static/control/hhvm.info and create proper, non-empty ~/static/control/fpm.info control file with PHP version listed: 5.5 or 5.4 or 5.3

so it can be chosen on per site / platform basis?

No, you can't choose PHP-FPM nor PHP-CLI version per site nor per platform, only per Octopus instance. That is exactly how it works with HHVM and it has been explained in both commit comment and my earlier comments above.=

[omega8cc]

Some docs just added: https://github.com/omega8cc/boa/blob/master/docs/HHVM.txt

[omega8cc]

Interesting benchmarks in the following article http://hhvm.com/blog/9293/lockdown-results-and-hhvm-performance /via travisc: https://www.drupal.org/node/2160975#comment-10494298