search

facebook / hhvm

A virtual machine for executing programs written in Hack.
https://hhvm.com
Other
18.13k stars 2.99k forks source link

HHVM performance on Linode KVM #6197

Closed AdeFlit closed 8 years ago

AdeFlit commented 9 years ago

We've been happily using HHVM to host our app on Linode (Xen) for the past year. Recently, they've started a switch to KVM. Unfortunately we've witnessed absolutely terrible HHVM performance on Linode's KVM hosts, achieving just 5-10% of the expected throughput.

I originally detailed the problem here, along with some perf top output: https://forum.linode.com/viewtopic.php?p=67850#p67850

The problem does seem to be specific to HHVM.

On the advice of #hhvm, I've posted a couple of perf reports here: https://gist.github.com/AdeFlit/5f8ba5144ee0c1e996d5 https://gist.github.com/AdeFlit/3e16eb34204beb1ee5d4

Linode are aware of the issue - and working on it - but I can't imagine they know where to start. I was hoping you might look at the problem from the HHVM side, to see if anything jumps out as a possible cause.

sandeepone commented 9 years ago

We're running couple of Linodes. Recently we upgraded one Linode from XEN to KVM, Seems stable so far from couple of days. For us performance seems improved, will be testing more time.

We're running Centos 7.1 x64 HHVM 3.9.1 Webscalesql 5.6.24

AdeFlit commented 9 years ago

I've reproduced the issue with WordPress (although without such a radical performance hit), so it doesn't seem specific to our codebase. We're running Ubuntu 14.04.

MarkGavalda commented 9 years ago

We run hundreds of high traffic (millions of hits daily) websites on HHVM and KVM based cloud providers, Linode included, and haven't noticed anything like what you're describing.

AdeFlit commented 9 years ago

So you've migrated your Linodes to KVM? The issue doesn't manifest on any other KVM provider. I can build an identical server image on two other hosts and not experience this problem.

MarkGavalda commented 9 years ago

Yes we migrated almost all of our VMs to KVM. Does the issue persist after starting HHVM? I mean if you keep it running for 15+ minutes so it has time to warm up?

sandeepone commented 9 years ago

We've moved all our Linodes to KVM and see performance improved. I think your issue might be with the WP script or plugin.

AdeFlit commented 9 years ago

Mark: Yes, it persists.

Sandeep: there is no problem on a Linode Xen or DigitalOcean KVM. There is no problem with PHP5 on any platform. The issue isn't with our code per se. But something unique to HHVM (and our code) is triggering what looks like kernel scheduling issues on Linode's KVM environment.

I'd really appreciate it if one of the team could look at the perf reports.

fredemmott commented 8 years ago

The first perf looks reasonable :/

The second one shows issues with the APIC timer in the kernel, which is causing IO issues. It also shows this being made worse by re-jitting a load of code (which won't be Linode-specific)

Linode would need to investigate the APIC timer/IO issue, though you might be able to lessen the effect by setting hhvm.jit_pseudomain=0 in your INI setting - retranslations are often caused by badly-jitted pseudomains.

AdeFlit commented 8 years ago

@fredemmott Thanks for taking the time to look at those reports. I'll pass that on.