HHVM segfault every 3 hours

[Lucky-Frost]

root@s3:~# lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.10
Release:        14.10
Codename:       utopic

/etc/hhvm/server.ini

; php options

pid = /var/run/hhvm/pid

; hhvm specific.

hhvm.server.port = 9000
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc

hhvm.http.default_timeout = 50
hhvm.server.connection_timeout_seconds = 50
hhvm.server.thread_count = 1000
hhvm.http.slow_query_threshold = 45000

root@s3:~# hhvm --version

HipHop VM 3.4.0 (rel)
Compiler: tags/HHVM-3.4.0-0-g817b3a07fc4e509ce15635dbc87778e5b3496663
Repo schema: 0e12aaa31fae66b5591f65603de50c9d62caafac
Extension API: 20140829

/var/log/hhvm/error.log

Failed to initialize central HHBC repository:\n  Failed to initialize schema in /var/run/hhvm/hhvm.hhbc: \n  Failed to open /var/www/.hhvm.hhbc: 14 - unable to open dat
Core dumped: Aborted
Stack trace in /tmp/stacktrace.20610.log

/tmp/stacktrace.20610.log

Host: s3
ProcessID: 20612
ThreadID: 7f04adfff700
ThreadPID: 20856
Name: unknown program
Type: Aborted
Runtime: hhvm
Version: tags/HHVM-3.4.0-0-g817b3a07fc4e509ce15635dbc87778e5b3496663
DebuggerCount: 0

Failed Assertion: /tmp/tmp.Z2v4DXDp7k/hphp/runtime/vm/repo.cpp:581: void HPHP::Repo::initCentral(): assertion `false' failed.
URL: /
Server: apidog.ru
false: Failed to initialize central HHBC repository:
  Failed to initialize schema in /var/run/hhvm/hhvm.hhbc:.
  Failed to open /var/www/.hhvm.hhbc: 14 - unable to open database file

Server_SERVER_NAME: s1.apidog.ru
ThreadType: Web Request

#0  gsignal at /lib/x86_64-linux-gnu/libc.so.6:0
#1  abort at /lib/x86_64-linux-gnu/libc.so.6:0
#2  HPHP::assert_fail(char const*, char const*, unsigned int, char const*) at /usr/bin/hhvm:0
#3  HPHP::Repo::initCentral() at /usr/bin/hhvm:0
#4  HPHP::Repo::Repo() at /usr/bin/hhvm:0
#5  HPHP::ThreadLocal<HPHP::Repo>::create() at /usr/bin/hhvm:0
#6  HPHP::Repo::get() at /usr/bin/hhvm:0
#7  HPHP::Unit::getLineNumber(int) const at /usr/bin/hhvm:0
#8  HPHP::createBacktrace(HPHP::BacktraceArgs const&) at /usr/bin/hhvm:0
#9  HPHP::ExtendedException::ExtendedException(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /usr/bin/hhvm:0
#10 HPHP::ExecutionContext::handleError(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool, HPHP::ExecutionContext::ErrorThrowMod
#11 HPHP::raise_typehint_error(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /usr/bin/hhvm:0
#12 HPHP::TypeConstraint::verifyFail(HPHP::Func const*, HPHP::TypedValue*, int) const at /usr/bin/hhvm:0
#13 void HPHP::ExecutionContext::dispatchImpl<true>() at /usr/bin/hhvm:0
#14 HPHP::ExecutionContext::dispatchBB() at /usr/bin/hhvm:0
#15 HPHP::jit::MCGenerator::enterTC(unsigned char*, void*) at /usr/bin/hhvm:0
#16 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, H
#17 HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) at /usr/bin/hhvm:0
#18 HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) at /usr/bin/hhvm:0
#19 HPHP::hphp_invoke(HPHP::ExecutionContext*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, HPHP::Array const&, HPHP::VRefParam
#20 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#21 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#22 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#23 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#24 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#25 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#26 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#27 clone at /lib/x86_64-linux-gnu/libc.so.6:0

PHP Stacktrace:

#0  mysqli_query(, SET NAMES utf8mb4) called at [/home/apidog/apidog.ru/public_html/base.php:154]
#1  APIdog::mysql(SELECT * FROM `backgrounds` WHERE `id`='5401' LIMIT 1, 1) called at [/home/apidog/apidog.ru/public_html/base.php:204]
#2  APIdog->getUserSettings() called at [/home/apidog/apidog.ru/public_html/index.php:159]

/var/log/apport.log

ERROR: apport (pid 23466) Sat Nov 22 10:54:44 2014: called for pid 20612, signal 6, core limit 0
ERROR: apport (pid 23466) Sat Nov 22 10:54:44 2014: executable: /usr/bin/hhvm (command line "/usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemon -vPidFile=/var/run/hhvm/pid")
ERROR: apport (pid 23466) Sat Nov 22 10:54:44 2014: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 23466) Sat Nov 22 10:54:44 2014: apport: report /var/crash/_usr_bin_hhvm.33.crash already exists and unseen, doing nothing to avoid disk usage DoS

Also I have some other stacktraces

[paulbiss]

Looks like the crash is occurring while generating a PHP backtrace, which requires loading line numbers from the bytecode repository. After the crash is the file /var/www/.hhvm.hhbc accessible by the user running the server?

[Lucky-Frost]

Yes, it's accessible by user after crash. The problem was missing in version 3.3.

[Lucky-Frost]

After a while there were other stacktraces hhvm/error.log

FastCGI protocol: received an invalid record
Core dumped: Segmentation fault
Stack trace in /tmp/stacktrace.7614.log
Finished singleJitRequest 1
Finished singleJitRequest 2
Finished singleJitRequest 3
Finished singleJitRequest 4
Finished singleJitRequest 5

stacktrace

Host: s1
ProcessID: 28445
ThreadID: 7f1477fff700
ThreadPID: 28448
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.4.0-0-g817b3a07fc4e509ce15635dbc87778e5b3496663
DebuggerCount: 0

ThreadType: Web Request
Server_SERVER_NAME: s1.apidog.ru
Server: apidog.ru
URL: /longpoll.php

# 0  folly::EventBase::runInEventBaseThread(std::function<void ()> const&) at /usr/bin/hhvm:0
# 1  HPHP::FastCGITransport::sendImpl(void const*, int, int, bool) at /usr/bin/hhvm:0
# 2  HPHP::Transport::sendRawInternal(void const*, int, int, bool, char const*) at /usr/bin/hhvm:0
# 3  HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
# 4  HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
# 5  HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
# 6  HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
# 7  HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
# 8  HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
# 9  start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
# 10 clone at /lib/x86_64-linux-gnu/libc.so.6:0

PHP Stacktrace:

and other

hhvm/error.log

Core dumped: Aborted
Stack trace in /tmp/stacktrace.1283.log
Finished singleJitRequest 1
Finished singleJitRequest 2
Finished singleJitRequest 3
Finished singleJitRequest 4
Finished singleJitRequest 5

/tmp/stacktrace

Host: s3
ProcessID: 1310
ThreadID: 7f5092bff700
ThreadPID: 1329
Name: unknown program
Type: Aborted
Runtime: hhvm
Version: tags/HHVM-3.4.0-0-g817b3a07fc4e509ce15635dbc87778e5b3496663
DebuggerCount: 0

# 0  gsignal at /lib/x86_64-linux-gnu/libc.so.6:0
# 1  abort at /lib/x86_64-linux-gnu/libc.so.6:0
# 2  folly::EventBase::runFunctionPtr(std::function<void ()>*) at /usr/bin/hhvm:0
# 3  folly::EventBase::FunctionRunner::messageAvailable(std::pair<void (*)(void*), void*>&&) at /usr/bin/hhvm:0
# 4  folly::NotificationQueue<std::pair<void (*)(void*), void*> >::Consumer::handlerReady(unsigned short) at /usr/bin/hhvm:0
# 5  event_base_loop at /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5:0
# 6  folly::EventBase::loopBody(int) at /usr/bin/hhvm:0
# 7  folly::EventBase::loopForever() at /usr/bin/hhvm:0
# 8  proxygen::WorkerThread::runLoop() at /usr/bin/hhvm:0
# 9  std::thread::_Impl<std::_Bind_simple<proxygen::WorkerThread::start()::{lambda()#1} ()> >::_M_run() at WorkerThread.cpp:0
# 10 std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) at /usr/lib/x86_64-linux
# 11 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
# 12 clone at /lib/x86_64-linux-gnu/libc.so.6:0

[apaloma]

We were hitting the same assert in HPHP::Repo::initCentral(). We were able to avoid the problem by increasing the maximum number of open file descriptors. What's the output of sysctl fs.file-max? Try increasing it. Also check ulimit -n

[Lucky-Frost]

su www-data -s /bin/sh
ulimit -n
262144

sysctl -p
fs.file-max = 262144

Version 3.3.1 does not have this problem, so, on my production servers i returned to it.

[Orvid]

This issue has been waiting for more info for more than 2 weeks. Closing for now, feel free to re-open it if you can provide more info.

https://github.com/facebook/hhvm/wiki/Human-Timeouts