search

facebook / hhvm

A virtual machine for executing programs written in Hack.
https://hhvm.com
Other
18.13k stars 2.99k forks source link

HHVM 3.2.0 debian fastcgi assertion failure - Core dumped: aborted #3422

Closed StrathCole closed 9 years ago

StrathCole commented 10 years ago

First one:

Host: server
ProcessID: 50929
ThreadID: 7f13d7bff700
ThreadPID: 51510
Name: /usr/bin/hhvm
Type: Aborted
Runtime: hhvm
Version: tags/HHVM-3.2.0-0-g01228273b8cf709aacbd3df1c51b1e690ecebac8
DebuggerCount: 0

Failed Assertion: /tmp/tmp.61ZngWqN7R/hphp/runtime/vm/jit/translator.cpp:1958: HPHP::JIT::Translator::TranslateResult HPHP::JIT::Translator::translateRegion(const HPHP::JIT::RegionDesc&, bool, HPHP::JIT::Translator::RegionBlacklist&, HPHP::JIT::TransFlags): assertion `0' failed.
URL: /detail/2239/.html
Server: www.******.de
0: data block = cold
message: Attempted to emit 5 byte(s) into a 74 byte DataBlock with 0 bytes available. This almost certainly means the TC is full. If this is the case, increasing Eval.JitASize, Eval.JitAColdSize, Eval.JitAFrozenSize and Eval.JitGlobalDataSize in the configuration file when running this script or application should fix this problem.

Server_SERVER_NAME: www.******.de
ThreadType: Web Request

#0  raise at /lib/x86_64-linux-gnu/libc.so.6:0
#1  abort at /lib/x86_64-linux-gnu/libc.so.6:0
#2  HPHP::assert_fail(char const*, char const*, unsigned int, char const*) at /usr/bin/hhvm:0
#3  HPHP::JIT::Translator::translateRegion(HPHP::JIT::RegionDesc const&, bool, HPHP::hphp_hash_set<HPHP::JIT::ProfSrcKey, HPHP::JIT::ProfSrcKey::Hasher, std::equal_to<HPHP::JIT::ProfSrcKey> >&, HPHP::JIT::TransFlags) at /usr/bin/hhvm:0
#4  HPHP::JIT::MCGenerator::translateWork(HPHP::JIT::TranslArgs const&) at /usr/bin/hhvm:0
#5  HPHP::JIT::MCGenerator::translate(HPHP::JIT::TranslArgs const&) at /usr/bin/hhvm:0
#6  HPHP::JIT::MCGenerator::retranslate(HPHP::JIT::TranslArgs const&) at /usr/bin/hhvm:0
#7  HPHP::JIT::MCGenerator::createTranslation(HPHP::JIT::TranslArgs const&) at /usr/bin/hhvm:0
#8  HPHP::JIT::MCGenerator::bindJmp(unsigned char*, HPHP::SrcKey, HPHP::JIT::ServiceRequest, HPHP::JIT::TransFlags, bool&) at /usr/bin/hhvm:0
#9  HPHP::JIT::MCGenerator::handleServiceRequest(HPHP::JIT::TReqInfo&, unsigned char*&, HPHP::SrcKey&) at /usr/bin/hhvm:0
#10 HPHP::JIT::MCGenerator::enterTC(unsigned char*, void*) at /usr/bin/hhvm:0
#11 HPHP::ExecutionContext::enterVM(HPHP::ActRec*, HPHP::ExecutionContext::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) at /usr/bin/hhvm:0
#12 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) at /usr/bin/hhvm:0
#13 HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) at /usr/bin/hhvm:0
#14 HPHP::invoke_file(HPHP::String const&, bool, char const*) at builtin-functions.cpp:0
#15 HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) at /usr/bin/hhvm:0
#16 HPHP::hphp_invoke(HPHP::ExecutionContext*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool, bool, bool) at /usr/bin/hhvm:0
#17 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#18 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#19 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#20 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#21 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#22 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#23 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#24 __clone at /lib/x86_64-linux-gnu/libc.so.6:0

PHP Stacktrace:

#0  include() called at [/var/www/clients/****/web/3rd-party/smarty/sysplugins/smarty_internal_templatebase.php:265]
#1  Smarty_Internal_TemplateBase->fetch(emwm_detail.tpl, emwm_detail|2239, , Object of class Smarty could not be converted to string, 1) called at [/var/www/clients/****/web/3rd-party/smarty/sysplugins/smarty_internal_templatebase.php:394]
#2  Smarty_Internal_TemplateBase->display(emwm_detail.tpl, emwm_detail|2239) called at [/var/www/clients****/web/modules/emwm_detail.php:42]
#3  include(/var/www/clients/****/web/modules/emwm_detail.php) called at [/var/www/clients/****/index.php:55]

Second one:

Host: server
ProcessID: 62287
ThreadID: 7fb9dd3ff700
ThreadPID: 63244
Name: /usr/bin/hhvm
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.2.0-0-g01228273b8cf709aacbd3df1c51b1e690ecebac8
DebuggerCount: 0

ThreadType: Web Request
Server_SERVER_NAME: www.****.de
Server: www.****.de
URL: /fussball/auswaertstabelle.html

#0  HPHP::JIT::FixupMap::fixup(HPHP::ExecutionContext*) const at /usr/bin/hhvm:0
#1  HPHP::JIT::MCGenerator::syncWork() at /usr/bin/hhvm:0
#2  HPHP::ExecutionContext::handleError(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool, HPHP::ExecutionContext::ErrorThrowMode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) at /usr/bin/hhvm:0
#3  HPHP::raise_error(char const*, ...) at /usr/bin/hhvm:0
#4  HPHP::StringData::reserve(unsigned long) at /usr/bin/hhvm:0
#5  HPHP::StringBuffer::growBy(int) at /usr/bin/hhvm:0
#6  HPHP::StringBuffer::appendHelper(char const*, int) at /usr/bin/hhvm:0
#7  HPHP::ExecutionContext::write(char const*, int) at /usr/bin/hhvm:0
#8  HPHP::print_string(HPHP::StringData*) at /usr/bin/hhvm:0
#9  HPHP::JIT::X64::BackEnd::enterTCHelper(unsigned char*, HPHP::JIT::TReqInfo&) at /usr/bin/hhvm:0
#10 HPHP::JIT::MCGenerator::enterTC(unsigned char*, void*) at /usr/bin/hhvm:0
#11 HPHP::ExecutionContext::enterVM(HPHP::ActRec*, HPHP::ExecutionContext::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) at /usr/bin/hhvm:0
#12 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) at /usr/bin/hhvm:0
#13 HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) at /usr/bin/hhvm:0
#14 HPHP::invoke_file(HPHP::String const&, bool, char const*) at builtin-functions.cpp:0
#15 HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) at /usr/bin/hhvm:0
#16 HPHP::hphp_invoke(HPHP::ExecutionContext*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool, bool, bool) at /usr/bin/hhvm:0
#17 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#18 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#19 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#20 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#21 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#22 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#23 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#24 __clone at /lib/x86_64-linux-gnu/libc.so.6:0

In the second case there is no PHP trace, only the above.

First web call was done by Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Second call was triggered by Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Both resulted in a 404 error (the page called does not exist). First call took quite long, to what I see from access logs (20 to 30 seconds).

StrathCole commented 10 years ago

Important info to add: The 404 error is a php page, not a static default 404 error page. Running with apache 2.2 and fastCGI.

StrathCole commented 10 years ago

Okay, installed the hhvm-dbg package and now I have this in error log:

/tmp/tmp.Uu8vpDr7Er/hphp/runtime/server/fastcgi/fastcgi-session.cpp:255: virtual size_t HPHP::FastCGISession::onIngress(const folly::IOBuf*): assertion `m_keepConn' failed.

and this stacktrace

Host: server
ProcessID: 18803
ThreadID: 7f39c3fff700
ThreadPID: 18820
Name: /usr/bin/hhvm
Type: Aborted
Runtime: hhvm
Version: tags/HHVM-3.2.0-0-g01228273b8cf709aacbd3df1c51b1e690ecebac8
DebuggerCount: 0

Failed Assertion: /tmp/tmp.Uu8vpDr7Er/hphp/runtime/server/fastcgi/fastcgi-session.cpp:255: virtual size_t HPHP::FastCGISession::onIngress(const folly::IOBuf*): assertion `m_keepConn' failed.

# 0  raise at /lib/x86_64-linux-gnu/libc.so.6:0
# 1  abort at /lib/x86_64-linux-gnu/libc.so.6:0
# 2  HPHP::assert_fail(char const*, char const*, unsigned int, char const*) at /tmp/tmp.Uu8vpDr7Er/hphp/util/assertions.cpp:33
# 3  HPHP::FastCGISession::onIngress(folly::IOBuf const*) at /tmp/tmp.Uu8vpDr7Er/hphp/runtime/server/fastcgi/fastcgi-session.cpp:255
# 4  HPHP::FastCGIConnection::readDataAvailable(unsigned long) at /tmp/tmp.Uu8vpDr7Er/hphp/runtime/server/fastcgi/fastcgi-server.cpp:116
# 5  apache::thrift::async::TAsyncSocket::handleRead() at /tmp/tmp.Uu8vpDr7Er/third-party/thrift/lib/cpp/async/TAsyncSocket.cpp:1272
# 6  apache::thrift::async::TAsyncSocket::ioReady(unsigned short) at /tmp/tmp.Uu8vpDr7Er/third-party/thrift/lib/cpp/async/TAsyncSocket.cpp:1168
# 7  apache::thrift::async::TAsyncSocket::IoHandler::handlerReady(unsigned short) at /tmp/tmp.Uu8vpDr7Er/third-party/thrift/lib/cpp/async/TAsyncSocket.h:510
# 8  apache::thrift::async::TEventHandler::libeventCallback(int, short, void*) at /usr/bin/hhvm:0
# 9  event_base_loop at /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5:0
# 10 apache::thrift::async::TEventBase::loop() at /tmp/tmp.Uu8vpDr7Er/third-party/thrift/lib/cpp/async/TEventBase.cpp:212
# 11 apache::thrift::async::TEventBase::loopForever() at /tmp/tmp.Uu8vpDr7Er/third-party/thrift/lib/cpp/async/TEventBase.cpp:276
# 12 facebook::proxygen::WorkerThread::runLoop() at /tmp/tmp.Uu8vpDr7Er/third-party/ti/proxygen/lib/workers/WorkerThread.cpp:128
# 13 std::thread::_Impl<std::_Bind_simple<facebook::proxygen::WorkerThread::start()::{lambda()#1} ()> >::_M_run() at /tmp/tmp.Uu8vpDr7Er/third-party/ti/proxygen/lib/workers/WorkerThread.cpp:34
# 14 std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) at /usr/lib/x86_64-linux-gnu/libstdc++.so.6:0
# 15 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
# 16 __clone at /lib/x86_64-linux-gnu/libc.so.6:0
kristapsk commented 10 years ago

Debug builds are broken, see #3084.

StrathCole commented 10 years ago

Okay, thank you. But first comments are with normal build, no debug.

StrathCole commented 10 years ago

Now I installed hhvm-nightly to check if this changes something, but:

Host: server
ProcessID: 50865
ThreadID: 7f37cd3ff700
ThreadPID: 51027
Name: /usr/bin/hhvm
Type: Segmentation fault
Runtime: hhvm
Version: heads/master-0-g8bc650d40490a6d4fce0e3464db1fdf054fcc6ea
DebuggerCount: 0

ThreadType: Web Request
Server_SERVER_NAME: www.****.de
Server: www.****.de
URL: /fussball/*********.html

# 0  HPHP::JIT::FixupMap::fixup(HPHP::ExecutionContext*) const at /usr/bin/hhvm:0
# 1  HPHP::JIT::MCGenerator::syncWork() at /usr/bin/hhvm:0
# 2  HPHP::ExecutionContext::handleError(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool, HPHP::ExecutionContext::ErrorThrowMode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) at /usr/bin/hhvm:0
# 3  HPHP::raise_error(char const*, ...) at /usr/bin/hhvm:0
# 4  HPHP::StringData::reserve(unsigned long) at /usr/bin/hhvm:0
# 5  HPHP::StringBuffer::growBy(int) at /usr/bin/hhvm:0
# 6  HPHP::StringBuffer::appendHelper(char const*, int) at /usr/bin/hhvm:0
# 7  HPHP::ExecutionContext::write(char const*, int) at /usr/bin/hhvm:0
# 8  HPHP::print_string(HPHP::StringData*) at /usr/bin/hhvm:0
# 9  HPHP::JIT::X64::BackEnd::enterTCHelper(unsigned char*, HPHP::JIT::TReqInfo&) at /usr/bin/hhvm:0
# 10 HPHP::JIT::MCGenerator::enterTC(unsigned char*, void*) at /usr/bin/hhvm:0
# 11 HPHP::ExecutionContext::enterVM(HPHP::ActRec*, HPHP::ExecutionContext::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) at /usr/bin/hhvm:0
# 12 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) at /usr/bin/hhvm:0
# 13 HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) at /usr/bin/hhvm:0
# 14 HPHP::invoke_file(HPHP::String const&, bool, char const*) at builtin-functions.cpp:0
# 15 HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) at /usr/bin/hhvm:0
# 16 HPHP::hphp_invoke(HPHP::ExecutionContext*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool, bool, bool) at /usr/bin/hhvm:0
# 17 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
# 18 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
# 19 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
# 20 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
# 21 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
# 22 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
# 23 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
# 24 __clone at /lib/x86_64-linux-gnu/libc.so.6:0

Again this was a 404 (handled by a php script) called by a bot, this time it was Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/). To clarify: The 404 is not generated by apache. The web call is redirected to a php script that checks if the virtual url exists and then gives a header('HTTP/1.0 404 NOT FOUND').

paulbiss commented 10 years ago

Your most recent crash looks like it's occurring in the process of attempting to grow a string buffer past its maximum size (hence the call to raise_error). It would also appear that the string in question is for buffering standard output.

Beyond those observations there isn't a lot to go on here, do you have an isolated example that consistently reproduces this crash available? Is there anything else going on besides the call to header()?

StrathCole commented 10 years ago

@paulbiss sadly there is no consistent way to reproduce it. I tried calling the same url multiple times but it did not crash. Then I left the site alone for a while and it crashed after some time. This crash reports are from a custom page using smarty template system. I had crashes like this on a default wordpress installation (it was 3.9 I think), too - at random times. I know that it is hard to track down an error that occurs randomly, but there is no more info. Are the debug builds working again? I could try using those to generate a better crash report then.

paulbiss commented 10 years ago

@StrathCole they should be working. Are you also still seeing the Translator::translateRegion() assert and the FastCGISession assert?

StrathCole commented 10 years ago

@paulbiss I will try it - had no page active, because they are live and I could not risk those out-times. I will get back to you, as soon as I have the next crash, after I activate the hhvm mode.

StrathCole commented 10 years ago

The package seems to have missing dependencies (debian wheezy): libmemcachedutil2 and libtcmalloc-minimal4 Installed them manually and the hhvm-nightly-dbg deamon starts now.

StrathCole commented 10 years ago

Okay, crash occured directly after starting the server and calling some pages:

Host: server
ProcessID: 1777
ThreadID: 7f886b587700
ThreadPID: 1923
Name: /usr/bin/hhvm
Type: Aborted
Runtime: hhvm
Version: heads/master-0-g86964dc199961ddd5dc903df9decaafa18f78368
DebuggerCount: 0

Failed Assertion: /tmp/tmp.DbgDmU7GTl/hphp/runtime/server/fastcgi/fastcgi-session.cpp:261: size_t HPHP::FastCGISession::onIngress(const folly::IOBuf*): assertion `m_keepConn' failed.

# 0  raise at /lib/x86_64-linux-gnu/libc.so.6:0
# 1  abort at /lib/x86_64-linux-gnu/libc.so.6:0
# 2  HPHP::assert_fail(char const*, char const*, unsigned int, char const*) at /tmp/tmp.DbgDmU7GTl/hphp/util/assertions.cpp:33
# 3  HPHP::FastCGISession::onIngress(folly::IOBuf const*) at /tmp/tmp.DbgDmU7GTl/hphp/runtime/server/fastcgi/fastcgi-session.cpp:263
# 4  HPHP::FastCGIConnection::readDataAvailable(unsigned long) at /tmp/tmp.DbgDmU7GTl/hphp/runtime/server/fastcgi/fastcgi-server.cpp:115
# 5  apache::thrift::async::TAsyncSocket::handleRead() at /tmp/tmp.DbgDmU7GTl/third-party/thrift/lib/cpp/async/TAsyncSocket.cpp:1272
# 6  apache::thrift::async::TAsyncSocket::ioReady(unsigned short) at /tmp/tmp.DbgDmU7GTl/third-party/thrift/lib/cpp/async/TAsyncSocket.cpp:1168
# 7  apache::thrift::async::TAsyncSocket::IoHandler::handlerReady(unsigned short) at /tmp/tmp.DbgDmU7GTl/third-party/thrift/lib/cpp/async/TAsyncSocket.h:510
# 8  apache::thrift::async::TEventHandler::libeventCallback(int, short, void*) at /tmp/tmp.DbgDmU7GTl/third-party/thrift/lib/cpp/async/TEventHandler.cpp:153
# 9  event_base_loop at /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5:0
# 10 apache::thrift::async::TEventBase::loop() at /tmp/tmp.DbgDmU7GTl/third-party/thrift/lib/cpp/async/TEventBase.cpp:212
# 11 apache::thrift::async::TEventBase::loopForever() at /tmp/tmp.DbgDmU7GTl/third-party/thrift/lib/cpp/async/TEventBase.cpp:276
# 12 proxygen::WorkerThread::runLoop() at /tmp/tmp.DbgDmU7GTl/third-party/proxygen/lib/workers/WorkerThread.cpp:128
# 13 proxygen::WorkerThread::start()::{lambda()#1}::operator()() const at /tmp/tmp.DbgDmU7GTl/third-party/proxygen/lib/workers/WorkerThread.cpp:34
# 14 void std::_Bind_simple<proxygen::WorkerThread::start()::{lambda()#1} ()>::_M_invoke<>(std::_Index_tuple<>) at /usr/include/c++/4.8/functional:1733
# 15 std::_Bind_simple<proxygen::WorkerThread::start()::{lambda()#1} ()>::operator()() at /usr/include/c++/4.8/functional:1720
# 16 std::thread::_Impl<std::_Bind_simple<proxygen::WorkerThread::start()::{lambda()#1} ()> >::_M_run() at /usr/include/c++/4.8/thread:115
# 17 std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) at /usr/lib/x86_64-linux-gnu/libstdc++.so.6:0
# 18 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
# 19 __clone at /lib/x86_64-linux-gnu/libc.so.6:0

Are debug builds really working again? This is the same stacktrace I got when I last tried the debug version when I was told they are broken.

StrathCole commented 10 years ago

@paulbiss Okay, it crashed again (hhvm-nightly, non-debug).

Host: server
ProcessID: 3981
ThreadID: 1e29f700
ThreadPID: 6073
Name: /usr/bin/hhvm
Type: Aborted
Runtime: hhvm
Version: heads/master-0-g86964dc199961ddd5dc903df9decaafa18f78368
DebuggerCount: 0

Failed Assertion: /tmp/tmp.CcOXc7OhJ6/hphp/runtime/vm/jit/mc-generator.cpp:1339: unsigned char* HPHP::jit::MCGenerator::getFreeStub(HPHP::CodeBlock&, HPHP::jit::CodeGenFixups*): assertion `m_freeStubs.m_list == nullptr || code.isValidCodeAddress(TCA(m_freeStubs.m_list))' failed.
URL: /index.php?section=xxxx
Server: www.xxxxxx.de
Server_SERVER_NAME: www.xxxxxx.de
ThreadType: Web Request

# 0  raise at /lib/x86_64-linux-gnu/libc.so.6:0
# 1  abort at /lib/x86_64-linux-gnu/libc.so.6:0
# 2  HPHP::assert_fail(char const*, char const*, unsigned int, char const*) at /usr/bin/hhvm:0
# 3  HPHP::jit::MCGenerator::getFreeStub(HPHP::DataBlock&, HPHP::jit::CodeGenFixups*) at /usr/bin/hhvm:0
# 4  HPHP::jit::x64::(anonymous namespace)::emitBindJ(HPHP::DataBlock&, HPHP::DataBlock&, HPHP::jit::ConditionCode, HPHP::SrcKey, HPHP::jit::ServiceRequest, HPHP::jit::TransFlags) at service-requests-x64.cpp:0
# 5  HPHP::jit::x64::(anonymous namespace)::Vgen::emit(std::vector<HPHP::jit::Vlabel, std::allocator<HPHP::jit::Vlabel> >&) at vasm-x64.cpp:0                           
# 6  HPHP::jit::x64::Vasm::finish(HPHP::jit::Abi const&, bool) at /usr/bin/hhvm:0                                                                                       
# 7  HPHP::jit::x64::BackEnd::genCodeImpl(HPHP::jit::IRUnit&, HPHP::jit::AsmInfo*) at /usr/bin/hhvm:0                                                                   
# 8  HPHP::jit::genCode(HPHP::jit::IRUnit&) at /usr/bin/hhvm:0                                                                                                          
# 9  HPHP::jit::MCGenerator::traceCodeGen() at /usr/bin/hhvm:0                                                                                                          
# 10 HPHP::jit::Translator::translateRegion(HPHP::jit::RegionDesc const&, bool, HPHP::hphp_hash_set<HPHP::jit::ProfSrcKey, HPHP::jit::ProfSrcKey::Hasher, std::equal_to<HPHP::jit::ProfSrcKey> >&, HPHP::jit::TransFlags) at /usr/bin/hhvm:0                                                                                                    
# 11 HPHP::jit::MCGenerator::translateWork(HPHP::jit::TranslArgs const&) at /usr/bin/hhvm:0                                                                             
# 12 HPHP::jit::MCGenerator::translate(HPHP::jit::TranslArgs const&) at /usr/bin/hhvm:0                                                                                 
# 13 HPHP::jit::MCGenerator::retranslate(HPHP::jit::TranslArgs const&) at /usr/bin/hhvm:0                                                                               
# 14 HPHP::jit::MCGenerator::createTranslation(HPHP::jit::TranslArgs const&) at /usr/bin/hhvm:0
# 15 HPHP::jit::MCGenerator::handleServiceRequest(HPHP::jit::TReqInfo&, unsigned char*&, HPHP::SrcKey&) at /usr/bin/hhvm:0
# 16 HPHP::jit::MCGenerator::enterTC(unsigned char*, void*) at /usr/bin/hhvm:0
# 17 HPHP::ExecutionContext::enterVM(HPHP::ActRec*, HPHP::ExecutionContext::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) at /usr/bin/hhvm:0
# 18 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) at /usr/bin/hhvm:0
# 19 HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) at /usr/bin/hhvm:0
# 20 HPHP::invoke_file(HPHP::String const&, bool, char const*) at builtin-functions.cpp:0
# 21 HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) at /usr/bin/hhvm:0
# 22 HPHP::hphp_invoke(HPHP::ExecutionContext*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool, bool, bool) at /usr/bin/hhvm:0
# 23 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
# 24 HPHP::HttpRequestHandler::handleRequestImpl(HPHP::Transport*) at /usr/bin/hhvm:0
# 25 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
# 26 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
# 27 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
# 28 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
# 29 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
# 30 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
# 31 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
# 32 __clone at /lib/x86_64-linux-gnu/libc.so.6:0

PHP Stacktrace:

#0  include() called at [/var/www/xxxxx/smarty/sysplugins/smarty_internal_templatebase.php:265]
#1  Smarty_Internal_TemplateBase->fetch(vereine_kader.tpl, club|1675|1, , Object of class Smarty could not be converted to string, 1) called at [/var/www/xxxxxx/smarty/sysplugins/smarty_internal_templatebase.php:394]
#2  Smarty_Internal_TemplateBase->display(vereine_kader.tpl, club|1675|1) called at [/var/www/xxxxxx/vereine_kader.php:19]
#3  include(/var/www/xxxxxx/vereine_kader.php) called at [/var/www/xxxxxx/index.php:55]

The corresponding lines from the templatebase.php from smarty are:

               try {
                    ob_start();
                    eval("?>" . $output);
                    $_output = ob_get_clean();
                }

It crashes at the "eval" call, so it might be somehow related to the output buffering you mentioned.

paulbiss commented 10 years ago

They should be working. I have a fix in for this bug but it hasn't been pushed to github yet (#3296).

StrathCole commented 10 years ago

@paulbiss okay, I'll wait for your patch being in nightly and then I'll try running again.

paulbiss commented 10 years ago

The fix was for the onIngress() crash by the way, the other I still suspect has something to do with output buffering. (6ff4bfc75242b2abec6ba7f0214a3663ac0dc563)

paulbiss commented 10 years ago

@StrathCole I've got a repro of the buffer overflow in HPHP::JIT::FixupMap::fixup, it's definitely from calling ob_start() and then writing too many bytes into the buffer (it should be raising a fatal not crashing, fix incoming).

The most recent trace you have (as well as the first one you posted) look like they're related to the translation cache filling up, I would try adjusting the runtime options that control the size of the JIT.

StrathCole commented 10 years ago

@paulbiss Can't it be, that the first/last traces are caused by the same (or a similar) bug? They both occur at the mentioned point in the script where the template is written after ob_start() using eval().

 try {
   ob_start();
   eval("?>" . $output);  /** this is where the crash occurs **/
   $_output = ob_get_clean();
}

If not: how can I check? I am not really familiar with adjusting JIT settings.

kowach commented 9 years ago

I was trying to switch to hhvm, but I'm getting same similar error every hour. 

Host: ***.net
ProcessID: 17784
ThreadID: 7fe459fff700
ThreadPID: 17794
Name: unknown program
Type: Aborted
Runtime: hhvm
Version: heads/master-0-g96dec459bb84f606df2fe15c8c82df684b20ed83
DebuggerCount: 0

Failed Assertion: /tmp/hhvm/hphp/runtime/vm/jit/translator.cpp:1942: HPHP::jit::Translator::TranslateResult HPHP::jit::Translator::translateRegion(const HPHP::jit::RegionDesc&, HPHP::jit::Translator::RegionBlacklist&, HPHP::jit::TransFlags): assertion `0' failed.
URL: /show/123
Server: ***.net
0: data block = cold
message: Attempted to emit 4 byte(s) into a 77 byte DataBlock with 1 bytes available. This almost certainly means the TC is full. If this is the case, increasing Eval.JitASize, Eval.JitAColdSize, Eval.JitAFrozenSize and Eval.JitGlobalDataSize in the configuration file when running this script or application should fix this problem.

Server_SERVER_NAME: localhost
ThreadType: Web Request

# 0  __restore_rt at sigaction.c:0
# 1  __GI_raise at :0
# 2  __GI_abort at :0
# 3  HPHP::assert_fail(char const*, char const*, unsigned int, char const*) at /usr/local/bin/hhvm:0
# 4  HPHP::jit::Translator::translateRegion(HPHP::jit::RegionDesc const&, HPHP::hphp_hash_set<HPHP::jit::ProfSrcKey, HPHP::jit::ProfSrcKey::Hasher, std::equal_to<HPHP::jit::ProfSrcKey> >&, HPHP::jit::TransFlags) at /usr/local/bin/hhvm:0
# 5  HPHP::jit::MCGenerator::translateWork(HPHP::jit::TranslArgs const&) at /usr/local/bin/hhvm:0
# 6  HPHP::jit::MCGenerator::translate(HPHP::jit::TranslArgs const&) at /usr/local/bin/hhvm:0
# 7  HPHP::jit::MCGenerator::retranslate(HPHP::jit::TranslArgs const&) at /usr/local/bin/hhvm:0
# 8  HPHP::jit::MCGenerator::createTranslation(HPHP::jit::TranslArgs const&) at /usr/local/bin/hhvm:0
# 9  HPHP::jit::MCGenerator::bindJmpccFirst(unsigned char*, int, int, bool, HPHP::jit::ConditionCode, bool&) at /usr/local/bin/hhvm:0
# 10 HPHP::jit::MCGenerator::handleServiceRequest(HPHP::jit::TReqInfo&, unsigned char*&, HPHP::SrcKey&) at /usr/local/bin/hhvm:0
# 11 HPHP::jit::MCGenerator::enterTC(unsigned char*, void*) at /usr/local/bin/hhvm:0
# 12 HPHP::ExecutionContext::enterVM(HPHP::ActRec*, HPHP::ExecutionContext::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) at /usr/local/bin/hhvm:0
# 13 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) at /usr/local/bin/hhvm:0
# 14 HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) at /usr/local/bin/hhvm:0
# 15 HPHP::invoke_file(HPHP::String const&, bool, char const*) at builtin-functions.cpp:0
# 16 HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) at /usr/local/bin/hhvm:0
# 17 HPHP::hphp_invoke(HPHP::ExecutionContext*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool, bool, bool) at /usr/local/bin/hhvm:0
# 18 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/local/bin/hhvm:0
# 19 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/local/bin/hhvm:0
# 20 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/local/bin/hhvm:0
# 21 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/local/bin/hhvm:0
# 22 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/local/bin/hhvm:0
# 23 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/local/bin/hhvm:0
# 24 HPHP::start_routine_wrapper(void*) at /usr/local/bin/hhvm:0
# 25 start_thread at pthread_create.c:0
# 26 __clone at /lib64/libc.so.6:0

PHP Stacktrace:

#0  include() called at [/var/www/lib/Smarty-3.1.13/libs/sysplugins/smarty_internal_templatebase.php:252]
#1  Smarty_Internal_TemplateBase->fetch(show.tpl, /show/123, index.tpl, Object of class Smarty could not be converted to string, 1) called at [/var/www/lib/Smarty-3.1.13/libs/sysplugins/smarty_internal_templatebase.php:374]
#2  Smarty_Internal_TemplateBase->display(show.tpl, /show/123, index.tpl) called at [/var/www/vhost/index.php:500]

At smarty_internal_templatebase.php:252 there is eval() of template code. And in this code is cached template with two includes() of simple modifier smarty plugins.

paulbiss commented 9 years ago

@kowach have you tried the suggestion from the abort message? This almost certainly means the TC is full. If this is the case, increasing Eval.JitASize, Eval.JitAColdSize, Eval.JitAFrozenSize and Eval.JitGlobalDataSize in the configuration file when running this script or application should fix this problem.

kowach commented 9 years ago

I've increased these about 10 times of default value: JitASize = 536870912 JitAFrozenSize = 536870912 but it didn't help.

paulbiss commented 9 years ago

Hmm, could you paste your config file?

leiffoged commented 9 years ago

Closing out due to inactivity: https://github.com/facebook/hhvm/wiki/Human-Timeouts