Open wxmet opened 4 years ago
wxmet
commented
4 years ago Not sure if this helps but this is in my server error log:
[Wed Nov 20 22:54:49.746326 2019] [core:error] [pid 27669] [client 10.25.7.66:59898] End of script output before headers: search,
CamJN
commented
4 years ago 
vaseched
commented
4 years ago I'm running into the exact same issue, and setting PassengerDisableSecurityUpdateCheck does not help
CamJN
commented
4 years ago @vaseched are you also running RHEL 5?
vaseched
commented
4 years ago I'm actually running on Amazon Linux 2. Specifically I'm running into this with:
PhusionPassenger-5.1.2
Ruby-2.5.8
Rails-5.2.0
AmazonLinux2, Linux Kernel 5.4List of all the loaded Ruby Gems:
did_you_mean 1.2.0
concurrent-ruby 1.0.5
i18n 0.9.5
thread_safe 0.3.6
tzinfo 1.2.7
activesupport 5.2.0
rack 2.0.9
rack-test 1.1.0
mini_portile2 2.3.0
nokogiri 1.8.5
loofah 2.0.3
rails-html-sanitizer 1.0.3
rails-dom-testing 2.0.3
builder 3.2.3
erubi 1.7.1
actionview 5.2.0
actionpack 5.2.0
rake 12.3.3
thor 0.20.3
method_source 0.9.2
railties 5.2.0
fileutils 1.0.2
etc 1.0.0
psych 3.0.2
stringio 0.0.1
strscan 1.0.0
openssl 2.1.2
date 1.0.0
bigdecimal 1.3.4
json 2.2.0
globalid 0.4.2
activejob 5.2.0
mime-types-data 3.2015.1120
mime-types 3.0
mail 2.6.4
actionmailer 5.2.0
sprockets 3.7.2
sprockets-rails 3.2.1
zlib 1.0.0
webrick 1.4.2
temple 0.8.2
tilt 2.0.10
haml 5.0.4
erubis 2.7.0
yajl-ruby 1.2.3
safe_yaml 1.0.4
crack 0.4.3
net-ldap 0.16.1
jmespath 1.4.0
aws-eventstream 1.1.0
aws-sigv4 1.1.4
aws-sdk-core 2.11.562
aws-sdk-resources 2.11.562
ipaddr 1.2.0
fcntl 1.0.0Here's a backtrace from the thread that's crashing, looks like I've got a bit more detail than the original issue.
########### Thread 10 (Thread 0x7f2e757fb700 (LWP 22443)) ###########
#0 0x00007f2e7f79f66e in raise () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x0000000000474a17 in Passenger::abortHandler (signo=11,
info=<optimized out>, ctx=<optimized out>)
at src/agent/Shared/Base.cpp:1024
state = {pid = 22432, signo = 11, info = 0x7f2e757f3f70,
messagePrefix = "[ pid=22432", '\000' <repeats 20 times>,
messageBuf = "[ pid=22432 ] Crash log dumped to /var/tmp/passenger-crash-log.1599040219\n\000son=SI_KERNEL, si_addr=0x0, randomSeed=1599040217\n", '\000' <repeats 898 times>}
child = <optimized out>
t = <optimized out>
crashLogFile = "/var/tmp/passenger-crash-log.1599040219", '\000' <repeats 216 times>
end = <optimized out>
#2 <signal handler called>
No symbol table info available.
#3 Passenger::convertLowerCase (
data=data@entry=0x7f2e4c00614b "Upgrade-Insecure-Requests: 1\r\nCache-Control: max-age=0\r\nAuthorization: Negotiate YIIHeAYGKwYBBQUCoIIHbDCCB2igFTATBgkqhkiG9xIBAgIGBiqFcCsOA6KCB00EggdJYIIHRQYJKoZIhvcSAQICAQBuggc0MIIHMKADAgEFoQMCAQ6iBwM"..., output=output@entry=0x7f2e4c001fa0 "upgrade-", len=len@entry=25)
at src/cxx_supportlib/Utils/StrIntUtilsNoStrictAliasing.cpp:122
gsToLowerMap = "\000\001\002\003\004\005\006\a\b\t\n\v\f\r\016\017\020\021\022\023\024\025\026\027\030\031\032\033\034\035\036\037 !\"#$%&'()*+,-./0123456789:;<=>?@abcdefghijklmnopqrstuvwxyz[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\177\200\201\202\203\204\205\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237\240\241\242\243\244\245\246\247\250\251\252\253\254\255\256\257\260\261\262\263\264\265\266\267\270\271\272\273\274\275\276\277\300\301\302\303\304\305\306\307"...
i = <optimized out>
eax = <optimized out>
ebx = <optimized out>
ustr = 0x7f2e4c00614b "Upgrade-Insecure-Requests: 1\r\nCache-Control: max-age=0\r\nAuthorization: Negotiate YIIHeAYGKwYBBQUCoIIHbDCCB2igFTATBgkqhkiG9xIBAgIGBiqFcCsOA6KCB00EggdJYIIHRQYJKoZIhvcSAQICAQBuggc0MIIHMKADAgEFoQMCAQ6iBwM"...
leftover = 1
imax = 3
s = 0x7f2e4c00614b
d = <optimized out>
#4 0x00000000005ae0ab in Passenger::ServerKit::HttpHeaderParser<Passenger::Core::Request, Passenger::ServerKit::HttpParseRequest>::onHeaderField (
parser=<optimized out>,
data=0x7f2e4c00614b "Upgrade-Insecure-Requests: 1\r\nCache-Control: max-age=0\r\nAuthorization: Negotiate YIIHeAYGKwYBBQUCoIIHbDCCB2igFTATBgkqhkiG9xIBAgIGBiqFcCsOA6KCB00EggdJYIIHRQYJKoZIhvcSAQICAQBuggc0MIIHMKADAgEFoQMCAQ6iBwM"...,
len=25) at src/cxx_supportlib/ServerKit/HttpHeaderParser.h:210
downcasedData = 0x7f2e4c001fa0 "upgrade-"
self = 0x7f2e757f5da0
#5 0x000000000046f797 in http_parser_execute (
parser=parser@entry=0x7f2e4c0016d8,
settings=settings@entry=0x7f2e757f5230,
data=data@entry=0x7f2e4c005a80 "GET / HTTP/1.1\r\nHost: hostname:1081\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:76.0) Gecko/20100101 Firefox/76.0\r\nAccept: text/html,application/xhtml+xml,appl"..., len=len@entry=4048)
at src/cxx_supportlib/ServerKit/http_parser.cpp:1389
c = 0 '\000'
ch = 58 ':'
unhex_val = <optimized out>
p = <optimized out>
header_field_mark = <optimized out>
header_value_mark = 0x0
url_mark = <optimized out>
body_mark = 0x0
status_mark = <optimized out>
__PRETTY_FUNCTION__ = "size_t http_parser_execute(http_parser*, const http_parser_settings*, const char*, size_t)"
#6 0x00000000005bfc16 in Passenger::ServerKit::HttpHeaderParser<Passenger::Core::Request, Passenger::ServerKit::HttpParseRequest>::http_parser_execute_and_handle_pause (paused=<synthetic pointer>: <optimized out>, len=4048,
data=0x7f2e4c005a80 "GET / HTTP/1.1\r\nHost: hostname:1081\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:76.0) Gecko/20100101 Firefox/76.0\r\nAccept: text/html,application/xhtml+xml,appl"..., settings=0x7f2e757f5230, parser=0x7f2e4c0016d8)
at src/cxx_supportlib/ServerKit/HttpHeaderParser.h:125
ret = <optimized out>
#7 Passenger::ServerKit::HttpHeaderParser<Passenger::Core::Request, Passenger::ServerKit::HttpParseRequest>::feed (this=this@entry=0x7f2e757f5da0,
buffer=...) at src/cxx_supportlib/ServerKit/HttpHeaderParser.h:465
__PRETTY_FUNCTION__ = "size_t Passenger::ServerKit::HttpHeaderParser<Message, MessageType>::feed(const Passenger::MemoryKit::mbuf&) [with Message = Passenger::Core::Request; MessageType = Passenger::ServerKit::HttpParseRequ"...
__p = {
function = 0x6a14a0 <Passenger::ServerKit::HttpHeaderParser<Passenger::Core::Request, Passenger::ServerKit::HttpParseRequest>::feed(Passenger::MemoryKit::mbuf const&)::__PRETTY_FUNCTION__> "size_t Passenger::ServerKit::HttpHeaderParser<Message, MessageType>::feed(const Passenger::MemoryKit::mbuf&) [with Message = Passenger::Core::Request; MessageType = Passenger::ServerKit::HttpParseRequ"...,
source = 0x67cee0 "src/cxx_supportlib/ServerKit/HttpHeaderParser.h", u = {data = 0x0, dataFunc = {func = 0x0, userData = 0x0}}, line = 447,
m_detached = false, m_hasDataFunc = false}
settings = {on_message_begin = 0x0,
on_url = 0x5ad970 <Passenger::ServerKit::HttpHeaderParser<Passenger::Core::Request, Passenger::ServerKit::HttpParseRequest>::_onURL(http_parser*, char const*, unsigned long)>,
on_status = 0x5c16f0 <Passenger::ServerKit::HttpHeaderParser<Passenger::Core::Request, Passenger::ServerKit::HttpParseRequest>::onStatus(http_parser*, char const*, unsigned long)>,
on_header_field = 0x5adfc0 <Passenger::ServerKit::HttpHeaderParser<Passenger::Core::Request, Passenger::ServerKit::HttpParseRequest>::onHeaderField(http_parser*, char const*, unsigned long)>,
on_header_value = 0x5ad850 <Passenger::ServerKit::HttpHeaderParser<Passenger::Core::Request, Passenger::ServerKit::HttpParseRequest>::onHeaderValue(http_parser*, char const*, unsigned long)>,
on_headers_complete = 0x5ae2f0 <Passenger::ServerKit::HttpHeaderParser<Passenger::Core::Request, Passenger::ServerKit::HttpParseRequest>::onHeadersComplete(http_parser*)>, on_body = 0x0, on_message_complete = 0x0}
ret = <optimized out>
paused = <optimized out>
#8 0x00000000005c0130 in Passenger::ServerKit::HttpServer<Passenger::Core::Controller, Passenger::Core::Client>::processClientDataWhenParsingHeaders (
this=this@entry=0x7f2e7fbf3010, client=<optimized out>,
client@entry=0xf610e0, req=<optimized out>, req@entry=0x7f2e4c000bd0,
buffer=..., errcode=errcode@entry=0)
at src/cxx_supportlib/ServerKit/HttpServer.h:272
ret = <optimized out>
__PRETTY_FUNCTION__ = "Passenger::ServerKit::Channel::Result Passenger::ServerKit::HttpServer<DerivedServer, Client>::processClientDataWhenParsingHeaders(Client*, Passenger::ServerKit::HttpServer<DerivedServer, Client>::Req"...
#9 0x00000000005c0ecb in Passenger::ServerKit::HttpServer<Passenger::Core::Controller, Passenger::Core::Client>::onClientDataReceived (this=0x7f2e7fbf3010,
client=0xf610e0, buffer=..., errcode=0)
at src/cxx_supportlib/ServerKit/HttpServer.h:724
__PRETTY_FUNCTION__ = "Passenger::ServerKit::Channel::Result Passenger::ServerKit::HttpServer<DerivedServer, Client>::onClientDataReceived(Client*, const Passenger::MemoryKit::mbuf&, int) [with DerivedServer = Passenger::Co"...
__p = {
function = 0x6a17a0 <Passenger::ServerKit::HttpServer<Passenger::Core::Controller, Passenger::Core::Client>::onClientDataReceived(Passenger::Core::Client*, Passenger::MemoryKit::mbuf const&, int)::__PRETTY_FUNCTION__> "Passenger::ServerKit::Channel::Result Passenger::ServerKit::HttpServer<DerivedServer, Client>::onClientDataReceived(Client*, const Passenger::MemoryKit::mbuf&, int) [with DerivedServer = Passenger::Co"...,
source = 0x6793d0 "src/cxx_supportlib/ServerKit/HttpServer.h", u = {
data = 0x5ac080 <Passenger::ServerKit::BaseServer<Passenger::Core::Controller, Passenger::Core::Client>::_getClientNameFromTracePoint(char*, unsigned int, void*)> "ATH\215\005\f\323\f", dataFunc = {
func = 0x5ac080 <Passenger::ServerKit::BaseServer<Passenger::Core::Controller, Passenger::Core::Client>::_getClientNameFromTracePoint(char*, unsigned int, void*)>, userData = 0xf610e0}}, line = 708, m_detached = false,
m_hasDataFunc = true}
req = 0x7f2e4c000bd0
ref = {request = 0x7f2e4c000bd0,
file = 0x6793d0 "src/cxx_supportlib/ServerKit/HttpServer.h",
line = 711}
ended = false
#10 0x000000000051b31b in Passenger::ServerKit::BaseServer<Passenger::Core::Controller, Passenger::Core::Client>::_onClientDataReceived (
_channel=<optimized out>, buffer=..., errcode=0)
at src/cxx_supportlib/ServerKit/Server.h:492
channel = <optimized out>
client = 0xf610e0
bufferSize = 4048
result = <optimized out>
#11 0x00000000004980bf in Passenger::ServerKit::Channel::callDataCallbackWithoutRefGuard (this=this@entry=0xf61108)
at src/cxx_supportlib/ServerKit/Channel.h:373
copy = {mbuf_block = 0x7f2e4c006a50,
start = 0x7f2e4c005a80 "GET / HTTP/1.1\r\nHost: hostname:1081\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:76.0) Gecko/20100101 Firefox/76.0\r\nAccept: text/html,application/xhtml+xml,appl"..., end = 0x7f2e4c006a50 "\357\276\255\336"}
generation = 0
cbResult = <optimized out>
__PRETTY_FUNCTION__ = "int Passenger::ServerKit::Channel::callDataCallbackWithoutRefGuard()"
#12 0x0000000000498a8a in Passenger::ServerKit::Channel::feedWithoutRefGuard (
this=this@entry=0xf61108, mbuf=...)
at src/cxx_supportlib/ServerKit/Channel.h:603
__PRETTY_FUNCTION__ = "int Passenger::ServerKit::Channel::feedWithoutRefGuard(Passenger::MemoryKit::mbuf&&)"
#13 0x000000000049f41e in Passenger::ServerKit::FdSourceChannel::onReadableWithoutRefGuard (this=0xf61108)
at src/cxx_supportlib/ServerKit/FdSourceChannel.h:91
buffer2 = {mbuf_block = 0x7f2e4c006a50,
start = 0x7f2e4c005a80 "GET / HTTP/1.1\r\nHost: hostname:1081\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:76.0) Gecko/20100101 Firefox/76.0\r\nAccept: text/html,application/xhtml+xml,appl"..., end = 0x7f2e4c006a50 "\357\276\255\336"}
generation = <optimized out>
i = <optimized out>
origBufferSize = 4048
done = false
ret = 4048
#14 0x00000000005b85f4 in Passenger::ServerKit::FdSourceChannel::startReading
(this=0xf61108) at src/cxx_supportlib/ServerKit/FdSourceChannel.h:186
No locals.
#15 Passenger::ServerKit::BaseServer<Passenger::Core::Controller, Passenger::Core::Client>::onClientsAccepted (this=0x7f2e7fbf3010, clients=<optimized out>,
size=<optimized out>) at src/cxx_supportlib/ServerKit/Server.h:543
client = 0xf610e0
i = <optimized out>
#16 0x000000000051b87d in Passenger::ServerKit::BaseServer<Passenger::Core::Controller, Passenger::Core::Client>::feedNewClients (this=0x7f2e7fbf3010,
fds=fds@entry=0x7f2e757fac3c, size=size@entry=1)
at src/cxx_supportlib/ServerKit/Server.h:797
__PRETTY_FUNCTION__ = "void Passenger::ServerKit::BaseServer<DerivedServer, Client>::feedNewClients(const int*, unsigned int) [with DerivedServer = Passenger::Core::Controller; Client = Passenger::Core::Client]"
acceptedClients = {0xf610e0, 0x0 <repeats 126 times>}
#17 0x000000000051c1c7 in Passenger::ServerKit::AcceptLoadBalancer<Passenger::Core::Controller>::feedNewClient (server=<optimized out>, fd=<optimized out>)
at src/cxx_supportlib/ServerKit/AcceptLoadBalancer.h:174
No locals.
#18 0x00000000005032df in boost::function0<void>::operator() (
this=0x7f2e4c000ba0)
at src/cxx_supportlib/vendor-modified/boost/function/function_template.hpp:770
No locals.
#19 Passenger::SafeLibev::runCommands (this=0xeaf2f0)
at src/cxx_supportlib/SafeLibev.h:98
l = {m = 0xeaf328, is_locked = false}
commands = Python Exception <class 'gdb.error'> value has been optimized out:
it =
{callback = {<boost::function0<void>> = {<boost::function_base> = {vtable = 0x91cbd1 <void boost::function0<void>::assign_to<boost::_bi::bind_t<void, void (*)(Passenger::Core::Controller*, int), boost::_bi::list2<boost::_bi::value<Passenger::Core::Controller*>, boost::_bi::value<int> > > >(boost::_bi::bind_t<void, void (*)(Passenger::Core::Controller*, int), boost::_bi::list2<boost::_bi::value<Passenger::Core::Controller*>, boost::_bi::value<int> > >)::stored_vtable+1>, functor = {obj_ptr = 0x51c1b0 <Passenger::ServerKit::AcceptLoadBalancer<Passenger::Core::Controller>::feedNewClient(Passenger::Core::Controller*, int)>, type = {type = 0x51c1b0 <Passenger::ServerKit::AcceptLoadBalancer<Passenger::Core::Controller>::feedNewClient(Passenger::Core::Controller*, int)>, const_qualified = 16, volatile_qualified = 48}, func_ptr = 0x51c1b0 <Passenger::ServerKit::AcceptLoadBalancer<Passenger::Core::Controller>::feedNewClient(Passenger::Core::Controller*, int)>, bound_memfunc_ptr = {memfunc_ptr = (void (boost::detail::function::X::*)(boost::detail::function::X * const, int)) 0x51c1b0 <Passenger::ServerKit::AcceptLoadBalancer<Passenger::Core::Controller>::feedNewClient(Passenger::Core::Controller*, int)>, this adjustment 139837688459280, obj_ptr = 0x80}, obj_ref = {obj_ptr = 0x51c1b0 <Passenger::ServerKit::AcceptLoadBalancer<Passenger::Core::Controller>::feedNewClient(Passenger::Core::Controller*, int)>, is_const_qualified = 16, is_volatile_qualified = 48}, data = -80 '\260'}}, static args = <optimized out>, static arity = <optimized out>}, <No data fields>}, id = 1, canceled = false}
#20 Passenger::SafeLibev::asyncHandler (loop=<optimized out>,
w=<optimized out>, revents=<optimized out>)
at src/cxx_supportlib/SafeLibev.h:81
self = 0xeaf2f0
#21 0x0000000000650c58 in ev_invoke_pending (loop=0xeaeb90)
at .../src/cxx_supportlib/vendor-modified/libev/ev.c:3326
p = <optimized out>
#22 0x0000000000654bc8 in ev_run (loop=0xeaeb90, flags=<optimized out>)
at .../src/cxx_supportlib/vendor-modified/libev/ev.c:3729
No locals.
#23 0x000000000064ad83 in boost::function0<void>::operator() (
this=0x7f2e757fae40)
at src/cxx_supportlib/vendor-modified/boost/function/function_template.hpp:770
No locals.
#24 oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) (func=..., ctx=...)
at src/cxx_supportlib/oxt/implementation.cpp:416
No locals.
#25 0x000000000048d3ef in boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > >::operator()<void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list0&, int) (a=<synthetic pointer>..., f=<optimized out>, this=<optimized out>)
at src/cxx_supportlib/vendor-modified/boost/bind/bind.hpp:319
No locals.
#26 boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > >::operator()() (this=<optimized out>)
at src/cxx_supportlib/vendor-modified/boost/bind/bind.hpp:1222
No locals.
#27 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() (this=<optimized out>)
at src/cxx_supportlib/vendor-modified/boost/thread/detail/thread.hpp:116
No locals.
#28 0x000000000061adce in boost::(anonymous namespace)::thread_proxy (
param=<optimized out>)
at src/cxx_supportlib/vendor-modified/boost/libs/thread/src/pthread/thread.cpp:167
thread_info = {px = 0x1110770, pn = {pi_ = 0x1110970}}
lock = <optimized out>
#29 0x00007f2e7f79540b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#30 0x00007f2e7dee9e7f in clone () from /lib64/libc.so.6
No symbol table info available.
Issue report
Fill in as much as possible so that we can understand, find and fix the problem.
Are you sure this is a bug in Passenger?
Yes.
Question 1: What is the problem? Passenger crashes when upgrading from 3.0.12 to 5.1.x. The application log returns a 200 when a request is made but passenger crashes with a seg fault:
`
Red Hat Enterprise Linux Server release 5.3 Amazon Linux Bare Metal release 2012.03 Ruby 2.5.x PhusionPassenger 5.1.x
Question 2: Passenger version and integration mode:
Your answer: enterprise 5.1.x/apache
Question 3: OS or Linux distro, platform (including version):
Red Hat Enterprise Linux Server release 5.3
Question 4: Passenger installation method:
Your answer: [ x] RubyGems + Gemfile [ ] RubyGems, no Gemfile [ ] Phusion APT repo [ ] Phusion YUM repo [ ] OS X Homebrew [ ] source tarball [ ] Other, please specify:
Question 5: Your app's programming language (including any version managers) and framework (including versions):
Ruby = 2.5.x Rails = 5.2, RubyGem-rake = 12.3.x; RubyGem-rails = 5.2.x; RubyGem-sass-rails = 5.0.x_1; RubyGem-coffee-rails = 5.0.x; RubyGem-bourbon = 6.0.x; RubyGem-uglifier = 4.1.x; NodeJS = 12.x; RubyGem-net-ldap = 0.16.x; RubyGem-mysql2 = 0.5.x; RubyGem-will_paginate = 3.1.x; RubyGem-yard = 0.9.x;