search

phusion / passenger

A fast and robust web server and application server for Ruby, Python and Node.js
https://www.phusionpassenger.com/
MIT License
4.99k stars 548 forks source link

Passenger standalone 5.0.6 crashes with SIGSEGV #1469

Closed karlsvec closed 9 years ago

karlsvec commented 9 years ago

I'm seeing occasional crashes with 5.0.6 standalone using the official packages from the APT repository on Ubuntu 14.04 LTS.

[ pid=27454, timestamp=1428447995 ] Process aborted! signo=SIGSEGV(11), reason=SI_KERNEL, si_addr=0x0, randomSeed=1428447984
[ pid=27454 ] Crash log dumped to /var/tmp/passenger-crash-log.1428447995
[ pid=27454 ] Date, uname and ulimits:
Tue Apr  7 23:06:35 UTC 2015
Linux 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     0
memory(kbytes)       unlimited
locked memory(kbytes) 64
process              15922
nofiles              1024
vmemory(kbytes)      unlimited
locks                unlimited
[ pid=27454 ] Phusion Passenger version: 5.0.6
[ pid=27454 ] libc backtrace available!
--------------------------------------
[ pid=27454 ] Backtrace with 36 frames:
Using backtrace sanitizer.
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d27ab]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d2057]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d335b]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d4015]
/lib/x86_64-linux-gnu/libpthread.so.0(?? at ??:0; +0x10340) [0x7f38881db340]
PassengerAgent server(__gnu_cxx::__alloc_traits<std::allocator<std::string> >::_S_propagate_on_move_assign() at ??:?) [0x47fdf8]
PassengerAgent server(__gnu_cxx::__alloc_traits<std::allocator<std::string> >::_S_propagate_on_move_assign() at ??:?) [0x480f01]
PassengerAgent server(__gnu_cxx::__alloc_traits<std::allocator<std::string> >::_S_propagate_on_move_assign() at ??:?) [0x481598]
PassengerAgent server(std::pair<std::string, std::string>::pair<char const*, std::string, void>(std::pair<char const*, std::string>&&) at ??:?) [0x483d03]
PassengerAgent server(std::pair<std::string, std::string>::pair<char const*, std::string, void>(std::pair<char const*, std::string>&&) at ??:?) [0x486c72]
PassengerAgent server(int&& std::forward<int>(std::remove_reference<int>::type&) at ??:?) [0x48bcc7]
PassengerAgent server(std::pair<std::string const, std::string>::pair<std::string, std::string, void>(std::pair<std::string, std::string>&&) at ??:?) [0x56d6fc]
PassengerAgent server(std::pair<std::string const, std::string>::pair<std::string, std::string, void>(std::pair<std::string, std::string>&&) at ??:?) [0x578f85]
PassengerAgent server(std::pair<std::string const, std::string>::pair<std::string, std::string, void>(std::pair<std::string, std::string>&&) at ??:?) [0x574994]
PassengerAgent server(Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>** std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>*>(Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>* const*, Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>* const*, Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>**) at ??:?) [0x5e4ce2]
PassengerAgent server(Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>** std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>*>(Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>* const*, Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>* const*, Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>**) at ??:?) [0x5e0b1b]
PassengerAgent server(Passenger::ServerAgent::AdminServer::Authorization* std::__uninitialized_copy<false>::__uninit_copy<Passenger::ServerAgent::AdminServer::Authorization*, Passenger::ServerAgent::AdminServer::Authorization*>(Passenger::ServerAgent::AdminServer::Authorization*, Passenger::ServerAgent::AdminServer::Authorization*, Passenger::ServerAgent::AdminServer::Authorization*) at ??:?) [0x5cbde6]
PassengerAgent server(std::numeric_limits<unsigned long long>::max() at ??:?) [0x50373f]
PassengerAgent server(std::numeric_limits<unsigned long long>::max() at ??:?) [0x504bee]
PassengerAgent server(std::numeric_limits<unsigned long long>::max() at ??:?) [0x506371]
PassengerAgent server(std::numeric_limits<unsigned long long>::max() at ??:?) [0x50606b]
PassengerAgent server(std::numeric_limits<unsigned long long>::max() at ??:?) [0x506025]
PassengerAgent server(oxt::trace_point** std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<oxt::trace_point*>(oxt::trace_point* const*, oxt::trace_point* const*, oxt::trace_point**) at ??:?) [0x691a28]
PassengerAgent server(oxt::trace_point** std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<oxt::trace_point*>(oxt::trace_point* const*, oxt::trace_point* const*, oxt::trace_point**) at ??:?) [0x692902]
PassengerAgent server(unsigned long long const& std::min<unsigned long long>(unsigned long long const&, unsigned long long const&) at ??:?) [0x540868]
PassengerAgent server(void std::_Destroy_aux<false>::__destroy<boost::re_detail::recursion_info<boost::match_results<__gnu_cxx::__normal_iterator<char const*, std::string>, std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*, std::string> > > > >*>(boost::re_detail::recursion_info<boost::match_results<__gnu_cxx::__normal_iterator<char const*, std::string>, std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*, std::string> > > > >*, boost::re_detail::recursion_info<boost::match_results<__gnu_cxx::__normal_iterator<char const*, std::string>, std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*, std::string> > > > >*) at ??:?) [0x5d58ff]
PassengerAgent server(__gnu_cxx::__normal_iterator<char const*, std::string>::difference_type __gnu_cxx::operator-<char const*, std::string>(__gnu_cxx::__normal_iterator<char const*, std::string> const&, __gnu_cxx::__normal_iterator<char const*, std::string> const&) at ??:?) [0x5d0461]
PassengerAgent server(void std::_Destroy_aux<false>::__destroy<Passenger::ServerAgent::AdminServer::Authorization*>(Passenger::ServerAgent::AdminServer::Authorization*, Passenger::ServerAgent::AdminServer::Authorization*) at ??:?) [0x5caaa4]
PassengerAgent server(std::remove_reference<std::string&>::type&& std::move<std::string&>(std::string&) at ??:?) [0x453e90]
PassengerAgent server(boost::thread_detail::rollback_once_region(boost::once_flag&) at ??:?) [0x6869f5]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d0aec]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d06d7]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4cfec2]
PassengerAgent server(boost::detail::get_current_thread_data() at ??:?) [0x681deb]
/lib/x86_64-linux-gnu/libpthread.so.0(?? at ??:0; +0x8182) [0x7f38881d3182]
/lib/x86_64-linux-gnu/libc.so.6(?? at ??:0; clone+0x6d) [0x7f38874d847d]
--------------------------------------
[ pid=27454 ] Dumping additional diagnostical information...
--------------------------------------
### Backtraces
Thread 'Main thread' (0x7f3888a6d7c0, LWP 27454):
     in 'void waitForExitEvent()' (Main.cpp:788)
     in 'void mainLoop()' (Main.cpp:681)
     in 'int runServer()' (Main.cpp:903)

Thread 'Main event loop: thread 1' (0x7f388229e700, LWP 27454):
     in 'Passenger::ServerKit::Channel::Result Passenger::ServerKit::HttpServer<DerivedServer, Client>::onClientDataReceived(Client*, const Passenger::MemoryKit::mbuf&, int) [with DerivedServer = Passenger::RequestHandler; Client = Passenger::Client]' (HttpServer.h:891) -- Client 1-10

Thread 'Pool garbage collector' (0x7f3888a2a700, LWP 27454):
     in 'static void Passenger::ApplicationPool2::Pool::garbageCollect(Passenger::ApplicationPool2::PoolPtr)' (GarbageCollection.h:50)

Thread 'Admin event loop' (0x7f3881a9d700, LWP 27454):
     (empty)

Thread 'Pool analytics collector' (0x7f3888a6b700, LWP 27454):
     in 'static void Passenger::ApplicationPool2::Pool::collectAnalytics(Passenger::ApplicationPool2::PoolPtr)' (AnalyticsCollection.h:56)

Thread 'PipeWatcher: PID 27478 stderr, fd 28' (0x7f388890d700, LWP 27477):
     in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1679)
     in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1660)

Thread 'PipeWatcher: PID 27478 stdout, fd 27' (0x7f38889a0700, LWP 27477):
     in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1679)
     in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1660)

Thread 'PipeWatcher: PID 27497 stdout, fd 26' (0x7f38888cc700, LWP 27477):
     in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1679)
     in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1660)

Thread 'Group process spawner: /opt/agsolver/pzm/layer/webapp/deploy/releases/5c6b1406835e99b8fc82e55aa02776e5b38c15d2#default' (0x7f388895f700, LWP 27460):
     in 'Passenger::ApplicationPool2::SmartSpawner::SpawnResult Passenger::ApplicationPool2::SmartSpawner::sendSpawnCommand(const Passenger::ApplicationPool2::Options&)' (SmartSpawner.h:638)
     in 'virtual Passenger::ApplicationPool2::SpawnObject Passenger::ApplicationPool2::SmartSpawner::spawn(const Passenger::ApplicationPool2::Options&)' (SmartSpawner.h:769)
     in 'void Passenger::ApplicationPool2::Group::spawnThreadRealMain(const SpawnerPtr&, const Passenger::ApplicationPool2::Options&, unsigned int)' (Implementation.cpp:1038)

### Request handler state (thread 1)

{
    "active_client_count" : 5,
    "active_clients" : 
    {
        "1-10" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr  7 23:06:35 2015",
                "relative" : "0s ago",
                "timestamp" : 1428447995.762682
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_response_body_already_read" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "app_response_body_fully_read" : true,
                "app_response_body_type" : "NO_BODY",
                "app_response_http_major" : 1,
                "app_response_http_minor" : 1,
                "app_response_http_state" : "COMPLETE",
                "app_response_want_keep_alive" : true,
                "app_sink_state" : 
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "pzm-development.agsolver.com",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/twitter/bootstrap/transition.js?body=1",
                "refcount" : 3,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr  7 23:06:35 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1428447995.762743
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-10",
            "number" : 10,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 3,
            "requests_begun" : 1
        },
        "1-6" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr  7 23:06:35 2015",
                "relative" : "0s ago",
                "timestamp" : 1428447995.735178
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_response_body_already_read" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "app_response_body_fully_read" : true,
                "app_response_body_type" : "NO_BODY",
                "app_response_http_major" : 1,
                "app_response_http_minor" : 1,
                "app_response_http_state" : "COMPLETE",
                "app_response_want_keep_alive" : false,
                "app_sink_state" : 
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "pzm-development.agsolver.com",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/jquery.js?body=1",
                "refcount" : 2,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr  7 23:06:35 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1428447995.735222
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-6",
            "number" : 6,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        },
        "1-7" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr  7 23:06:35 2015",
                "relative" : "0s ago",
                "timestamp" : 1428447995.735684
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_sink_state" : 
                {
                    "callback_in_progress" : true,
                    "end_acked" : true,
                    "ended" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "end_acked" : true,
                    "ended" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "pzm-development.agsolver.com",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/jquery_ujs.js?body=1",
                "refcount" : 2,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr  7 23:06:35 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1428447995.735736
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-7",
            "number" : 7,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        },
        "1-8" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr  7 23:06:35 2015",
                "relative" : "0s ago",
                "timestamp" : 1428447995.737176
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_sink_state" : 
                {
                    "callback_in_progress" : true,
                    "end_acked" : true,
                    "ended" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "end_acked" : true,
                    "ended" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "pzm-development.agsolver.com",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/jquery.iframe-transport.js?body=1",
                "refcount" : 2,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr  7 23:06:35 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1428447995.737229
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-8",
            "number" : 8,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        },
        "1-9" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr  7 23:06:35 2015",
                "relative" : "0s ago",
                "timestamp" : 1428447995.737354
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_sink_state" : 
                {
                    "callback_in_progress" : true,
                    "end_acked" : true,
                    "ended" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "end_acked" : true,
                    "ended" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "pzm-development.agsolver.com",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/jquery.remotipart.js?body=1",
                "refcount" : 2,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr  7 23:06:35 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1428447995.737396
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-9",
            "number" : 9,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 123,
    "free_request_count" : 0,
    "mbuf_pool" : 
    {
        "active_blocks" : 20,
        "active_memory" : 
        {
            "bytes" : 10240,
            "human_readable" : "10.0 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 1,
        "offset" : 448,
        "spare_memory" : 
        {
            "bytes" : 512,
            "human_readable" : "512 bytes"
        }
    },
    "pid" : 27454,
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 13357,
    "total_clients_accepted" : 10,
    "total_requests_begun" : 10,
    "turbocaching" : 
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 0
    }
}

### Request handler config (thread 1)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : true,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Pool state (simple)
----------- General information -----------
Max pool size : 5
Processes     : 1
Requests in top-level queue : 0

----------- Application groups -----------
/opt/agsolver/pzm/layer/webapp/deploy/releases/5c6b1406835e99b8fc82e55aa02776e5b38c15d2#default:
  App root: /opt/agsolver/pzm/layer/webapp/deploy/releases/5c6b1406835e99b8fc82e55aa02776e5b38c15d2
  (spawning 1 new process...)
  Requests in queue: 4
  * PID: 27497   Sessions: 0       Processed: 5       Uptime: 6s
    CPU: 56%     Memory  : 84M     Last used: 0s ago
    Disabling...
    URL     : http://127.0.0.1:45196
    Password: <redacted>

### mbuf stats

nfree_mbuf_blockq  : 1
nactive_mbuf_blockq: 20
mbuf_block_chunk_size: 512

### Pool state (XML)
<?xml version="1.0" encoding="iso8859-1" ?>
<info version="3"><passenger_version>5.0.6</passenger_version><process_count>1</process_count><max>5</max><capacity_used>2</capacity_used><get_wait_list_size>0</get_wait_list_size><get_wait_list></get_wait_list><supergroups><supergroup><name>/opt/agsolver/pzm/layer/webapp/deploy/releases/5c6b1406835e99b8fc82e55aa02776e5b38c15d2</name><state>READY</state><get_wait_list_size>0</get_wait_list_size><capacity_used>2</capacity_used><secret>VYj2ivMdvseBbQSuUF7AiA2QR5cjmEW73bT1u6gP5rT</secret><group default="true"><name>/opt/agsolver/pzm/layer/webapp/deploy/releases/5c6b1406835e99b8fc82e55aa02776e5b38c15d2&#35;default</name><component_name>default</component_name><app_root>/opt/agsolver/pzm/layer/webapp/deploy/releases/5c6b1406835e99b8fc82e55aa02776e5b38c15d2</app_root><app_type>rack</app_type><environment>web_development</environment><uuid>knAEoAElB0VxMCbFSPex</uuid><enabled_process_count>0</enabled_process_count><disabling_process_count>1</disabling_process_count><disabled_process_count>0</disabled_process_count><capacity_used>2</capacity_used><get_wait_list_size>4</get_wait_list_size><disable_wait_list_size>1</disable_wait_list_size><processes_being_spawned>1</processes_being_spawned><spawning/><secret>rACKyivcuunrIgM7</secret><life_status>ALIVE</life_status><options><app_root>/opt/agsolver/pzm/layer/webapp/deploy/releases/5c6b1406835e99b8fc82e55aa02776e5b38c15d2</app_root><app_group_name>/opt/agsolver/pzm/layer/webapp/deploy/releases/5c6b1406835e99b8fc82e55aa02776e5b38c15d2</app_group_name><app_type>rack</app_type><start_command>ruby&#9;/usr/share/passenger/helper-scripts/rack-loader.rb</start_command><startup_file>config.ru</startup_file><process_title>Passenger RubyApp</process_title><log_level>3</log_level><start_timeout>90000</start_timeout><environment>web_development</environment><base_uri>/</base_uri><spawn_method>smart</spawn_method><default_user>nobody</default_user><default_group>nogroup</default_group><ruby>ruby</ruby><python>python</python><nodejs>node</nodejs><logging_agent_address>unix:/tmp/passenger.zGn4uMT/agents.s/logging</logging_agent_address><logging_agent_username>logging</logging_agent_username><logging_agent_password>7xaNkqA0tdSFfwlqrHczNKg4</logging_agent_password><debugger>false</debugger><analytics>false</analytics><group_secret>rACKyivcuunrIgM7</group_secret><min_processes>1</min_processes><max_processes>0</max_processes><max_preloader_idle_time>-1</max_preloader_idle_time><max_out_of_band_work_instances>1</max_out_of_band_work_instances></options><processes><process><pid>27497</pid><sticky_session_id>2083839312</sticky_session_id><gupid>16b45ea-m9ATV51R3c</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>5</processed><spawner_creation_time>1428447985529230</spawner_creation_time><spawn_start_time>1428447989962882</spawn_start_time><spawn_end_time>1428447989966596</spawn_end_time><last_used>1428447995734744</last_used><last_used_desc>0s ago</last_used_desc><uptime>6s</uptime><life_status>ALIVE</life_status><enabled>DISABLING</enabled><has_metrics>true</has_metrics><cpu>56</cpu><rss>151776</rss><pss>118367</pss><private_dirty>86688</private_dirty><swap>0</swap><real_memory>86688</real_memory><vmsize>1014684</vmsize><process_group_id>27454</process_group_id><command>Passenger RubyApp: /opt/agsolver/pzm/layer/webapp/deploy/releases/5c6b1406835e99b8fc82e55aa02776e5b38c15d2</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.zGn4uMT/apps.s/ruby.wtTGajWs1dvsH1fGLTlf0Q9nPbtUlGcTwZPa2ZE81xmHwtVfql3oZqDPDkE9X63</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:45196</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process></processes></group></supergroup></supergroups></info>

--------------------------------------
[ pid=27454 ] Open files and file descriptors:
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF   NODE NAME
Passenger 27454 nobody  cwd    DIR              202,1     4096 172292 /opt/agsolver/pzm/layer/webapp/deploy/releases/5c6b1406835e99b8fc82e55aa02776e5b38c15d2
Passenger 27454 nobody  rtd    DIR              202,1     4096      2 /
Passenger 27454 nobody  txt    REG              202,1  3907864  17402 /usr/lib/passenger/support-binaries/PassengerAgent
Passenger 27454 nobody  mem    REG              202,1    47712 396074 /lib/x86_64-linux-gnu/libnss_files-2.19.so
Passenger 27454 nobody  mem    REG              202,1    47760 396076 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
Passenger 27454 nobody  mem    REG              202,1    97296 396071 /lib/x86_64-linux-gnu/libnsl-2.19.so
Passenger 27454 nobody  mem    REG              202,1    39824 396072 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
Passenger 27454 nobody  mem    REG              202,1    43368 396045 /lib/x86_64-linux-gnu/libcrypt-2.19.so
Passenger 27454 nobody  mem    REG              202,1   754880   8130 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
Passenger 27454 nobody  mem    REG              202,1   295816   8135 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
Passenger 27454 nobody  mem    REG              202,1    56768   8088 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
Passenger 27454 nobody  mem    REG              202,1   166040   8089 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
Passenger 27454 nobody  mem    REG              202,1    30944   8067 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
Passenger 27454 nobody  mem    REG              202,1    85680   6820 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
Passenger 27454 nobody  mem    REG              202,1   204104   8050 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
Passenger 27454 nobody  mem    REG              202,1   657712   8082 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
Passenger 27454 nobody  mem    REG              202,1   552584   8091 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
Passenger 27454 nobody  mem    REG              202,1    35856   8102 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
Passenger 27454 nobody  mem    REG              202,1    14256 396060 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
Passenger 27454 nobody  mem    REG              202,1    18416 396057 /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
Passenger 27454 nobody  mem    REG              202,1   268368   8114 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
Passenger 27454 nobody  mem    REG              202,1    80256   8132 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
Passenger 27454 nobody  mem    REG              202,1   251872   8090 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
Passenger 27454 nobody  mem    REG              202,1   109296   8126 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
Passenger 27454 nobody  mem    REG              202,1   101240 396095 /lib/x86_64-linux-gnu/libresolv-2.19.so
Passenger 27454 nobody  mem    REG              202,1    43672   8104 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
Passenger 27454 nobody  mem    REG              202,1    14592 396044 /lib/x86_64-linux-gnu/libcom_err.so.2.1
Passenger 27454 nobody  mem    REG              202,1   186824   8101 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
Passenger 27454 nobody  mem    REG              202,1   831616   8103 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
Passenger 27454 nobody  mem    REG              202,1    14664 396049 /lib/x86_64-linux-gnu/libdl-2.19.so
Passenger 27454 nobody  mem    REG              202,1   524704 393687 /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
Passenger 27454 nobody  mem    REG              202,1   775112   6819 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
Passenger 27454 nobody  mem    REG              202,1   323056   8106 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
Passenger 27454 nobody  mem    REG              202,1    59640   8105 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
Passenger 27454 nobody  mem    REG              202,1   285848   8086 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
Passenger 27454 nobody  mem    REG              202,1  1930528 397473 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Passenger 27454 nobody  mem    REG              202,1   383112 397472 /lib/x86_64-linux-gnu/libssl.so.1.0.0
Passenger 27454 nobody  mem    REG              202,1   105696   8125 /usr/lib/x86_64-linux-gnu/librtmp.so.0
Passenger 27454 nobody  mem    REG              202,1   207128   8100 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
Passenger 27454 nobody  mem    REG              202,1  1840928 396040 /lib/x86_64-linux-gnu/libc-2.19.so
Passenger 27454 nobody  mem    REG              202,1    90080 396054 /lib/x86_64-linux-gnu/libgcc_s.so.1
Passenger 27454 nobody  mem    REG              202,1  1071552 396063 /lib/x86_64-linux-gnu/libm-2.19.so
Passenger 27454 nobody  mem    REG              202,1   979056   8131 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
Passenger 27454 nobody  mem    REG              202,1    31792 396096 /lib/x86_64-linux-gnu/librt-2.19.so
Passenger 27454 nobody  mem    REG              202,1   141574 396093 /lib/x86_64-linux-gnu/libpthread-2.19.so
Passenger 27454 nobody  mem    REG              202,1   100728 396113 /lib/x86_64-linux-gnu/libz.so.1.2.8
Passenger 27454 nobody  mem    REG              202,1   422696   8056 /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
Passenger 27454 nobody  mem    REG              202,1   149120 396030 /lib/x86_64-linux-gnu/ld-2.19.so
Passenger 27454 nobody    0r   CHR                1,3      0t0   5430 /dev/null
Passenger 27454 nobody    1w  FIFO                0,8      0t0 242336 pipe
Passenger 27454 nobody    2w  FIFO                0,8      0t0 242336 pipe
Passenger 27454 nobody    3u  unix 0x0000000000000000      0t0 241956 socket
Passenger 27454 nobody    5w  FIFO                0,8      0t0 242336 pipe
Passenger 27454 nobody    8r  FIFO                0,8      0t0 241988 pipe
Passenger 27454 nobody    9w  FIFO                0,8      0t0 241988 pipe
Passenger 27454 nobody   10r  FIFO                0,8      0t0 241989 pipe
Passenger 27454 nobody   11w  FIFO                0,8      0t0 241989 pipe
Passenger 27454 nobody   12r  FIFO                0,8      0t0 241990 pipe
Passenger 27454 nobody   13w  FIFO                0,8      0t0 241990 pipe
Passenger 27454 nobody   14u  unix 0x0000000000000000      0t0 241991 /tmp/passenger.zGn4uMT/agents.s/server
Passenger 27454 nobody   15u  unix 0x0000000000000000      0t0 241992 /var/run/passenger/passenger.sock
Passenger 27454 nobody   16u  unix 0x0000000000000000      0t0 241994 /tmp/passenger.zGn4uMT/agents.s/server_admin
Passenger 27454 nobody   17r   CHR                1,9      0t0   5435 /dev/urandom
Passenger 27454 nobody   18r   CHR                1,9      0t0   5435 /dev/urandom
Passenger 27454 nobody   19u  0000                0,9        0   5385 anon_inode
Passenger 27454 nobody   20u  0000                0,9        0   5385 anon_inode
Passenger 27454 nobody   21u  0000                0,9        0   5385 anon_inode
Passenger 27454 nobody   22u  0000                0,9        0   5385 anon_inode
Passenger 27454 nobody   23u  0000                0,9        0   5385 anon_inode
Passenger 27454 nobody   24u  0000                0,9        0   5385 anon_inode
Passenger 27454 nobody   25u  unix 0x0000000000000000      0t0 242334 /var/run/passenger/passenger.sock
Passenger 27454 nobody   26u  unix 0x0000000000000000      0t0 242194 socket
Passenger 27454 nobody   27u  unix 0x0000000000000000      0t0 242050 socket
Passenger 27454 nobody   28r  FIFO                0,8      0t0 242051 pipe
Passenger 27454 nobody   29u  unix 0x0000000000000000      0t0 242303 socket
Passenger 27454 nobody   30u  unix 0x0000000000000000      0t0 242320 /var/run/passenger/passenger.sock
Passenger 27454 nobody   31u  unix 0x0000000000000000      0t0 242323 /var/run/passenger/passenger.sock
Passenger 27454 nobody   32u  unix 0x0000000000000000      0t0 242326 /var/run/passenger/passenger.sock
Passenger 27454 nobody   33u  unix 0x0000000000000000      0t0 242329 /var/run/passenger/passenger.sock
Passenger 27454 nobody   34u  unix 0x0000000000000000      0t0 242331 socket
--------------------------------------
[ pid=27454 ] Dumping a more detailed backtrace with crash-watch...
*** ERROR ***: 'gdb' isn't installed. Please install it first.
       Debian/Ubuntu: sudo apt-get install gdb
RedHat/CentOS/Fedora: sudo yum install gdb
            Mac OS X: please install the Developer Tools or XCode
             FreeBSD: use the devel/gdb port

I'm using Nginx as a proxy for Passenger standalone. Passenger is configured to use the built-in engine and listen on a Unix socket. The crash is generally accompanied by something like this in Nginx's error log:

2015/04/07 23:06:36 [error] 27509#0: *7849 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: <redacted>, server: <redacted>, request: "GET /assets/twitter/bootstrap/alert.js?body=1 HTTP/1.1", upstream: "http://unix:/var/run/passenger/passenger.sock:/assets/twitter/bootstrap/alert.js?body=1", host: "<redacted>", referrer: "<redacted>/"
karlsvec commented 9 years ago

Another crash log, this time including the detailed gdb trace:

[ pid=6231, timestamp=1428524913 ] Process aborted! signo=SIGSEGV(11), reason=SEGV_MAPERR, si_addr=0x410, randomSeed=1428523890
[ pid=6231 ] Crash log dumped to /var/tmp/passenger-crash-log.1428524913
[ pid=6231 ] Date, uname and ulimits:
Wed Apr  8 20:28:33 UTC 2015
Linux 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     0
memory(kbytes)       unlimited
locked memory(kbytes) 64
process              15922
nofiles              1024
vmemory(kbytes)      unlimited
locks                unlimited
[ pid=6231 ] Phusion Passenger version: 5.0.6
[ pid=6231 ] libc backtrace available!
--------------------------------------
[ pid=6231 ] Backtrace with 33 frames:
Using backtrace sanitizer.
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d27ab]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d2057]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d335b]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d4015]
/lib/x86_64-linux-gnu/libpthread.so.0(?? at ??:0; +0x10340) [0x7fa2f4d47340]
PassengerAgent server(__gnu_cxx::__alloc_traits<std::allocator<std::string> >::_S_propagate_on_move_assign() at ??:?) [0x48152a]
PassengerAgent server(std::pair<std::string, std::string>::pair<char const*, std::string, void>(std::pair<char const*, std::string>&&) at ??:?) [0x486c40]
PassengerAgent server(int&& std::forward<int>(std::remove_reference<int>::type&) at ??:?) [0x48bcc7]
PassengerAgent server(std::pair<std::string const, std::string>::pair<std::string, std::string, void>(std::pair<std::string, std::string>&&) at ??:?) [0x56d6fc]
PassengerAgent server(std::pair<std::string const, std::string>::pair<std::string, std::string, void>(std::pair<std::string, std::string>&&) at ??:?) [0x578f85]
PassengerAgent server(std::pair<std::string const, std::string>::pair<std::string, std::string, void>(std::pair<std::string, std::string>&&) at ??:?) [0x574994]
PassengerAgent server(Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>** std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>*>(Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>* const*, Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>* const*, Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>**) at ??:?) [0x5e4ce2]
PassengerAgent server(Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>** std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>*>(Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>* const*, Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>* const*, Passenger::ServerKit::HttpClient<Passenger::ServerAgent::Request>**) at ??:?) [0x5e0b1b]
PassengerAgent server(Passenger::ServerAgent::AdminServer::Authorization* std::__uninitialized_copy<false>::__uninit_copy<Passenger::ServerAgent::AdminServer::Authorization*, Passenger::ServerAgent::AdminServer::Authorization*>(Passenger::ServerAgent::AdminServer::Authorization*, Passenger::ServerAgent::AdminServer::Authorization*, Passenger::ServerAgent::AdminServer::Authorization*) at ??:?) [0x5cbde6]
PassengerAgent server(std::numeric_limits<unsigned long long>::max() at ??:?) [0x50373f]
PassengerAgent server(std::numeric_limits<unsigned long long>::max() at ??:?) [0x504bee]
PassengerAgent server(std::numeric_limits<unsigned long long>::max() at ??:?) [0x506371]
PassengerAgent server(std::numeric_limits<unsigned long long>::max() at ??:?) [0x50606b]
PassengerAgent server(std::numeric_limits<unsigned long long>::max() at ??:?) [0x506025]
PassengerAgent server(oxt::trace_point** std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<oxt::trace_point*>(oxt::trace_point* const*, oxt::trace_point* const*, oxt::trace_point**) at ??:?) [0x691a28]
PassengerAgent server(oxt::trace_point** std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<oxt::trace_point*>(oxt::trace_point* const*, oxt::trace_point* const*, oxt::trace_point**) at ??:?) [0x692902]
PassengerAgent server(unsigned long long const& std::min<unsigned long long>(unsigned long long const&, unsigned long long const&) at ??:?) [0x540868]
PassengerAgent server(void std::_Destroy_aux<false>::__destroy<boost::re_detail::recursion_info<boost::match_results<__gnu_cxx::__normal_iterator<char const*, std::string>, std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*, std::string> > > > >*>(boost::re_detail::recursion_info<boost::match_results<__gnu_cxx::__normal_iterator<char const*, std::string>, std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*, std::string> > > > >*, boost::re_detail::recursion_info<boost::match_results<__gnu_cxx::__normal_iterator<char const*, std::string>, std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*, std::string> > > > >*) at ??:?) [0x5d58ff]
PassengerAgent server(__gnu_cxx::__normal_iterator<char const*, std::string>::difference_type __gnu_cxx::operator-<char const*, std::string>(__gnu_cxx::__normal_iterator<char const*, std::string> const&, __gnu_cxx::__normal_iterator<char const*, std::string> const&) at ??:?) [0x5d0461]
PassengerAgent server(void std::_Destroy_aux<false>::__destroy<Passenger::ServerAgent::AdminServer::Authorization*>(Passenger::ServerAgent::AdminServer::Authorization*, Passenger::ServerAgent::AdminServer::Authorization*) at ??:?) [0x5caaa4]
PassengerAgent server(std::remove_reference<std::string&>::type&& std::move<std::string&>(std::string&) at ??:?) [0x453e90]
PassengerAgent server(boost::thread_detail::rollback_once_region(boost::once_flag&) at ??:?) [0x6869f5]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d0aec]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4d06d7]
PassengerAgent server(boost::thread_interrupted::~thread_interrupted() at ??:?) [0x4cfec2]
PassengerAgent server(boost::detail::get_current_thread_data() at ??:?) [0x681deb]
/lib/x86_64-linux-gnu/libpthread.so.0(?? at ??:0; +0x8182) [0x7fa2f4d3f182]
/lib/x86_64-linux-gnu/libc.so.6(?? at ??:0; clone+0x6d) [0x7fa2f404447d]
--------------------------------------
[ pid=6231 ] Dumping additional diagnostical information...
--------------------------------------
### Backtraces
Thread 'Main thread' (0x7fa2f55d97c0, LWP 6231):
     in 'void waitForExitEvent()' (Main.cpp:788)
     in 'void mainLoop()' (Main.cpp:681)
     in 'int runServer()' (Main.cpp:903)

Thread 'Pool garbage collector' (0x7fa2f5596700, LWP 6231):
     in 'static void Passenger::ApplicationPool2::Pool::garbageCollect(Passenger::ApplicationPool2::PoolPtr)' (GarbageCollection.h:50)

Thread 'Pool analytics collector' (0x7fa2f55d7700, LWP 6231):
     in 'static void Passenger::ApplicationPool2::Pool::collectAnalytics(Passenger::ApplicationPool2::PoolPtr)' (AnalyticsCollection.h:56)

Thread 'Main event loop: thread 1' (0x7fa2eee0a700, LWP 6231):
     in 'Passenger::ServerKit::Channel::Result Passenger::ServerKit::HttpServer<DerivedServer, Client>::onClientDataReceived(Client*, const Passenger::MemoryKit::mbuf&, int) [with DerivedServer = Passenger::RequestHandler; Client = Passenger::Client]' (HttpServer.h:891) -- Client 1-41

Thread 'Admin event loop' (0x7fa2ee609700, LWP 6231):
     (empty)

Thread 'PipeWatcher: PID 6697 stderr, fd 28' (0x7fa2f5438700, LWP 6696):
     in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1679)
     in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1660)

Thread 'PipeWatcher: PID 6727 stdout, fd 26' (0x7fa2ed5cc700, LWP 6696):
     in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1679)
     in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1660)

Thread 'Group process spawner: /opt/agsolver/pzm/layer/webapp/deploy/releases/3deb570d0ee6c4a95c7f978d9e8080cacaf59195#default' (0x7fa2f5479700, LWP 6236):
     in 'std::string Passenger::ApplicationPool2::Spawner::readMessageLine(Details&) [with Details = Passenger::ApplicationPool2::SmartSpawner::StartupDetails; std::string = std::basic_string<char>]' (Spawner.h:788)
     in 'std::string Passenger::ApplicationPool2::SmartSpawner::negotiatePreloaderStartup(Passenger::ApplicationPool2::SmartSpawner::StartupDetails&)' (SmartSpawner.h:565)
     in 'void Passenger::ApplicationPool2::SmartSpawner::startPreloader()' (SmartSpawner.h:264)
     in 'virtual Passenger::ApplicationPool2::SpawnObject Passenger::ApplicationPool2::SmartSpawner::spawn(const Passenger::ApplicationPool2::Options&)' (SmartSpawner.h:765)
     in 'void Passenger::ApplicationPool2::Group::spawnThreadRealMain(const SpawnerPtr&, const Passenger::ApplicationPool2::Options&, unsigned int)' (Implementation.cpp:1038)

Thread 'Background I/O capturer' (0x7fa2f548a700, LWP 6984):
     in 'void Passenger::ApplicationPool2::Spawner::BackgroundIOCapturer::capture()' (Spawner.h:136)

### Request handler state (thread 1)

{
    "active_client_count" : 2,
    "active_clients" : 
    {
        "1-40" : 
        {
            "connected_at" : 
            {
                "local" : "Wed Apr  8 20:28:33 2015",
                "relative" : "0s ago",
                "timestamp" : 1428524913.327502
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_response_body_already_read" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "app_response_body_fully_read" : true,
                "app_response_body_type" : "NO_BODY",
                "app_response_http_major" : 1,
                "app_response_http_minor" : 1,
                "app_response_http_state" : "COMPLETE",
                "app_response_want_keep_alive" : true,
                "app_sink_state" : 
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "pzm-development.agsolver.com",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/bootstrap.min.css?body=1",
                "refcount" : 2,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Wed Apr  8 20:28:33 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1428524913.327546
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-40",
            "number" : 40,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        },
        "1-41" : 
        {
            "connected_at" : 
            {
                "local" : "Wed Apr  8 20:28:33 2015",
                "relative" : "0s ago",
                "timestamp" : 1428524913.370112
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_response_body_already_read" : 
                {
                    "bytes" : 6219,
                    "human_readable" : "6.1 KB"
                },
                "app_response_body_fully_read" : true,
                "app_response_body_type" : "CHUNKED",
                "app_response_end_chunk_reached" : true,
                "app_response_http_major" : 1,
                "app_response_http_minor" : 1,
                "app_response_http_state" : "PARSING_CHUNKED_BODY",
                "app_response_want_keep_alive" : true,
                "app_sink_state" : 
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "pzm-development.agsolver.com",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/pzm-logo.svg",
                "refcount" : 3,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Wed Apr  8 20:28:33 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1428524913.370189
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-41",
            "number" : 41,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 3,
            "requests_begun" : 1
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 126,
    "free_request_count" : 1,
    "mbuf_pool" : 
    {
        "active_blocks" : 10,
        "active_memory" : 
        {
            "bytes" : 5120,
            "human_readable" : "5.0 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 8,
        "offset" : 448,
        "spare_memory" : 
        {
            "bytes" : 4096,
            "human_readable" : "4.0 KB"
        }
    },
    "pid" : 6231,
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 80770,
    "total_clients_accepted" : 41,
    "total_requests_begun" : 41,
    "turbocaching" : 
    {
        "fetches" : 4,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 4
    }
}

### Request handler config (thread 1)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : true,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Pool state (simple)
----------- General information -----------
Max pool size : 5
Processes     : 1
Requests in top-level queue : 0

----------- Application groups -----------
/opt/agsolver/pzm/layer/webapp/deploy/releases/3deb570d0ee6c4a95c7f978d9e8080cacaf59195#default:
  App root: /opt/agsolver/pzm/layer/webapp/deploy/releases/3deb570d0ee6c4a95c7f978d9e8080cacaf59195
  (spawning 1 new process...)
  Requests in queue: 1
  * PID: 6727    Sessions: 0       Processed: 5       Uptime: 5m 56s
    CPU: 0%      Memory  : 144M    Last used: 0s ago
    Disabling...
    URL     : http://127.0.0.1:58093
    Password: <redacted>

### mbuf stats

nfree_mbuf_blockq  : 8
nactive_mbuf_blockq: 10
mbuf_block_chunk_size: 512

### Pool state (XML)
<?xml version="1.0" encoding="iso8859-1" ?>
<info version="3"><passenger_version>5.0.6</passenger_version><process_count>1</process_count><max>5</max><capacity_used>2</capacity_used><get_wait_list_size>0</get_wait_list_size><get_wait_list></get_wait_list><supergroups><supergroup><name>/opt/agsolver/pzm/layer/webapp/deploy/releases/3deb570d0ee6c4a95c7f978d9e8080cacaf59195</name><state>READY</state><get_wait_list_size>0</get_wait_list_size><capacity_used>2</capacity_used><secret>NbKB4ea4I3gykQXKWldQcbSuU4fnyHrfNvUiKLobkRE</secret><group default="true"><name>/opt/agsolver/pzm/layer/webapp/deploy/releases/3deb570d0ee6c4a95c7f978d9e8080cacaf59195&#35;default</name><component_name>default</component_name><app_root>/opt/agsolver/pzm/layer/webapp/deploy/releases/3deb570d0ee6c4a95c7f978d9e8080cacaf59195</app_root><app_type>rack</app_type><environment>web_development</environment><uuid>Abf3WCrXKZy2LX44UHPK</uuid><enabled_process_count>0</enabled_process_count><disabling_process_count>1</disabling_process_count><disabled_process_count>0</disabled_process_count><capacity_used>2</capacity_used><get_wait_list_size>1</get_wait_list_size><disable_wait_list_size>1</disable_wait_list_size><processes_being_spawned>1</processes_being_spawned><spawning/><secret>PVISxrAv7GG5I631</secret><life_status>ALIVE</life_status><options><app_root>/opt/agsolver/pzm/layer/webapp/deploy/releases/3deb570d0ee6c4a95c7f978d9e8080cacaf59195</app_root><app_group_name>/opt/agsolver/pzm/layer/webapp/deploy/releases/3deb570d0ee6c4a95c7f978d9e8080cacaf59195</app_group_name><app_type>rack</app_type><start_command>ruby&#9;/usr/share/passenger/helper-scripts/rack-loader.rb</start_command><startup_file>config.ru</startup_file><process_title>Passenger RubyApp</process_title><log_level>3</log_level><start_timeout>90000</start_timeout><environment>web_development</environment><base_uri>/</base_uri><spawn_method>smart</spawn_method><default_user>nobody</default_user><default_group>nogroup</default_group><ruby>ruby</ruby><python>python</python><nodejs>node</nodejs><logging_agent_address>unix:/tmp/passenger.G2OIoli/agents.s/logging</logging_agent_address><logging_agent_username>logging</logging_agent_username><logging_agent_password>F5NwNnDl4ZAtoWLs59wt5Utf</logging_agent_password><debugger>false</debugger><analytics>false</analytics><group_secret>PVISxrAv7GG5I631</group_secret><min_processes>1</min_processes><max_processes>0</max_processes><max_preloader_idle_time>-1</max_preloader_idle_time><max_out_of_band_work_instances>1</max_out_of_band_work_instances></options><processes><process><pid>6727</pid><sticky_session_id>122868425</sticky_session_id><gupid>16b4ae6-zahIhDp9RZ</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>5</processed><spawner_creation_time>1428523890506400</spawner_creation_time><spawn_start_time>1428524557766818</spawn_start_time><spawn_end_time>1428524557774664</spawn_end_time><last_used>1428524913345597</last_used><last_used_desc>0s ago</last_used_desc><uptime>5m 56s</uptime><life_status>ALIVE</life_status><enabled>DISABLING</enabled><has_metrics>true</has_metrics><cpu>0</cpu><rss>152004</rss><pss>149439</pss><private_dirty>148388</private_dirty><swap>0</swap><real_memory>148388</real_memory><vmsize>1014676</vmsize><process_group_id>6231</process_group_id><command>Passenger RubyApp: /opt/agsolver/pzm/layer/webapp/deploy/releases/3deb570d0ee6c4a95c7f978d9e8080cacaf59195</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.G2OIoli/apps.s/ruby.rTYxfqY9dSUoIj9SNUXooqg2yfeydyd4Z9WJrPBFZoSdPEPLDfiT2rGIh46P8sb</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:58093</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process></processes></group></supergroup></supergroups></info>

--------------------------------------
[ pid=6231 ] Open files and file descriptors:
COMMAND    PID   USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
Passenger 6231 nobody  cwd    DIR              202,1     4096  180790 /opt/agsolver/pzm/layer/webapp/deploy/releases/3deb570d0ee6c4a95c7f978d9e8080cacaf59195
Passenger 6231 nobody  rtd    DIR              202,1     4096       2 /
Passenger 6231 nobody  txt    REG              202,1  3907864   17402 /usr/lib/passenger/support-binaries/PassengerAgent
Passenger 6231 nobody  mem    REG              202,1    47712  396074 /lib/x86_64-linux-gnu/libnss_files-2.19.so
Passenger 6231 nobody  mem    REG              202,1    47760  396076 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
Passenger 6231 nobody  mem    REG              202,1    97296  396071 /lib/x86_64-linux-gnu/libnsl-2.19.so
Passenger 6231 nobody  mem    REG              202,1    39824  396072 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
Passenger 6231 nobody  mem    REG              202,1    43368  396045 /lib/x86_64-linux-gnu/libcrypt-2.19.so
Passenger 6231 nobody  mem    REG              202,1   754880    8130 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
Passenger 6231 nobody  mem    REG              202,1   295816    8135 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
Passenger 6231 nobody  mem    REG              202,1    56768    8088 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
Passenger 6231 nobody  mem    REG              202,1   166040    8089 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
Passenger 6231 nobody  mem    REG              202,1    30944    8067 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
Passenger 6231 nobody  mem    REG              202,1    85680    6820 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
Passenger 6231 nobody  mem    REG              202,1   204104    8050 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
Passenger 6231 nobody  mem    REG              202,1   657712    8082 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
Passenger 6231 nobody  mem    REG              202,1   552584    8091 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
Passenger 6231 nobody  mem    REG              202,1    35856    8102 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
Passenger 6231 nobody  mem    REG              202,1    14256  396060 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
Passenger 6231 nobody  mem    REG              202,1    18416  396057 /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
Passenger 6231 nobody  mem    REG              202,1   268368    8114 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
Passenger 6231 nobody  mem    REG              202,1    80256   15851 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
Passenger 6231 nobody  mem    REG              202,1   251872    8090 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
Passenger 6231 nobody  mem    REG              202,1   109296    8126 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
Passenger 6231 nobody  mem    REG              202,1   101240  396095 /lib/x86_64-linux-gnu/libresolv-2.19.so
Passenger 6231 nobody  mem    REG              202,1    43672    8104 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
Passenger 6231 nobody  mem    REG              202,1    14592  396044 /lib/x86_64-linux-gnu/libcom_err.so.2.1
Passenger 6231 nobody  mem    REG              202,1   186824    8101 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
Passenger 6231 nobody  mem    REG              202,1   831616    8103 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
Passenger 6231 nobody  mem    REG              202,1    14664  396049 /lib/x86_64-linux-gnu/libdl-2.19.so
Passenger 6231 nobody  mem    REG              202,1   524704  393687 /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
Passenger 6231 nobody  mem    REG              202,1   775112    6819 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
Passenger 6231 nobody  mem    REG              202,1   323056    8106 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
Passenger 6231 nobody  mem    REG              202,1    59640    8105 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
Passenger 6231 nobody  mem    REG              202,1   285848    8086 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
Passenger 6231 nobody  mem    REG              202,1  1930528  397473 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Passenger 6231 nobody  mem    REG              202,1   383112  397472 /lib/x86_64-linux-gnu/libssl.so.1.0.0
Passenger 6231 nobody  mem    REG              202,1   105696    8125 /usr/lib/x86_64-linux-gnu/librtmp.so.0
Passenger 6231 nobody  mem    REG              202,1   207128    8100 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
Passenger 6231 nobody  mem    REG              202,1  1840928  396040 /lib/x86_64-linux-gnu/libc-2.19.so
Passenger 6231 nobody  mem    REG              202,1    90080  396054 /lib/x86_64-linux-gnu/libgcc_s.so.1
Passenger 6231 nobody  mem    REG              202,1  1071552  396063 /lib/x86_64-linux-gnu/libm-2.19.so
Passenger 6231 nobody  mem    REG              202,1   979056    8131 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
Passenger 6231 nobody  mem    REG              202,1    31792  396096 /lib/x86_64-linux-gnu/librt-2.19.so
Passenger 6231 nobody  mem    REG              202,1   141574  396093 /lib/x86_64-linux-gnu/libpthread-2.19.so
Passenger 6231 nobody  mem    REG              202,1   100728  396113 /lib/x86_64-linux-gnu/libz.so.1.2.8
Passenger 6231 nobody  mem    REG              202,1   422696    8056 /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
Passenger 6231 nobody  mem    REG              202,1   149120  396030 /lib/x86_64-linux-gnu/ld-2.19.so
Passenger 6231 nobody    0r   CHR                1,3      0t0    5430 /dev/null
Passenger 6231 nobody    1w  FIFO                0,8      0t0 1006732 pipe
Passenger 6231 nobody    2w  FIFO                0,8      0t0 1006732 pipe
Passenger 6231 nobody    3u  unix 0x0000000000000000      0t0  998217 socket
Passenger 6231 nobody    5w  FIFO                0,8      0t0 1006732 pipe
Passenger 6231 nobody    8r  FIFO                0,8      0t0  998250 pipe
Passenger 6231 nobody    9w  FIFO                0,8      0t0  998250 pipe
Passenger 6231 nobody   10r  FIFO                0,8      0t0  998251 pipe
Passenger 6231 nobody   11w  FIFO                0,8      0t0  998251 pipe
Passenger 6231 nobody   12r  FIFO                0,8      0t0  998252 pipe
Passenger 6231 nobody   13w  FIFO                0,8      0t0  998252 pipe
Passenger 6231 nobody   14u  unix 0x0000000000000000      0t0  998253 /tmp/passenger.G2OIoli/agents.s/server
Passenger 6231 nobody   15u  unix 0x0000000000000000      0t0  998254 /var/run/passenger/passenger.sock
Passenger 6231 nobody   16u  unix 0x0000000000000000      0t0  998256 /tmp/passenger.G2OIoli/agents.s/server_admin
Passenger 6231 nobody   17r   CHR                1,9      0t0    5435 /dev/urandom
Passenger 6231 nobody   18r   CHR                1,9      0t0    5435 /dev/urandom
Passenger 6231 nobody   19u  0000                0,9        0    5385 anon_inode
Passenger 6231 nobody   20u  0000                0,9        0    5385 anon_inode
Passenger 6231 nobody   21u  0000                0,9        0    5385 anon_inode
Passenger 6231 nobody   22u  0000                0,9        0    5385 anon_inode
Passenger 6231 nobody   23u  0000                0,9        0    5385 anon_inode
Passenger 6231 nobody   24u  0000                0,9        0    5385 anon_inode
Passenger 6231 nobody   25u  unix 0x0000000000000000      0t0 1006730 /var/run/passenger/passenger.sock
Passenger 6231 nobody   26u  unix 0x0000000000000000      0t0 1003520 socket
Passenger 6231 nobody   27u  unix 0x0000000000000000      0t0 1003355 socket
Passenger 6231 nobody   28r  FIFO                0,8      0t0 1003356 pipe
Passenger 6231 nobody   29u  unix 0x0000000000000000      0t0 1003526 socket
Passenger 6231 nobody   31u  unix 0x0000000000000000      0t0 1006678 /var/run/passenger/passenger.sock
Passenger 6231 nobody   33u  unix 0x0000000000000000      0t0 1006681 socket
Passenger 6231 nobody   34r  FIFO                0,8      0t0 1006682 pipe
--------------------------------------
[ pid=6231 ] Dumping a more detailed backtrace with crash-watch...
Found gdb at: /usr/bin/gdb
*******************************************************
*
*    Current thread (1) backtrace
*
*******************************************************

    #0  0x00007fa2f403bda3 in select () at ../sysdeps/unix/syscall-template.S:81
    No locals.
    #1  0x000000000068b140 in ?? ()
    No symbol table info available.
    #2  0x00000000005482fe in ?? ()
    No symbol table info available.
    #3  0x0000000000547bdd in ?? ()
    No symbol table info available.
    #4  0x0000000000549149 in ?? ()
    No symbol table info available.
    #5  0x000000000054c733 in ?? ()
    No symbol table info available.
    #6  0x000000000043fb5f in _start ()
    No symbol table info available.

*******************************************************
*
*    All thread backtraces
*
*******************************************************

########### Thread 9 (Thread 0x7fa2f55d7700 (LWP 6233)) ###########

    #0  0x00007fa2f4d46b9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
    No locals.
    #1  0x000000000068bf18 in ?? ()
    No symbol table info available.
    #2  0x000000000068beb4 in ?? ()
    No symbol table info available.
    #3  0x00000000005679cf in ?? ()
    No symbol table info available.
    #4  0x00000000005d4348 in ?? ()
    No symbol table info available.
    #5  0x00000000005ce71d in ?? ()
    No symbol table info available.
    #6  0x00000000005c7bd3 in ?? ()
    No symbol table info available.
    #7  0x0000000000453e90 in ?? ()
    No symbol table info available.
    #8  0x0000000000477043 in ?? ()
    No symbol table info available.
    #9  0x00000000004c98d4 in ?? ()
    No symbol table info available.
    #10 0x00000000004c47cb in ?? ()
    No symbol table info available.
    #11 0x00000000004bf445 in ?? ()
    No symbol table info available.
    #12 0x00000000004b84bf in ?? ()
    No symbol table info available.
    #13 0x0000000000453e90 in ?? ()
    No symbol table info available.
    #14 0x00000000006869f5 in ?? ()
    No symbol table info available.
    #15 0x00000000004d0aec in ?? ()
    No symbol table info available.
    #16 0x00000000004d06d7 in ?? ()
    No symbol table info available.
    #17 0x00000000004cfec2 in ?? ()
    No symbol table info available.
    #18 0x0000000000681deb in ?? ()
    No symbol table info available.
    #19 0x00007fa2f4d3f182 in start_thread (arg=0x7fa2f55d7700)
        at pthread_create.c:312
            __res = <optimized out>
            pd = 0x7fa2f55d7700
            now = <optimized out>
            unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140337877972736, 
                    4209129618161667268, 0, 0, 140337877973440, 140337877972736, 
                    -4238943137447670588, -4238946414689776444}, 
                  mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
                data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
            not_first_call = <optimized out>
            pagesize_m1 = <optimized out>
            sp = <optimized out>
            freesize = <optimized out>
            __PRETTY_FUNCTION__ = "start_thread"
    #20 0x00007fa2f404447d in clone ()
        at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
    No locals.

########### Thread 8 (Thread 0x7fa2f5596700 (LWP 6234)) ###########

    #0  pthread_cond_timedwait@@GLIBC_2.3.2 ()
        at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
    No locals.
    #1  0x00000000004763de in ?? ()
    No symbol table info available.
    #2  0x0000000000475a57 in ?? ()
    No symbol table info available.
    #3  0x000000000059bc0e in ?? ()
    No symbol table info available.
    #4  0x0000000000569c64 in ?? ()
    No symbol table info available.
    #5  0x00000000005d4348 in ?? ()
    No symbol table info available.
    #6  0x00000000005ce71d in ?? ()
    No symbol table info available.
    #7  0x00000000005c7bd3 in ?? ()
    No symbol table info available.
    #8  0x0000000000453e90 in ?? ()
    No symbol table info available.
    #9  0x0000000000477043 in ?? ()
    No symbol table info available.
    #10 0x00000000004c98d4 in ?? ()
    No symbol table info available.
    #11 0x00000000004c47cb in ?? ()
    No symbol table info available.
    #12 0x00000000004bf445 in ?? ()
    No symbol table info available.
    #13 0x00000000004b84bf in ?? ()
    No symbol table info available.
    #14 0x0000000000453e90 in ?? ()
    No symbol table info available.
    #15 0x00000000006869f5 in ?? ()
    No symbol table info available.
    #16 0x00000000004d0aec in ?? ()
    No symbol table info available.
    #17 0x00000000004d06d7 in ?? ()
    No symbol table info available.
    #18 0x00000000004cfec2 in ?? ()
    No symbol table info available.
    #19 0x0000000000681deb in ?? ()
    No symbol table info available.
    #20 0x00007fa2f4d3f182 in start_thread (arg=0x7fa2f5596700)
        at pthread_create.c:312
            __res = <optimized out>
            pd = 0x7fa2f5596700
            now = <optimized out>
            unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140337877706496, 
                    4209129618161667268, 0, 0, 140337877707200, 140337877706496, 
                    -4238943173418021692, -4238946414689776444}, 
                  mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
                data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
            not_first_call = <optimized out>
            pagesize_m1 = <optimized out>
            sp = <optimized out>
            freesize = <optimized out>
            __PRETTY_FUNCTION__ = "start_thread"
    #21 0x00007fa2f404447d in clone ()
        at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
    No locals.

########### Thread 7 (Thread 0x7fa2eee0a700 (LWP 6236)) ###########

    #0  0x00007fa2f4d4720b in raise (sig=19)
        at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
            resultvar = 0
            pid = <optimized out>
    #1  0x00000000004d4201 in ?? ()
    No symbol table info available.
    #2  <signal handler called>
    No locals.
    #3  0x000000000048152a in ?? ()
    No symbol table info available.
    #4  0x0000000000486c40 in ?? ()
    No symbol table info available.
    #5  0x000000000048bcc7 in ?? ()
    No symbol table info available.
    #6  0x000000000056d6fc in ?? ()
    No symbol table info available.
    #7  0x0000000000578f85 in ?? ()
    No symbol table info available.
    #8  0x0000000000574994 in ?? ()
    No symbol table info available.
    #9  0x00000000005e4ce2 in ?? ()
    No symbol table info available.
    #10 0x00000000005e0b1b in ?? ()
    No symbol table info available.
    #11 0x00000000005cbde6 in ?? ()
    No symbol table info available.
    #12 0x000000000050373f in ?? ()
    No symbol table info available.
    #13 0x0000000000504bee in ?? ()
    No symbol table info available.
    #14 0x0000000000506371 in ?? ()
    No symbol table info available.
    #15 0x000000000050606b in ?? ()
    No symbol table info available.
    #16 0x0000000000506025 in ?? ()
    No symbol table info available.
    #17 0x0000000000691a28 in ?? ()
    No symbol table info available.
    #18 0x0000000000692902 in ?? ()
    No symbol table info available.
    #19 0x0000000000540868 in ?? ()
    No symbol table info available.
    #20 0x00000000005d58ff in ?? ()
    No symbol table info available.
    #21 0x00000000005d0461 in ?? ()
    No symbol table info available.
    #22 0x00000000005caaa4 in ?? ()
    No symbol table info available.
    #23 0x0000000000453e90 in ?? ()
    No symbol table info available.
    #24 0x00000000006869f5 in ?? ()
    No symbol table info available.
    #25 0x00000000004d0aec in ?? ()
    No symbol table info available.
    #26 0x00000000004d06d7 in ?? ()
    No symbol table info available.
    #27 0x00000000004cfec2 in ?? ()
    No symbol table info available.
    #28 0x0000000000681deb in ?? ()
    No symbol table info available.
    #29 0x00007fa2f4d3f182 in start_thread (arg=0x7fa2eee0a700)
        at pthread_create.c:312
            __res = <optimized out>
            pd = 0x7fa2eee0a700
            now = <optimized out>
            unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140337769129728, 
                    4209129618161667268, 0, 0, 140337769130432, 140337769129728, 
                    -4239003429661700924, -4238946414689776444}, 
                  mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
                data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
            not_first_call = <optimized out>
            pagesize_m1 = <optimized out>
            sp = <optimized out>
            freesize = <optimized out>
            __PRETTY_FUNCTION__ = "start_thread"
    #30 0x00007fa2f404447d in clone ()
        at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
    No locals.

########### Thread 6 (Thread 0x7fa2ee609700 (LWP 6237)) ###########

    #0  0x00007fa2f4044b13 in epoll_wait ()
        at ../sysdeps/unix/syscall-template.S:81
    No locals.
    #1  0x000000000068f2f0 in ?? ()
    No symbol table info available.
    #2  0x0000000000692805 in ?? ()
    No symbol table info available.
    #3  0x0000000000540868 in ?? ()
    No symbol table info available.
    #4  0x00000000005d58ff in ?? ()
    No symbol table info available.
    #5  0x00000000005d0461 in ?? ()
    No symbol table info available.
    #6  0x00000000005caaa4 in ?? ()
    No symbol table info available.
    #7  0x0000000000453e90 in ?? ()
    No symbol table info available.
    #8  0x00000000006869f5 in ?? ()
    No symbol table info available.
    #9  0x00000000004d0aec in ?? ()
    No symbol table info available.
    #10 0x00000000004d06d7 in ?? ()
    No symbol table info available.
    #11 0x00000000004cfec2 in ?? ()
    No symbol table info available.
    #12 0x0000000000681deb in ?? ()
    No symbol table info available.
    #13 0x00007fa2f4d3f182 in start_thread (arg=0x7fa2ee609700)
        at pthread_create.c:312
            __res = <optimized out>
            pd = 0x7fa2ee609700
            now = <optimized out>
            unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140337760737024, 
                    4209129618161667268, 0, 0, 140337760737728, 140337760737024, 
                    -4239002329613202236, -4238946414689776444}, 
                  mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
                data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
            not_first_call = <optimized out>
            pagesize_m1 = <optimized out>
            sp = <optimized out>
            freesize = <optimized out>
            __PRETTY_FUNCTION__ = "start_thread"
    #14 0x00007fa2f404447d in clone ()
        at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
    No locals.

########### Thread 5 (Thread 0x7fa2f5438700 (LWP 6722)) ###########

    #0  0x00007fa2f4d463bd in read () at ../sysdeps/unix/syscall-template.S:81
    No locals.
    #1  0x00000000006896e0 in ?? ()
    No symbol table info available.
    #2  0x00000000004744a7 in ?? ()
    No symbol table info available.
    #3  0x0000000000474370 in ?? ()
    No symbol table info available.
    #4  0x00000000004c96aa in ?? ()
    No symbol table info available.
    #5  0x00000000004c42eb in ?? ()
    No symbol table info available.
    #6  0x00000000004bebc0 in ?? ()
    No symbol table info available.
    #7  0x0000000000453e90 in ?? ()
    No symbol table info available.
    #8  0x00000000006869f5 in ?? ()
    No symbol table info available.
    #9  0x00000000004d0aec in ?? ()
    No symbol table info available.
    #10 0x00000000004d06d7 in ?? ()
    No symbol table info available.
    #11 0x00000000004cfec2 in ?? ()
    No symbol table info available.
    #12 0x0000000000681deb in ?? ()
    No symbol table info available.
    #13 0x00007fa2f4d3f182 in start_thread (arg=0x7fa2f5438700)
        at pthread_create.c:312
            __res = <optimized out>
            pd = 0x7fa2f5438700
            now = <optimized out>
            unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140337876272896, 
                    4209129618161667268, 0, 0, 140337876273600, 140337876272896, 
                    -4238943258243625788, -4238946414689776444}, 
                  mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
                data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
            not_first_call = <optimized out>
            pagesize_m1 = <optimized out>
            sp = <optimized out>
            freesize = <optimized out>
            __PRETTY_FUNCTION__ = "start_thread"
    #14 0x00007fa2f404447d in clone ()
        at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
    No locals.

########### Thread 4 (Thread 0x7fa2ed5cc700 (LWP 6730)) ###########

    #0  0x00007fa2f4d463bd in read () at ../sysdeps/unix/syscall-template.S:81
    No locals.
    #1  0x00000000006896e0 in ?? ()
    No symbol table info available.
    #2  0x00000000004744a7 in ?? ()
    No symbol table info available.
    #3  0x0000000000474370 in ?? ()
    No symbol table info available.
    #4  0x00000000004c96aa in ?? ()
    No symbol table info available.
    #5  0x00000000004c42eb in ?? ()
    No symbol table info available.
    #6  0x00000000004bebc0 in ?? ()
    No symbol table info available.
    #7  0x0000000000453e90 in ?? ()
    No symbol table info available.
    #8  0x00000000006869f5 in ?? ()
    No symbol table info available.
    #9  0x00000000004d0aec in ?? ()
    No symbol table info available.
    #10 0x00000000004d06d7 in ?? ()
    No symbol table info available.
    #11 0x00000000004cfec2 in ?? ()
    No symbol table info available.
    #12 0x0000000000681deb in ?? ()
    No symbol table info available.
    #13 0x00007fa2f4d3f182 in start_thread (arg=0x7fa2ed5cc700)
        at pthread_create.c:312
            __res = <optimized out>
            pd = 0x7fa2ed5cc700
            now = <optimized out>
            unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140337743709952, 
                    4209129618161667268, 0, 0, 140337743710656, 140337743709952, 
                    -4238995903805256508, -4238946414689776444}, 
                  mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
                data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
            not_first_call = <optimized out>
            pagesize_m1 = <optimized out>
            sp = <optimized out>
            freesize = <optimized out>
            __PRETTY_FUNCTION__ = "start_thread"
    #14 0x00007fa2f404447d in clone ()
        at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
    No locals.

########### Thread 3 (Thread 0x7fa2f5479700 (LWP 6984)) ###########

    #0  0x00007fa2f403712d in poll () at ../sysdeps/unix/syscall-template.S:81
    No locals.
    #1  0x000000000068b2be in ?? ()
    No symbol table info available.
    #2  0x0000000000449fb6 in ?? ()
    No symbol table info available.
    #3  0x000000000044a159 in ?? ()
    No symbol table info available.
    #4  0x0000000000551bda in ?? ()
    No symbol table info available.
    #5  0x0000000000552224 in ?? ()
    No symbol table info available.
    #6  0x000000000059800f in ?? ()
    No symbol table info available.
    #7  0x000000000055f0f1 in ?? ()
    No symbol table info available.
    #8  0x000000000055c7b3 in ?? ()
    No symbol table info available.
    #9  0x0000000000560ed2 in ?? ()
    No symbol table info available.
    #10 0x000000000046f745 in ?? ()
    No symbol table info available.
    #11 0x000000000046f135 in ?? ()
    No symbol table info available.
    #12 0x00000000004cb354 in ?? ()
    No symbol table info available.
    #13 0x00000000004c6c34 in ?? ()
    No symbol table info available.
    #14 0x00000000004c20d1 in ?? ()
    No symbol table info available.
    #15 0x00000000004bbf4d in ?? ()
    No symbol table info available.
    #16 0x0000000000453e90 in ?? ()
    No symbol table info available.
    #17 0x0000000000477043 in ?? ()
    No symbol table info available.
    #18 0x00000000004c98d4 in ?? ()
    No symbol table info available.
    #19 0x00000000004c47cb in ?? ()
    No symbol table info available.
    #20 0x00000000004bf445 in ?? ()
    No symbol table info available.
    #21 0x00000000004b84bf in ?? ()
    No symbol table info available.
    #22 0x0000000000453e90 in ?? ()
    No symbol table info available.
    #23 0x00000000006869f5 in ?? ()
    No symbol table info available.
    #24 0x00000000004d0aec in ?? ()
    No symbol table info available.
    #25 0x00000000004d06d7 in ?? ()
    No symbol table info available.
    #26 0x00000000004cfec2 in ?? ()
    No symbol table info available.
    #27 0x0000000000681deb in ?? ()
    No symbol table info available.
    #28 0x00007fa2f4d3f182 in start_thread (arg=0x7fa2f5479700)
        at pthread_create.c:312
            __res = <optimized out>
            pd = 0x7fa2f5479700
            now = <optimized out>
            unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140337876539136, 
                    4209129618161667268, 0, 0, 140337876539840, 140337876539136, 
                    -4238943222273274684, -4238946414689776444}, 
                  mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
                data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
            not_first_call = <optimized out>
            pagesize_m1 = <optimized out>
            sp = <optimized out>
            freesize = <optimized out>
            __PRETTY_FUNCTION__ = "start_thread"
    #29 0x00007fa2f404447d in clone ()
        at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
    No locals.

########### Thread 2 (Thread 0x7fa2f548a700 (LWP 6986)) ###########

    #0  0x00007fa2f4d463bd in read () at ../sysdeps/unix/syscall-template.S:81
    No locals.
    #1  0x00000000006896e0 in ?? ()
    No symbol table info available.
    #2  0x00000000005522f3 in ?? ()
    No symbol table info available.
    #3  0x00000000005d7297 in ?? ()
    No symbol table info available.
    #4  0x00000000005d352c in ?? ()
    No symbol table info available.
    #5  0x00000000005cdc4f in ?? ()
    No symbol table info available.
    #6  0x00000000005c6ab4 in ?? ()
    No symbol table info available.
    #7  0x0000000000453e90 in ?? ()
    No symbol table info available.
    #8  0x00000000006869f5 in ?? ()
    No symbol table info available.
    #9  0x00000000004d0aec in ?? ()
    No symbol table info available.
    #10 0x00000000004d06d7 in ?? ()
    No symbol table info available.
    #11 0x00000000004cfec2 in ?? ()
    No symbol table info available.
    #12 0x0000000000681deb in ?? ()
    No symbol table info available.
    #13 0x00007fa2f4d3f182 in start_thread (arg=0x7fa2f548a700)
        at pthread_create.c:312
            __res = <optimized out>
            pd = 0x7fa2f548a700
            now = <optimized out>
            unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140337876608768, 
                    4209129618161667268, 0, 0, 140337876609472, 140337876608768, 
                    -4238943300119556924, -4238946414689776444}, 
                  mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
                data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
            not_first_call = <optimized out>
            pagesize_m1 = <optimized out>
            sp = <optimized out>
            freesize = <optimized out>
            __PRETTY_FUNCTION__ = "start_thread"
    #14 0x00007fa2f404447d in clone ()
        at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
    No locals.

########### Thread 1 (Thread 0x7fa2f55d97c0 (LWP 6231)) ###########

    #0  0x00007fa2f403bda3 in select () at ../sysdeps/unix/syscall-template.S:81
    No locals.
    #1  0x000000000068b140 in ?? ()
    No symbol table info available.
    #2  0x00000000005482fe in ?? ()
    No symbol table info available.
    #3  0x0000000000547bdd in ?? ()
    No symbol table info available.
    #4  0x0000000000549149 in ?? ()
    No symbol table info available.
    #5  0x000000000054c733 in ?? ()
    No symbol table info available.
    #6  0x000000000043fb5f in _start ()
    No symbol table info available.
FooBarWidget commented 9 years ago

Thank you for reporting this. We will investigate this issue.

FooBarWidget commented 9 years ago

Hi @karlsvec. Due to lack of debugging symbols, it isn't entirely clear what the cause of this problem is. The few readable backtraces in the report seem to make little sense. However, we fixed a crash a few days ago which was related to handling of sub-URI deployments. Are you using sub-URI deployments?

We've also modified the Debian packages to include a -dbg version which includes debugging symbols, so that in the future users can send us better bug reports.

karlsvec commented 9 years ago

@FooBarWidget I'm not using a sub-URI deployment, but I'll try the stable-5.0 branch and see if it makes a difference.

karlsvec commented 9 years ago

@FooBarWidget I'm seeing the SIGSEGV crash in the latest stable-5.0 branch as well:

UPDATE: Instead of continuing to spam this issue with long crash logs embedded in comments, here's a link to a Gist: https://gist.github.com/karlsvec/db8d0354ab540b987737. This Gist contains the most recent crash log, including the gdb trace. Passenger was built from source from the stable-5.0 branch, without optimizations.

[ pid=13778, timestamp=1429632981 ] Process aborted! signo=SIGSEGV(11), reason=SEGV_MAPERR, si_addr=0x74, randomSeed=1429630885
[ pid=13778 ] Crash log dumped to /var/tmp/passenger-crash-log.1429632981
[ pid=13778 ] Date, uname and ulimits:
Tue Apr 21 16:16:21 UTC 2015
Linux 3.16.0-34-generic #47~14.04.1-Ubuntu SMP Fri Apr 10 17:49:16 UTC 2015 x86_64 x86_64
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     0
memory(kbytes)       unlimited
locked memory(kbytes) 64
process              15725
nofiles              1024
vmemory(kbytes)      unlimited
locks                unlimited
[ pid=13778 ] Phusion Passenger version: 5.0.7
[ pid=13778 ] libc backtrace available!
--------------------------------------
[ pid=13778 ] Backtrace with 36 frames:
Using backtrace sanitizer.
PassengerAgent server(dumpBacktrace at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/agents/Base.cpp:551) [0x4d27fb]
PassengerAgent server(runInSubprocessWithTimeLimit at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/agents/Base.cpp:398) [0x4d20a7]
PassengerAgent server(dumpDiagnostics at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/agents/Base.cpp:740) [0x4d33ab]
PassengerAgent server(abortHandler at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/agents/Base.cpp:985) [0x4d4065]
/lib/x86_64-linux-gnu/libpthread.so.0(?? at ??:0; +0x10340) [0x7ffa060e1340]
PassengerAgent server(Passenger::ApplicationPool2::Socket::busyness() const at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ApplicationPool2/Socket.h:233) [0x47fe48]
PassengerAgent server(Passenger::ApplicationPool2::Process::findSessionSocketWithLowestBusyness() const at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ApplicationPool2/Process.h:443) [0x480f51]
PassengerAgent server(Passenger::ApplicationPool2::Process::newSession(unsigned long long) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ApplicationPool2/Process.h:582) [0x4815e8]
PassengerAgent server(Passenger::ApplicationPool2::Group::newSession(Passenger::ApplicationPool2::Process*, unsigned long long) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ApplicationPool2/Group.h:424) [0x483d53]
PassengerAgent server(Passenger::ApplicationPool2::Group::get(Passenger::ApplicationPool2::Options const&, Passenger::ApplicationPool2::GetCallback const&, boost::container::vector<boost::function<void ()>, std::allocator<boost::function<void ()> > >&) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ApplicationPool2/Group.h:1013) [0x486cc2]
PassengerAgent server(Passenger::ApplicationPool2::SuperGroup::get(Passenger::ApplicationPool2::Options const&, Passenger::ApplicationPool2::GetCallback const&, boost::container::vector<boost::function<void ()>, std::allocator<boost::function<void ()> > >&) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ApplicationPool2/SuperGroup.h:619) [0x48bd17]
PassengerAgent server(Passenger::ApplicationPool2::Pool::asyncGet(Passenger::ApplicationPool2::Options const&, Passenger::ApplicationPool2::GetCallback const&, bool) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ApplicationPool2/Pool.h:567) [0x56da24]
PassengerAgent server(Passenger::RequestHandler::checkoutSession(Passenger::Client*, Passenger::Request*) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/agents/HelperAgent/RequestHandler/CheckoutSession.cpp:54) [0x579587]
PassengerAgent server(Passenger::RequestHandler::onRequestBegin(Passenger::Client*, Passenger::Request*) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/agents/HelperAgent/RequestHandler/InitRequest.cpp:80) [0x574d26]
PassengerAgent server(Passenger::ServerKit::HttpServer<Passenger::RequestHandler, Passenger::Client>::processClientDataWhenParsingHeaders(Passenger::Client*, Passenger::Request*, Passenger::MemoryKit::mbuf const&, int) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ServerKit/HttpServer.h:282) [0x5e5450]
PassengerAgent server(Passenger::ServerKit::HttpServer<Passenger::RequestHandler, Passenger::Client>::onClientDataReceived(Passenger::Client*, Passenger::MemoryKit::mbuf const&, int) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ServerKit/HttpServer.h:899) [0x5e1289]
PassengerAgent server(Passenger::ServerKit::BaseServer<Passenger::RequestHandler, Passenger::Client>::_onClientDataReceived(Passenger::ServerKit::Channel*, Passenger::MemoryKit::mbuf const&, int) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ServerKit/Server.h:483) [0x5cc554]
PassengerAgent server(Passenger::ServerKit::Channel::callDataCallbackWithoutRefGuard() at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ServerKit/Channel.h:280) [0x503a67]
PassengerAgent server(Passenger::ServerKit::Channel::feedWithoutRefGuard(Passenger::MemoryKit::mbuf&&) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ServerKit/Channel.h:511) [0x504f16]
PassengerAgent server(Passenger::ServerKit::FdSourceChannel::onReadableWithoutRefGuard() at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ServerKit/FdSourceChannel.h:91) [0x506699]
PassengerAgent server(Passenger::ServerKit::FdSourceChannel::onReadable(ev_io*, int) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ServerKit/FdSourceChannel.h:55 (discriminator 1)) [0x506393]
PassengerAgent server(Passenger::ServerKit::FdSourceChannel::_onReadable(ev_loop*, ev_io*, int) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/ServerKit/FdSourceChannel.h:51) [0x50634d]
PassengerAgent server(ev_invoke_pending at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/libev/ev.c:3001 (discriminator 1)) [0x692196]
PassengerAgent server(ev_run at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/libev/ev.c:3408) [0x693070]
PassengerAgent server(startBackgroundLoop at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/common/BackgroundEventLoop.cpp:69 (discriminator 1)) [0x540b90]
PassengerAgent server(void boost::_bi::list1<boost::_bi::value<Passenger::BackgroundEventLoop*> >::operator()<void (*)(Passenger::BackgroundEventLoop*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(Passenger::BackgroundEventLoop*), boost::_bi::list0&, int) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/boost/bind/bind.hpp:254) [0x5d606d]
PassengerAgent server(boost::_bi::bind_t<void, void (*)(Passenger::BackgroundEventLoop*), boost::_bi::list1<boost::_bi::value<Passenger::BackgroundEventLoop*> > >::operator()() at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/boost/bind/bind_template.hpp:21) [0x5d0bcf]
PassengerAgent server(boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, void (*)(Passenger::BackgroundEventLoop*), boost::_bi::list1<boost::_bi::value<Passenger::BackgroundEventLoop*> > >, void>::invoke(boost::detail::function::function_buffer&) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/boost/function/function_template.hpp:154) [0x5cb212]
PassengerAgent server(boost::function0<void>::operator()() const at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/boost/function/function_template.hpp:767) [0x453ee0]
PassengerAgent server(oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/oxt/implementation.cpp:415) [0x689c8d]
PassengerAgent server(void boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > >::operator()<void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list0&, int) at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/boost/bind/bind.hpp:313 (discriminator 1)) [0x4d0b3c]
PassengerAgent server(boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > >::operator()() at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/boost/bind/bind_template.hpp:21) [0x4d0727]
PassengerAgent server(boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/boost/thread/detail/thread.hpp:118) [0x4cff12]
PassengerAgent server(thread_proxy at /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/ext/boost/libs/thread/src/pthread/thread.cpp:181) [0x681ddf]
/lib/x86_64-linux-gnu/libpthread.so.0(?? at ??:0; +0x8182) [0x7ffa060d9182]
/lib/x86_64-linux-gnu/libc.so.6(?? at ??:0; clone+0x6d) [0x7ffa053de47d]
--------------------------------------
[ pid=13778 ] Dumping additional diagnostical information...
--------------------------------------
### Backtraces
Thread 'Main thread' (0x7ffa069737c0, LWP 13778):
     in 'void waitForExitEvent()' (Main.cpp:788)
     in 'void mainLoop()' (Main.cpp:681)
     in 'int runServer()' (Main.cpp:903)

Thread 'Pool analytics collector' (0x7ffa06971700, LWP 13778):
     in 'static void Passenger::ApplicationPool2::Pool::collectAnalytics(Passenger::ApplicationPool2::PoolPtr)' (AnalyticsCollection.h:56)

Thread 'Pool garbage collector' (0x7ffa06930700, LWP 13778):
     in 'static void Passenger::ApplicationPool2::Pool::garbageCollect(Passenger::ApplicationPool2::PoolPtr)' (GarbageCollection.h:50)

Thread 'Main event loop: thread 1' (0x7ff9f3fff700, LWP 13778):
     in 'Passenger::ServerKit::Channel::Result Passenger::ServerKit::HttpServer<DerivedServer, Client>::onClientDataReceived(Client*, const Passenger::MemoryKit::mbuf&, int) [with DerivedServer = Passenger::RequestHandler; Client = Passenger::Client]' (HttpServer.h:891) -- Client 1-84

Thread 'Main event loop: thread 2' (0x7ff9f37fe700, LWP 13778):
     (empty)

Thread 'Admin event loop' (0x7ff9f2ffd700, LWP 13778):
     (empty)

Thread 'Load balancer' (0x7ff9f27fc700, LWP 13778):
     (empty)

Thread 'PipeWatcher: PID 13804 stderr, fd 30' (0x7ffa001a4700, LWP 13803):
     in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1679)
     in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1660)

Thread 'PipeWatcher: PID 13917 stdout, fd 40' (0x7ff9f177e700, LWP 13901):
     in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1679)
     in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1660)

Thread 'Group process spawner: /opt/agsolver/pzm/layer/webapp/deploy/releases/f1c14e945aacca168decb2d7d024d1fd8b81108a#default' (0x7ffa0685d700, LWP 13785):
     in 'std::string Passenger::ApplicationPool2::Spawner::readMessageLine(Details&) [with Details = Passenger::ApplicationPool2::SmartSpawner::StartupDetails; std::string = std::basic_string<char>]' (Spawner.h:788)
     in 'std::string Passenger::ApplicationPool2::SmartSpawner::negotiatePreloaderStartup(Passenger::ApplicationPool2::SmartSpawner::StartupDetails&)' (SmartSpawner.h:565)
     in 'void Passenger::ApplicationPool2::SmartSpawner::startPreloader()' (SmartSpawner.h:264)
     in 'virtual Passenger::ApplicationPool2::SpawnObject Passenger::ApplicationPool2::SmartSpawner::spawn(const Passenger::ApplicationPool2::Options&)' (SmartSpawner.h:765)
     in 'void Passenger::ApplicationPool2::Group::spawnThreadRealMain(const SpawnerPtr&, const Passenger::ApplicationPool2::Options&, unsigned int)' (Implementation.cpp:1038)

Thread 'Background I/O capturer' (0x7ffa067c2700, LWP 15084):
     in 'void Passenger::ApplicationPool2::Spawner::BackgroundIOCapturer::capture()' (Spawner.h:136)

### Request handler state (thread 1)

{
    "active_client_count" : 3,
    "active_clients" : 
    {
        "1-82" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr 21 16:16:21 2015",
                "relative" : "0s ago",
                "timestamp" : 1429632981.575178
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_sink_state" : 
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "localhost:8443",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/home.css?body=1",
                "refcount" : 2,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr 21 16:16:21 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1429632981.575298
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-82",
            "number" : 82,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        },
        "1-83" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr 21 16:16:21 2015",
                "relative" : "0s ago",
                "timestamp" : 1429632981.575178
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_sink_state" : 
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "localhost:8443",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/jquery_ujs.js?body=1",
                "refcount" : 2,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr 21 16:16:21 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1429632981.575298
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-83",
            "number" : 83,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        },
        "1-84" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr 21 16:16:21 2015",
                "relative" : "0s ago",
                "timestamp" : 1429632981.616784
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_sink_state" : 
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "localhost:8443",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/twitter/bootstrap/transition.js?body=1",
                "refcount" : 3,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr 21 16:16:21 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1429632981.616851
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-84",
            "number" : 84,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 3,
            "requests_begun" : 1
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 125,
    "free_request_count" : 1,
    "mbuf_pool" : 
    {
        "active_blocks" : 15,
        "active_memory" : 
        {
            "bytes" : 7680,
            "human_readable" : "7.5 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 247,
        "offset" : 448,
        "spare_memory" : 
        {
            "bytes" : 126464,
            "human_readable" : "123.5 KB"
        }
    },
    "pid" : 13778,
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 178480,
    "total_clients_accepted" : 84,
    "total_requests_begun" : 84,
    "turbocaching" : 
    {
        "fetches" : 5,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 5
    }
}

### Request handler state (thread 2)

{
    "active_client_count" : 3,
    "active_clients" : 
    {
        "2-81" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr 21 16:16:21 2015",
                "relative" : "0s ago",
                "timestamp" : 1429632981.575560
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_sink_state" : 
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "localhost:8443",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/jquery.js?body=1",
                "refcount" : 2,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr 21 16:16:21 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1429632981.575722
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "2-81",
            "number" : 81,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        },
        "2-82" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr 21 16:16:21 2015",
                "relative" : "0s ago",
                "timestamp" : 1429632981.575560
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_sink_state" : 
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "localhost:8443",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/jquery.iframe-transport.js?body=1",
                "refcount" : 2,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr 21 16:16:21 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1429632981.575722
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "2-82",
            "number" : 82,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        },
        "2-83" : 
        {
            "connected_at" : 
            {
                "local" : "Tue Apr 21 16:16:21 2015",
                "relative" : "0s ago",
                "timestamp" : 1429632981.578369
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_sink_state" : 
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : false,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "localhost:8443",
                "http_major" : 1,
                "http_minor" : 0,
                "http_state" : "COMPLETE",
                "method" : "GET",
                "path" : "/assets/jquery.remotipart.js?body=1",
                "refcount" : 2,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Tue Apr 21 16:16:21 2015",
                    "relative" : "0s ago",
                    "timestamp" : 1429632981.578577
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "2-83",
            "number" : 83,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 125,
    "free_request_count" : 2,
    "mbuf_pool" : 
    {
        "active_blocks" : 15,
        "active_memory" : 
        {
            "bytes" : 7680,
            "human_readable" : "7.5 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 290,
        "offset" : 448,
        "spare_memory" : 
        {
            "bytes" : 148480,
            "human_readable" : "145.0 KB"
        }
    },
    "pid" : 13778,
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 178562,
    "total_clients_accepted" : 83,
    "total_requests_begun" : 83,
    "turbocaching" : 
    {
        "fetches" : 4,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 4
    }
}

### Request handler config (thread 1)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : true,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 2)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : true,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Pool state (simple)
----------- General information -----------
Max pool size : 6
Processes     : 1
Requests in top-level queue : 0

----------- Application groups -----------
/opt/agsolver/pzm/layer/webapp/deploy/releases/f1c14e945aacca168decb2d7d024d1fd8b81108a#default:
  App root: /opt/agsolver/pzm/layer/webapp/deploy/releases/f1c14e945aacca168decb2d7d024d1fd8b81108a
  (spawning 1 new process...)
  Requests in queue: 5
  * PID: 13917   Sessions: 0       Processed: 25      Uptime: 32m 49s
    CPU: 0%      Memory  : 160M    Last used: 0s ago
    Disabling...
    URL     : http://127.0.0.1:57252
    Password: LzKMmrHWFCu3ajgb

### mbuf stats

nfree_mbuf_blockq  : 247
nactive_mbuf_blockq: 15
mbuf_block_chunk_size: 512

### Pool state (XML)
<?xml version="1.0" encoding="iso8859-1" ?>
<info version="3"><passenger_version>5.0.7</passenger_version><process_count>1</process_count><max>6</max><capacity_used>2</capacity_used><get_wait_list_size>0</get_wait_list_size><get_wait_list></get_wait_list><supergroups><supergroup><name>/opt/agsolver/pzm/layer/webapp/deploy/releases/f1c14e945aacca168decb2d7d024d1fd8b81108a</name><state>READY</state><get_wait_list_size>0</get_wait_list_size><capacity_used>2</capacity_used><secret>kFU2vs5YZiMiZcHrDVLUZnnhVNDdhZMYiggd4dEa54K</secret><group default="true"><name>/opt/agsolver/pzm/layer/webapp/deploy/releases/f1c14e945aacca168decb2d7d024d1fd8b81108a&#35;default</name><component_name>default</component_name><app_root>/opt/agsolver/pzm/layer/webapp/deploy/releases/f1c14e945aacca168decb2d7d024d1fd8b81108a</app_root><app_type>rack</app_type><environment>web_development</environment><uuid>jeF1emqLhlZp5UCM5t0K</uuid><enabled_process_count>0</enabled_process_count><disabling_process_count>1</disabling_process_count><disabled_process_count>0</disabled_process_count><capacity_used>2</capacity_used><get_wait_list_size>5</get_wait_list_size><disable_wait_list_size>1</disable_wait_list_size><processes_being_spawned>1</processes_being_spawned><spawning/><secret>LzKMmrHWFCu3ajgb</secret><life_status>ALIVE</life_status><options><app_root>/opt/agsolver/pzm/layer/webapp/deploy/releases/f1c14e945aacca168decb2d7d024d1fd8b81108a</app_root><app_group_name>/opt/agsolver/pzm/layer/webapp/deploy/releases/f1c14e945aacca168decb2d7d024d1fd8b81108a</app_group_name><app_type>rack</app_type><start_command>ruby&#9;/usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/helper-scripts/rack-loader.rb</start_command><startup_file>config.ru</startup_file><process_title>Passenger RubyApp</process_title><log_level>3</log_level><start_timeout>90000</start_timeout><environment>web_development</environment><base_uri>/</base_uri><spawn_method>smart</spawn_method><default_user>nobody</default_user><default_group>nogroup</default_group><ruby>ruby</ruby><python>python</python><nodejs>node</nodejs><logging_agent_address>unix:/tmp/passenger.lPwm0gD/agents.s/logging</logging_agent_address><logging_agent_username>logging</logging_agent_username><logging_agent_password>GZWc2dddOwjtXd6MKtthdYm6</logging_agent_password><debugger>false</debugger><analytics>false</analytics><group_secret>LzKMmrHWFCu3ajgb</group_secret><min_processes>1</min_processes><max_processes>0</max_processes><max_preloader_idle_time>-1</max_preloader_idle_time><max_out_of_band_work_instances>1</max_out_of_band_work_instances></options><processes><process><pid>13917</pid><sticky_session_id>370532587</sticky_session_id><gupid>16b92ef-RhSiBTrvc2</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>25</processed><spawner_creation_time>1429630886308765</spawner_creation_time><spawn_start_time>1429631012183852</spawn_start_time><spawn_end_time>1429631012186132</spawn_end_time><last_used>1429632981568656</last_used><last_used_desc>0s ago</last_used_desc><uptime>32m 49s</uptime><life_status>ALIVE</life_status><enabled>DISABLING</enabled><has_metrics>true</has_metrics><cpu>0</cpu><rss>173252</rss><pss>167118</pss><private_dirty>163964</private_dirty><swap>0</swap><real_memory>163964</real_memory><vmsize>1012300</vmsize><process_group_id>13778</process_group_id><command>Passenger RubyApp: /opt/agsolver/pzm/layer/webapp/deploy/releases/f1c14e945aacca168decb2d7d024d1fd8b81108a</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.lPwm0gD/apps.s/ruby.sQr3Wp9iUxrbw5R7DlhOvZ9FP73XQfSvLNLOxHXSxZBxJSDqyGtzGhtSChyKUhZ</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:57252</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process></processes></group></supergroup></supergroups></info>

--------------------------------------
[ pid=13778 ] Open files and file descriptors:
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
Passenger 13778 nobody  cwd    DIR              252,0     4096  272036 /opt/agsolver/pzm/layer/webapp/deploy/releases/f1c14e945aacca168decb2d7d024d1fd8b81108a
Passenger 13778 nobody  rtd    DIR              252,0     4096       2 /
Passenger 13778 nobody  txt    REG              252,0 22031405  798352 /usr/local/rvm/gems/ruby-2.1.6/gems/passenger-5.0.7/buildout/support-binaries/PassengerAgent
Passenger 13778 nobody  mem    REG              252,0    47712 2233743 /lib/x86_64-linux-gnu/libnss_files-2.19.so
Passenger 13778 nobody  mem    REG              252,0    47760 2233763 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
Passenger 13778 nobody  mem    REG              252,0    97296 2233752 /lib/x86_64-linux-gnu/libnsl-2.19.so
Passenger 13778 nobody  mem    REG              252,0    39824 2233751 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
Passenger 13778 nobody  mem    REG              252,0    43368 2233749 /lib/x86_64-linux-gnu/libcrypt-2.19.so
Passenger 13778 nobody  mem    REG              252,0   754880 1706806 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
Passenger 13778 nobody  mem    REG              252,0   295816 1706827 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
Passenger 13778 nobody  mem    REG              252,0    56768 1706823 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
Passenger 13778 nobody  mem    REG              252,0   166040 1706825 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
Passenger 13778 nobody  mem    REG              252,0    30944 1706768 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
Passenger 13778 nobody  mem    REG              252,0    85680 1706817 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
Passenger 13778 nobody  mem    REG              252,0   204104 1706821 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
Passenger 13778 nobody  mem    REG              252,0   657712 1706819 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
Passenger 13778 nobody  mem    REG              252,0   552584 1708122 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
Passenger 13778 nobody  mem    REG              252,0    35856 1708124 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
Passenger 13778 nobody  mem    REG              252,0    14256 2233451 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
Passenger 13778 nobody  mem    REG              252,0    18416 2228420 /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
Passenger 13778 nobody  mem    REG              252,0   268368 1706796 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
Passenger 13778 nobody  mem    REG              252,0    80256 1704007 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
Passenger 13778 nobody  mem    REG              252,0   251872 1708126 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
Passenger 13778 nobody  mem    REG              252,0   109296 1712106 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
Passenger 13778 nobody  mem    REG              252,0   101240 2233741 /lib/x86_64-linux-gnu/libresolv-2.19.so
Passenger 13778 nobody  mem    REG              252,0    43672 1712075 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
Passenger 13778 nobody  mem    REG              252,0    14592 2228410 /lib/x86_64-linux-gnu/libcom_err.so.2.1
Passenger 13778 nobody  mem    REG              252,0   186824 1712077 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
Passenger 13778 nobody  mem    REG              252,0   831616 1712082 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
Passenger 13778 nobody  mem    REG              252,0    14664 2233746 /lib/x86_64-linux-gnu/libdl-2.19.so
Passenger 13778 nobody  mem    REG              252,0   524704 2228415 /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
Passenger 13778 nobody  mem    REG              252,0   775112 1704386 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
Passenger 13778 nobody  mem    REG              252,0   323056 1712108 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
Passenger 13778 nobody  mem    REG              252,0    59640 1712109 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
Passenger 13778 nobody  mem    REG              252,0   285848 1712084 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
Passenger 13778 nobody  mem    REG              252,0  1930528 2228418 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Passenger 13778 nobody  mem    REG              252,0   383112 2228417 /lib/x86_64-linux-gnu/libssl.so.1.0.0
Passenger 13778 nobody  mem    REG              252,0   105696 1712113 /usr/lib/x86_64-linux-gnu/librtmp.so.0
Passenger 13778 nobody  mem    REG              252,0   207128 1712086 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
Passenger 13778 nobody  mem    REG              252,0  1840928 2233757 /lib/x86_64-linux-gnu/libc-2.19.so
Passenger 13778 nobody  mem    REG              252,0    90080 2228416 /lib/x86_64-linux-gnu/libgcc_s.so.1
Passenger 13778 nobody  mem    REG              252,0  1071552 2233742 /lib/x86_64-linux-gnu/libm-2.19.so
Passenger 13778 nobody  mem    REG              252,0   979056 1706808 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
Passenger 13778 nobody  mem    REG              252,0    31792 2233760 /lib/x86_64-linux-gnu/librt-2.19.so
Passenger 13778 nobody  mem    REG              252,0   141574 2233758 /lib/x86_64-linux-gnu/libpthread-2.19.so
Passenger 13778 nobody  mem    REG              252,0   100728 2228510 /lib/x86_64-linux-gnu/libz.so.1.2.8
Passenger 13778 nobody  mem    REG              252,0   422696 1706518 /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
Passenger 13778 nobody  mem    REG              252,0   149120 2233754 /lib/x86_64-linux-gnu/ld-2.19.so
Passenger 13778 nobody    0r   CHR                1,3      0t0      15 /dev/null
Passenger 13778 nobody    1w  FIFO                0,8      0t0   50403 pipe
Passenger 13778 nobody    2w  FIFO                0,8      0t0   50403 pipe
Passenger 13778 nobody    3u  unix 0x0000000000000000      0t0   29767 socket
Passenger 13778 nobody    5w  FIFO                0,8      0t0   50403 pipe
Passenger 13778 nobody    8r  FIFO                0,8      0t0   29795 pipe
Passenger 13778 nobody    9w  FIFO                0,8      0t0   29795 pipe
Passenger 13778 nobody   10r  FIFO                0,8      0t0   29796 pipe
Passenger 13778 nobody   11w  FIFO                0,8      0t0   29796 pipe
Passenger 13778 nobody   12r  FIFO                0,8      0t0   29797 pipe
Passenger 13778 nobody   13w  FIFO                0,8      0t0   29797 pipe
Passenger 13778 nobody   14u  unix 0x0000000000000000      0t0   29798 /tmp/passenger.lPwm0gD/agents.s/server
Passenger 13778 nobody   15u  unix 0x0000000000000000      0t0   29799 /var/run/passenger/passenger.sock
Passenger 13778 nobody   16u  unix 0x0000000000000000      0t0   29801 /tmp/passenger.lPwm0gD/agents.s/server_admin
Passenger 13778 nobody   17r   CHR                1,9      0t0      20 /dev/urandom
Passenger 13778 nobody   18r   CHR                1,9      0t0      20 /dev/urandom
Passenger 13778 nobody   19u  0000                0,9        0    7496 anon_inode
Passenger 13778 nobody   20u  0000                0,9        0    7496 anon_inode
Passenger 13778 nobody   21u  0000                0,9        0    7496 anon_inode
Passenger 13778 nobody   22u  0000                0,9        0    7496 anon_inode
Passenger 13778 nobody   23u  0000                0,9        0    7496 anon_inode
Passenger 13778 nobody   24u  0000                0,9        0    7496 anon_inode
Passenger 13778 nobody   25u  0000                0,9        0    7496 anon_inode
Passenger 13778 nobody   26u  0000                0,9        0    7496 anon_inode
Passenger 13778 nobody   27u  unix 0x0000000000000000      0t0   51301 /var/run/passenger/passenger.sock
Passenger 13778 nobody   28u  unix 0x0000000000000000      0t0   50344 /var/run/passenger/passenger.sock
Passenger 13778 nobody   29u  unix 0x0000000000000000      0t0   29819 socket
Passenger 13778 nobody   30r  FIFO                0,8      0t0   29820 pipe
Passenger 13778 nobody   31u  unix 0x0000000000000000      0t0   50345 /var/run/passenger/passenger.sock
Passenger 13778 nobody   32u  unix 0x0000000000000000      0t0   50346 /var/run/passenger/passenger.sock
Passenger 13778 nobody   33u  unix 0x0000000000000000      0t0   50347 /var/run/passenger/passenger.sock
Passenger 13778 nobody   34u  unix 0x0000000000000000      0t0   33211 socket
Passenger 13778 nobody   35u  unix 0x0000000000000000      0t0   51299 /var/run/passenger/passenger.sock
Passenger 13778 nobody   36u  unix 0x0000000000000000      0t0   50350 socket
Passenger 13778 nobody   37r  FIFO                0,8      0t0   50351 pipe
Passenger 13778 nobody   40u  unix 0x0000000000000000      0t0   31214 socket
--------------------------------------
[ pid=13778 ] Dumping a more detailed backtrace with crash-watch...
Found gdb at: /usr/bin/gdb
ERROR: Cannot attach to process.

This may be the result of kernel ptrace() hardening. Try disabling it with:
  sudo sh -c 'echo 0 > /proc/sys/kernel/yama/ptrace_scope'
See http://askubuntu.com/questions/41629/after-upgrade-gdb-wont-attach-to-process for more information.

I'll post a crash log with the detailed gdb trace as soon as I have one available.

FooBarWidget commented 9 years ago

Thanks, that report is a lot clearer. It has a lot of resemblance with https://github.com/phusion/passenger/issues/1459. Might even be the same issue.

blakmatrix commented 9 years ago

Using passenger docker on aws (m3.xlarge - 15GiB men, 4 cores, node.js app) occasionally causes our app to crash, we wouldn't have noticed anything but users started complaining that their sessions would drop occasionally.

Unfortuneately I am unable to reproduce this issue and I can't make sense out of the error output. We had a hunch that maybe IE 11 was causing the crash, but again we can't reproduce the error

Here is the log output: (log level 7) (there are 3 errors in the log, it looks like the subsequent two are semi related to the first SegFault)

It happens quite often so if i can be of help by twiddling flags or anything let me know.

https://gist.github.com/blakmatrix/c7907607272fcd016ccf

I will note that I reduced min process to 1 vs. 4 and I get the same issue.

dhampik commented 9 years ago

We have similar issue. Passenger 5.0.6 as nginx module (from standard passenger packages for Debian), like in https://github.com/phusion/passenger/issues/1459 Standard Rails app, nothing unusual. There are not much requests, because it's a staging server, but from time to time we see 502 error on frontend and similar crash logs on the server. Hopefully reverting to 4.0 will help.

PS: Sorry for offtopic, but I was not able to find a way to downgrade a package via apt-get to 4.0.59 version. Any chance that someone advises me how to do it?

karlsvec commented 9 years ago

@dhampik Passenger 4 packages are now in a separate APT repository: https://blog.phusion.nl/2015/03/08/passenger-4-apt-repository-now-available/

stormsilver commented 9 years ago

I will add my own stack trace, I don't know if this is related or not (perhaps a wholly different bug). I am on Mac OS X 10.10.3, with passenger built as an Apache 2.4.10 module

https://gist.github.com/stormsilver/1e065ce8838d94f14e71

OnixGH commented 9 years ago

@karlsvec @blakmatrix @stormsilver your traces look similar indeed, are you using Out-Of-Band Garbage Collection?

blakmatrix commented 9 years ago

unsure, I use whatever is the default in passenger-docker

OnixGH commented 9 years ago

@blakmatrix we have a range of images for passenger-docker, which one are you using? Ruby? Version?

stormsilver commented 9 years ago

@OnixGH Yes I am - here's the relevant snippet. This is a Rails 4.2.1 app under Ruby 2.1.2p95 running in development mode. What else can I tell you? 

if defined?(PhusionPassenger)
  # enable OOB GC, there is more to do here to take advantage of the Ruby 2.1 GC, but it doesn't work with Passenger's smart spawn, which is a huge bummer: http://blog.phusion.nl/2014/01/31/phusion-passenger-now-supports-the-new-ruby-2-1-out-of-band-gc/
  PhusionPassenger.require_passenger_lib 'rack/out_of_band_gc'
  use PhusionPassenger::Rack::OutOfBandGc, strategy: :counting, frequency: 5
  # use PhusionPassenger::Rack::OutOfBandGc, :strategy => :gctools_oobgc
end
karlsvec commented 9 years ago

@OnixGH: Yes, I am using out-of-band GC. My configuration is the same as @stormsilver's. I'm on Ruby 2.1.6.

blakmatrix commented 9 years ago

@OnixGH docker-full

OnixGH commented 9 years ago

We've found a way to reproduce the crash. It seems to trigger only when OOBGC is enabled, which @karlsvec @stormsilver @shirosaki also confirmed to have enabled.

OnixGH commented 9 years ago

@blakmatrix do you have OOBGC enabled (like @stormsilver posted, for example in your config.ru)? It's something that your app configuration controls, regardless of which passenger image or version you use.

OnixGH commented 9 years ago

Bug found & fixed in stable (coming in 5.0.8).

blakmatrix commented 9 years ago

@OnixGH our docker is configured for a node.js application. As far as I know I'm not using OOBGC.

stormsilver commented 9 years ago

Huzzah! Nice work, thank you!

On Apr 30, 2015, at 9:52 AM, OnixGH notifications@github.com wrote:

Bug found & fixed in stable (coming in 5.0.8).

— Reply to this email directly or view it on GitHub.

OnixGH commented 9 years ago

@blakmatrix thanks for following up. OOBGC process pausing is only relevant for ruby, so you might be experiencing a different issue. In any case it would help a lot if you had a trace with debugging symbols like the others did (you can install passenger-debuginfo to get this).

germs12 commented 9 years ago

I am having this same issue with Phusion Passenger version 5.0.9

Standard Rails App and high number of requests. But it manifests regardless of the traffic. I am not sure of my Garbage Collector, but I am more than willing to help if someone can provide some direction. I can post my dump here: https://gist.github.com/germs12/f52b511d411eaae96688

germs12 commented 9 years ago

Because I was seeing this error in production (shame on me) I didn't have a lot of time to debug. I upgraded to the latest 5.0.13 and the error is no more. :fireworks:

I'd do that if I found myself experiencing this issue in the future.

MisterBarbudo commented 9 years ago

Hello everyone. We encounter exactly the same problem. I checked at our stack trace and it's quite the same than the others. I was looking for some OOBGC info into our configuration files but no clue.

We have a bitnami redmine installation (Ruby 2.0.0 and Passenger 5.0.4) and i can't figured out where i could find something. First of all, I was looking for information on the bitnami community forum but here it's the only place where i finally find someone with the same issue.

I checked our apps config.ru files but there isn't much information in it.

If anyone has a clue, except than upgrade the passenger version, (we can't) it would be appreciated,

Thx

(As you probably saw i'am not a native english speaker so please excuse me if i burned your eyes ;-) )

germs12 commented 9 years ago

@MisterBarbudo Upgrade passenger and it should fix the issue.

OnixGH commented 9 years ago

@MisterBarbudo I'm assuming that by saying you can't upgrade the passenger version yourself you mean that bitnami is controlling it? In that case it seems to me that you should contact their support, we can't do much but fix the bug and that was more than 3 months ago.

MisterBarbudo commented 9 years ago

First of all, thanks for yours answers

@germs12 @OnixGH yes totally. It's an all inclusive package and i don't really know much about passenger so i don't want to risk to break something

I was looking for a way to upgrade passenger but our server can't download packages from the web (the only way i found so far to upgrade ).

I wiil look at it tomorow. Thx.

MisterBarbudo commented 8 years ago

Hello guys,

Eventually, I got the time to work on this issue. I managed to upgrade Passenger from 5.0.4 to 5.0.20 and you were right, it fixes the app.

Thx.

OnixGH commented 8 years ago

:+1: