search

phusion / passenger

A fast and robust web server and application server for Ruby, Python and Node.js
https://www.phusionpassenger.com/
MIT License
4.99k stars 548 forks source link

not a passenger issue (was: Passenger crashes using Fedora project packages on Fedora 25 + Apache) #1900

Closed Tcharl closed 7 years ago

Tcharl commented 7 years ago

Hi,

I just migrated to Fedora 25, and I encounter a crash of passenger (running a redmine behind an apache). Unfortunately, I don't know nothing about Ruby, Passenger or whatever, so here are the raw logs If you want more information, please ask by sending me a mail.

** redmine conf ****

<VirtualHost *:80>
      ServerName issues.osgiliath.net
    RailsEnv production
    Options FollowSymLinks
    RailsBaseURI /
     Options FollowSymLinks
                Options -MultiViews
    PassengerResolveSymlinksInDocumentRoot on
    PassengerForceMaxConcurrentRequestsPerProcess 0
    #PassengerAppRoot /usr/share/redmine
    RewriteEngine On
        RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]
   </VirtualHost>
<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile /etc/ssl/certs-web/issues.crt
        SSLCACertificateFile /etc/ssl/certs-web/cacert.crt
        SSLCertificateKeyFile /etc/ssl/certs-web/pki/issues.key.pem
    SSLCertificateChainFile    /etc/ssl/certs-web/1_root_bundle.crt
    ServerName issues.osgiliath.net
    PassengerSpawnMethod direct
       DocumentRoot /usr/share/redmine/public
    ProxyRequests Off
    ProxyVia Off
    ProxyPreserveHost On
    RequestHeader set X-Forwarded-Port 443
    RequestHeader set X-Forwarded-Proto https
        RailsEnv production
        ErrorLog /var/log/httpd/redmine.error.log
      <Directory /usr/share/redmine/public>
         Options FollowSymLinks
                Options -MultiViews
    RailsBaseURI /
    PassengerResolveSymlinksInDocumentRoot on
    PassengerForceMaxConcurrentRequestsPerProcess 0
      </Directory>
   </VirtualHost>

***** Passenger conf **

<IfModule mod_passenger.c>
   PassengerRoot /usr/share/passenger/phusion_passenger/locations.ini
   PassengerRuby /usr/bin/ruby
PassengerForceMaxConcurrentRequestsPerProcess 0 
</IfModule>

* Log *****

[ 2016-11-27 16:43:28.0212 30593/7fd7b8055700 age/Cor/Spa/PipeWatcher.h:95 ]: Cannot read from process 0 stdout: Bad file descriptor (errno=9)
[ pid=30593, timestamp=1480261408 ] Process aborted! signo=SIGSEGV(11), reason=SEGV_MAPERR, si_addr=0x400, randomSeed=1480261371
[ pid=30593 ] Crash log dumped to /var/tmp/passenger-crash-log.1480261408
[ pid=30593 ] Date, uname and ulimits:
Sun Nov 27 16:43:28 CET 2016
Linux 4.8.8-300.fc25.x86_64 #1 SMP Tue Nov 15 18:10:06 UTC 2016 x86_64 x86_64
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 128119
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 128119
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[ pid=30593 ] Phusion Passenger version: 5.0.30
[ pid=30593 ] libc backtrace available!
--------------------------------------
[ pid=30593 ] Backtrace with 28 frames:
Using backtrace sanitizer.
Passenger core(+0x61a7b)[0x556b4e3f1a7b]
Passenger core(+0x28d09)[0x556b4e3b8d09]
Passenger core(+0x63bee)[0x556b4e3f3bee]
/lib64/libpthread.so.0(?? at ??:0; +0x115c0) [0x7fd7bd9ba5c0]
Passenger core(+0x13fd82)[0x556b4e4cfd82]
Passenger core(+0x14051d)[0x556b4e4d051d]
Passenger core(+0x143893)[0x556b4e4d3893]
Passenger core(+0x157161)[0x556b4e4e7161]
Passenger core(+0x158597)[0x556b4e4e8597]
Passenger core(+0x187dd0)[0x556b4e517dd0]
Passenger core(+0x1a24f6)[0x556b4e5324f6]
Passenger core(+0x1a3d24)[0x556b4e533d24]
Passenger core(+0x1c1e29)[0x556b4e551e29]
Passenger core(+0x1c2878)[0x556b4e552878]
Passenger core(+0x119a96)[0x556b4e4a9a96]
Passenger core(+0x89a33)[0x556b4e419a33]
Passenger core(+0x8a3b5)[0x556b4e41a3b5]
Passenger core(+0x93c8c)[0x556b4e423c8c]
Passenger core(+0x124dd7)[0x556b4e4b4dd7]
Passenger core(+0x125577)[0x556b4e4b5577]
Passenger core(+0xf6b87)[0x556b4e486b87]
/lib64/libev.so.4(?? at ??:0; ev_invoke_pending+0x53) [0x7fd7be4b9113]
/lib64/libev.so.4(?? at ??:0; ev_run+0x968) [0x7fd7be4bc708]
Passenger core(+0x2513b5)[0x556b4e5e13b5]
Passenger core(+0x7d020)[0x556b4e40d020]
Passenger core(+0x220246)[0x556b4e5b0246]
/lib64/libpthread.so.0(?? at ??:0; +0x76ca) [0x7fd7bd9b06ca]
/lib64/libc.so.6(?? at ??:0; clone+0x5f) [0x7fd7bca36f6f]
--------------------------------------
[ pid=30593 ] Dumping additional diagnostical information...
--------------------------------------
### Backtraces
Thread 'Main thread' (0x7fd7be8a8840, LWP 30593):
     in 'void waitForExitEvent()' (CoreMain.cpp:875)
     in 'void mainLoop()' (CoreMain.cpp:768)
     in 'int runCore()' (CoreMain.cpp:995)

Thread 'Pool analytics collector' (0x7fd7be8a7700, LWP 30593):
     in 'static void Passenger::ApplicationPool2::Pool::collectAnalytics(Passenger::ApplicationPool2::PoolPtr)' (AnalyticsCollection.cpp:64)

Thread 'Pool garbage collector' (0x7fd7be866700, LWP 30593):
     in 'static void Passenger::ApplicationPool2::Pool::garbageCollect(Passenger::ApplicationPool2::PoolPtr)' (GarbageCollection.cpp:62)

Thread 'Main event loop: thread 1' (0x7fd7b37fe700, LWP 30593):
     in 'Passenger::ServerKit::Channel::Result Passenger::ServerKit::HttpServer<DerivedServer, Client>::onClientDataReceived(Client*, const Passenger::MemoryKit::mbuf&, int) [with DerivedServer = Passenger::Core::Controller; Client = Passenger::Core::Client]' (HttpServer.h:708) -- Client 1-1

Thread 'Main event loop: thread 1: libuv poller' (0x7fd7b845d700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 2' (0x7fd7b2ffd700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 2: libuv poller' (0x7fd7b83dc700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 3' (0x7fd7b27fc700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 3: libuv poller' (0x7fd7b835b700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 4' (0x7fd7b1ffb700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 4: libuv poller' (0x7fd7b82da700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 5' (0x7fd7b17fa700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 5: libuv poller' (0x7fd7b8259700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 6' (0x7fd7b0ff9700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 6: libuv poller' (0x7fd7b81d8700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 7' (0x7fd777fff700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 7: libuv poller' (0x7fd7b8157700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 8' (0x7fd7777fe700, LWP 30593):
     (empty)

Thread 'Main event loop: thread 8: libuv poller' (0x7fd7b80d6700, LWP 30593):
     (empty)

Thread 'API event loop' (0x7fd776ffd700, LWP 30593):
     (empty)

Thread 'API event loop: libuv poller' (0x7fd7b07f8700, LWP 30593):
     (empty)

Thread 'Load balancer' (0x7fd7767fc700, LWP 30593):
     (empty)

### Request handler state (thread 1)
{
    "active_client_count" : 1,
    "active_clients" : 
    {
        "1-1" : 
        {
            "connected_at" : 
            {
                "local" : "Sun Nov 27 16:43:28 2016",
                "relative" : "0s ago",
                "relative_timestamp" : 0,
                "timestamp" : 1480261408.1801231
            },
            "connection_state" : "ACTIVE",
            "current_request" : 
            {
                "app_sink_state" : 
                {
                    "callback_in_progress" : true,
                    "end_acked" : true,
                    "ended" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" : 
                {
                    "callback_in_progress" : true,
                    "end_acked" : true,
                    "ended" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "flags" : 
                {
                    "dechunk_response" : true,
                    "https" : true,
                    "request_body_buffering" : false
                },
                "host" : "issues.osgiliath.net",
                "http_major" : 1,
                "http_minor" : 1,
                "http_state" : "COMPLETE",
                "last_data_receive_time" : 
                {
                    "local" : "Sun Nov 27 16:43:28 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : 0,
                    "timestamp" : 1480261408.180089
                },
                "last_data_send_time" : null,
                "method" : "GET",
                "path" : "/",
                "refcount" : 3,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session_checkout_try" : 0,
                "started_at" : 
                {
                    "local" : "Sun Nov 27 16:43:28 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : 0,
                    "timestamp" : 1480261408.18011
                },
                "state" : "CHECKING_OUT_SESSION",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-1",
            "number" : 1,
            "output_channel_state" : 
            {
                "bytes_buffered" : 
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 3,
            "requests_begun" : 1
        }
    },
    "client_accept_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 127,
    "free_request_count" : 0,
    "mbuf_pool" : 
    {
        "active_blocks" : 1,
        "active_memory" : 
        {
            "bytes" : 4096,
            "human_readable" : "4.0 KB"
        },
        "chunk_size" : 4096,
        "free_blocks" : 0,
        "offset" : 4048,
        "spare_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        }
    },
    "peak_active_client_count" : 1,
    "pid" : 30593,
    "request_begin_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 0,
    "total_clients_accepted" : 1,
    "total_requests_begun" : 1,
    "turbocaching" : 
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : null,
        "store_successes" : 0,
        "stores" : 0
    }
}
### Request handler state (thread 2)
{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 0,
    "mbuf_pool" : 
    {
        "active_blocks" : 0,
        "active_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 4096,
        "free_blocks" : 0,
        "offset" : 4048,
        "spare_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        }
    },
    "peak_active_client_count" : 0,
    "pid" : 30593,
    "request_begin_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 0,
    "total_clients_accepted" : 0,
    "total_requests_begun" : 0,
    "turbocaching" : 
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : null,
        "store_successes" : 0,
        "stores" : 0
    }
}
### Request handler state (thread 3)
{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 0,
    "mbuf_pool" : 
    {
        "active_blocks" : 0,
        "active_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 4096,
        "free_blocks" : 0,
        "offset" : 4048,
        "spare_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        }
    },
    "peak_active_client_count" : 0,
    "pid" : 30593,
    "request_begin_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 0,
    "total_clients_accepted" : 0,
    "total_requests_begun" : 0,
    "turbocaching" : 
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : null,
        "store_successes" : 0,
        "stores" : 0
    }
}
### Request handler state (thread 4)
{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 0,
    "mbuf_pool" : 
    {
        "active_blocks" : 0,
        "active_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 4096,
        "free_blocks" : 0,
        "offset" : 4048,
        "spare_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        }
    },
    "peak_active_client_count" : 0,
    "pid" : 30593,
    "request_begin_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 0,
    "total_clients_accepted" : 0,
    "total_requests_begun" : 0,
    "turbocaching" : 
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : null,
        "store_successes" : 0,
        "stores" : 0
    }
}
### Request handler state (thread 5)
{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 0,
    "mbuf_pool" : 
    {
        "active_blocks" : 0,
        "active_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 4096,
        "free_blocks" : 0,
        "offset" : 4048,
        "spare_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        }
    },
    "peak_active_client_count" : 0,
    "pid" : 30593,
    "request_begin_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 0,
    "total_clients_accepted" : 0,
    "total_requests_begun" : 0,
    "turbocaching" : 
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : null,
        "store_successes" : 0,
        "stores" : 0
    }
}
### Request handler state (thread 6)
{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 0,
    "mbuf_pool" : 
    {
        "active_blocks" : 0,
        "active_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 4096,
        "free_blocks" : 0,
        "offset" : 4048,
        "spare_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        }
    },
    "peak_active_client_count" : 0,
    "pid" : 30593,
    "request_begin_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 0,
    "total_clients_accepted" : 0,
    "total_requests_begun" : 0,
    "turbocaching" : 
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : null,
        "store_successes" : 0,
        "stores" : 0
    }
}
### Request handler state (thread 7)
{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 0,
    "mbuf_pool" : 
    {
        "active_blocks" : 0,
        "active_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 4096,
        "free_blocks" : 0,
        "offset" : 4048,
        "spare_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        }
    },
    "peak_active_client_count" : 0,
    "pid" : 30593,
    "request_begin_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 0,
    "total_clients_accepted" : 0,
    "total_requests_begun" : 0,
    "turbocaching" : 
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : null,
        "store_successes" : 0,
        "stores" : 0
    }
}
### Request handler state (thread 8)
{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 0,
    "mbuf_pool" : 
    {
        "active_blocks" : 0,
        "active_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 4096,
        "free_blocks" : 0,
        "offset" : 4048,
        "spare_memory" : 
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        }
    },
    "peak_active_client_count" : 0,
    "pid" : 30593,
    "request_begin_speed" : 
    {
        "1h" : 
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 0
        },
        "1m" : 
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 0
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 0,
    "total_clients_accepted" : 0,
    "total_requests_begun" : 0,
    "turbocaching" : 
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : null,
        "store_successes" : 0,
        "stores" : 0
    }
}
### Request handler config (thread 1)
{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}
### Request handler config (thread 2)
{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}
### Request handler config (thread 3)
{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}
### Request handler config (thread 4)
{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}
### Request handler config (thread 5)
{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}
### Request handler config (thread 6)
{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}
### Request handler config (thread 7)
{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}
### Request handler config (thread 8)
{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}
### Pool state (simple)
----------- General information -----------
Max pool size : 6
App groups    : 0
Processes     : 0
Requests in top-level queue : 0

----------- Application groups -----------

### mbuf stats

nfree_mbuf_blockq  : 0
nactive_mbuf_blockq: 1
mbuf_block_chunk_size: 4096

### Pool state (XML)
<?xml version="1.0" encoding="iso8859-1" ?>
<info version="3"><passenger_version>5.0.30</passenger_version><group_count>0</group_count><process_count>0</process_count><max>6</max><capacity_used>0</capacity_used><get_wait_list_size>0</get_wait_list_size><get_wait_list></get_wait_list><supergroups></supergroups></info>

--------------------------------------
[ pid=30593 ] Open files and file descriptors:
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
lsof: WARNING: can't stat() ext4 file system /home/charlie/Downloads
      Output information may be incomplete.
lsof: WARNING: can't stat() ext4 file system /home/charlie/Videos
      Output information may be incomplete.
COMMAND     PID   USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
Passenger 30593 apache  cwd       DIR              253,0     4096        2 /
Passenger 30593 apache  rtd       DIR              253,0     4096        2 /
Passenger 30593 apache  txt       REG              253,0  3193584  5243184 /usr/lib64/passenger/support-binaries/PassengerAgent
Passenger 30593 apache  mem       REG              253,0    11368   180318 /usr/lib64/libfreebl3.so
Passenger 30593 apache  mem       REG              253,0   471584   135085 /usr/lib64/libpcre.so.1.2.7
Passenger 30593 apache  mem       REG              253,0    37208   197519 /usr/lib64/libcrypt-nss-2.24.so
Passenger 30593 apache  mem       REG              253,0   153184   134978 /usr/lib64/libselinux.so.1
Passenger 30593 apache  mem       REG              253,0   120136   163824 /usr/lib64/libsasl2.so.3.0.0
Passenger 30593 apache  mem       REG              253,0   114680   148546 /usr/lib64/libresolv-2.24.so
Passenger 30593 apache  mem       REG              253,0    15128   166331 /usr/lib64/libkeyutils.so.1.5
Passenger 30593 apache  mem       REG              253,0    61232   189101 /usr/lib64/libkrb5support.so.0.1
Passenger 30593 apache  mem       REG              253,0  1240184   180469 /usr/lib64/libunistring.so.2.0.0
Passenger 30593 apache  mem       REG              253,0  2485224   189025 /usr/lib64/libcrypto.so.1.0.2j
Passenger 30593 apache  mem       REG              253,0   467696   189027 /usr/lib64/libssl.so.1.0.2j
Passenger 30593 apache  mem       REG              253,0   333488   198478 /usr/lib64/libldap-2.4.so.2.10.7
Passenger 30593 apache  mem       REG              253,0    60808   152153 /usr/lib64/liblber-2.4.so.2.10.7
Passenger 30593 apache  mem       REG              253,0    15248   196577 /usr/lib64/libcom_err.so.2.1
Passenger 30593 apache  mem       REG              253,0   201296   189053 /usr/lib64/libk5crypto.so.3.1
Passenger 30593 apache  mem       REG              253,0   951440   135674 /usr/lib64/libkrb5.so.3.3
Passenger 30593 apache  mem       REG              253,0   320832   131127 /usr/lib64/libgssapi_krb5.so.2.2
Passenger 30593 apache  mem       REG              253,0   253136   197151 /usr/lib64/libnspr4.so
Passenger 30593 apache  mem       REG              253,0    19720   197154 /usr/lib64/libplc4.so
Passenger 30593 apache  mem       REG              253,0    15576   197158 /usr/lib64/libplds4.so
Passenger 30593 apache  mem       REG              253,0   188424   197161 /usr/lib64/libnssutil3.so
Passenger 30593 apache  mem       REG              253,0  1229696   196504 /usr/lib64/libnss3.so
Passenger 30593 apache  mem       REG              253,0   164040   202802 /usr/lib64/libsmime3.so
Passenger 30593 apache  mem       REG              253,0   314456   202803 /usr/lib64/libssl3.so
Passenger 30593 apache  mem       REG              253,0    51984   185426 /usr/lib64/libpsl.so.5.0.1
Passenger 30593 apache  mem       REG              253,0   185360   187999 /usr/lib64/libssh2.so.1.0.1
Passenger 30593 apache  mem       REG              253,0   225208   165215 /usr/lib64/libidn2.so.0.0.10
Passenger 30593 apache  mem       REG              253,0   157416   154587 /usr/lib64/libnghttp2.so.14.9.0
Passenger 30593 apache  mem       REG              253,0   112432   134923 /usr/lib64/libnsl-2.24.so
Passenger 30593 apache  mem       REG              253,0  2115832   134911 /usr/lib64/libc-2.24.so
Passenger 30593 apache  mem       REG              253,0    92872   134926 /usr/lib64/libgcc_s-6.2.1-20160916.so.1
Passenger 30593 apache  mem       REG              253,0  1169416   139945 /usr/lib64/libm-2.24.so
Passenger 30593 apache  mem       REG              253,0  1590720   139346 /usr/lib64/libstdc++.so.6.0.22
Passenger 30593 apache  mem       REG              253,0    19736   134915 /usr/lib64/libdl-2.24.so
Passenger 30593 apache  mem       REG              253,0    44056   148568 /usr/lib64/librt-2.24.so
Passenger 30593 apache  mem       REG              253,0   151112   140793 /usr/lib64/libpthread-2.24.so
Passenger 30593 apache  mem       REG              253,0    89520   162397 /usr/lib64/libz.so.1.2.8
Passenger 30593 apache  mem       REG              253,0   523904   190093 /usr/lib64/libcurl.so.4.4.0
Passenger 30593 apache  mem       REG              253,0   212376   150561 /usr/lib64/libjsoncpp.so.1.7.7
Passenger 30593 apache  mem       REG              253,0   155344   197435 /usr/lib64/libuv.so.1.0.0
Passenger 30593 apache  mem       REG              253,0    56584   154462 /usr/lib64/libev.so.4.0.0
Passenger 30593 apache  mem       REG              253,0   182608   134904 /usr/lib64/ld-2.24.so
Passenger 30593 apache    0r      CHR                1,3      0t0     1028 /dev/null
Passenger 30593 apache    1w     FIFO               0,10      0t0 17565587 pipe
Passenger 30593 apache    2w     FIFO               0,10      0t0 17565587 pipe
Passenger 30593 apache    3u     unix 0xffff8f7da768e400      0t0 17566163 type=STREAM
Passenger 30593 apache    5w     FIFO               0,10      0t0 17565587 pipe
Passenger 30593 apache    8r     FIFO               0,10      0t0 17568103 pipe
Passenger 30593 apache    9w     FIFO               0,10      0t0 17568103 pipe
Passenger 30593 apache   10r     FIFO               0,10      0t0 17568104 pipe
Passenger 30593 apache   11w     FIFO               0,10      0t0 17568104 pipe
Passenger 30593 apache   12r     FIFO               0,10      0t0 17568105 pipe
Passenger 30593 apache   13w     FIFO               0,10      0t0 17568105 pipe
Passenger 30593 apache   14u     unix 0xffff8f7ddfa9ac00      0t0 17568106 /tmp/passenger.pxVRlLP/agents.s/core type=STREAM
Passenger 30593 apache   15u     unix 0xffff8f7ddfa9b000      0t0 17568107 /tmp/passenger.pxVRlLP/agents.s/core_api type=STREAM
Passenger 30593 apache   16r      CHR                1,9      0t0     1033 /dev/urandom
Passenger 30593 apache   17u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   18u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   19r     FIFO               0,10      0t0 17568108 pipe
Passenger 30593 apache   20w     FIFO               0,10      0t0 17568108 pipe
Passenger 30593 apache   21u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   22r     FIFO               0,10      0t0 17568109 pipe
Passenger 30593 apache   23w     FIFO               0,10      0t0 17568109 pipe
Passenger 30593 apache   24u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   25u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   26u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   27u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   28u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   29u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   30r     FIFO               0,10      0t0 17568110 pipe
Passenger 30593 apache   31w     FIFO               0,10      0t0 17568110 pipe
Passenger 30593 apache   32u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   33u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   34u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   35u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   36r     FIFO               0,10      0t0 17568111 pipe
Passenger 30593 apache   37w     FIFO               0,10      0t0 17568111 pipe
Passenger 30593 apache   38u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   39u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   40u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   41u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   42r     FIFO               0,10      0t0 17568112 pipe
Passenger 30593 apache   43w     FIFO               0,10      0t0 17568112 pipe
Passenger 30593 apache   44u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   45u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   46u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   47u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   48r     FIFO               0,10      0t0 17568113 pipe
Passenger 30593 apache   49w     FIFO               0,10      0t0 17568113 pipe
Passenger 30593 apache   50u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   51u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   52u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   53u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   54r     FIFO               0,10      0t0 17568114 pipe
Passenger 30593 apache   55w     FIFO               0,10      0t0 17568114 pipe
Passenger 30593 apache   56u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   57u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   58u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   59u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   60r     FIFO               0,10      0t0 17568115 pipe
Passenger 30593 apache   61w     FIFO               0,10      0t0 17568115 pipe
Passenger 30593 apache   62u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   63u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   64u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   65u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   66r     FIFO               0,10      0t0 17568116 pipe
Passenger 30593 apache   67w     FIFO               0,10      0t0 17568116 pipe
Passenger 30593 apache   68u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   69u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   70u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   71u  a_inode               0,11        0     8226 [eventpoll]
Passenger 30593 apache   72r     FIFO               0,10      0t0 17568117 pipe
Passenger 30593 apache   73w     FIFO               0,10      0t0 17568117 pipe
Passenger 30593 apache   74u  a_inode               0,11        0     8226 [eventfd]
Passenger 30593 apache   75u     unix 0xffff8f78043c2c00      0t0 17567133 /tmp/passenger.pxVRlLP/agents.s/core type=STREAM
--------------------------------------
[ pid=30593 ] Dumping a more detailed backtrace with crash-watch...
Found gdb at: /usr/bin/gdb
*******************************************************
*
*    Current thread (1) backtrace
*
*******************************************************

    #0  0x00007fd7bca2cd63 in select () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e5e42d3 in oxt::syscalls::select(int, fd_set*, fd_set*, fd_set*, timeval*) ()
    No symbol table info available.
    #2  0x0000556b4e45ac6d in waitForExitEvent() ()
    No symbol table info available.
    #3  0x0000556b4e45d07c in runCore() ()
    No symbol table info available.
    #4  0x0000556b4e45f162 in coreMain(int, char**) ()
    No symbol table info available.
    #5  0x0000556b4e3b9af5 in main ()
    No symbol table info available.

*******************************************************
*
*    All thread backtraces
*
*******************************************************

########### Thread 22 (Thread 0x7fd7767fc700 (LWP 30616)) ###########

    #0  0x00007fd7bca2b00d in poll () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e4a7c41 in Passenger::ServerKit::AcceptLoadBalancer<Passenger::Core::Controller>::mainLoop() ()
    No symbol table info available.
    #2  0x0000556b4e3d7269 in Passenger::runAndPrintExceptions(boost::function<void ()> const&, bool) ()
    No symbol table info available.
    #3  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #4  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #5  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #6  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #7  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 21 (Thread 0x7fd7b07f8700 (LWP 30615)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e44f863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 20 (Thread 0x7fd776ffd700 (LWP 30614)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x00007fd7be4b9ac5 in epoll_poll () from /lib64/libev.so.4
    No symbol table info available.
    #2  0x00007fd7be4bc089 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #3  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #4  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #5  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #6  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #7  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 19 (Thread 0x7fd7b80d6700 (LWP 30613)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e44f863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 18 (Thread 0x7fd7777fe700 (LWP 30612)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x00007fd7be4b9ac5 in epoll_poll () from /lib64/libev.so.4
    No symbol table info available.
    #2  0x00007fd7be4bc089 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #3  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #4  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #5  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #6  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #7  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 17 (Thread 0x7fd7b8157700 (LWP 30611)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e44f863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 16 (Thread 0x7fd777fff700 (LWP 30610)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x00007fd7be4b9ac5 in epoll_poll () from /lib64/libev.so.4
    No symbol table info available.
    #2  0x00007fd7be4bc089 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #3  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #4  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #5  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #6  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #7  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 15 (Thread 0x7fd7b81d8700 (LWP 30609)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e44f863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 14 (Thread 0x7fd7b0ff9700 (LWP 30608)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x00007fd7be4b9ac5 in epoll_poll () from /lib64/libev.so.4
    No symbol table info available.
    #2  0x00007fd7be4bc089 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #3  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #4  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #5  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #6  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #7  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 13 (Thread 0x7fd7b8259700 (LWP 30607)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e44f863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 12 (Thread 0x7fd7b17fa700 (LWP 30606)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x00007fd7be4b9ac5 in epoll_poll () from /lib64/libev.so.4
    No symbol table info available.
    #2  0x00007fd7be4bc089 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #3  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #4  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #5  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #6  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #7  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 11 (Thread 0x7fd7b82da700 (LWP 30605)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e44f863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 10 (Thread 0x7fd7b1ffb700 (LWP 30604)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x00007fd7be4b9ac5 in epoll_poll () from /lib64/libev.so.4
    No symbol table info available.
    #2  0x00007fd7be4bc089 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #3  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #4  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #5  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #6  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #7  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 9 (Thread 0x7fd7b835b700 (LWP 30603)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e44f863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 8 (Thread 0x7fd7b27fc700 (LWP 30602)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x00007fd7be4b9ac5 in epoll_poll () from /lib64/libev.so.4
    No symbol table info available.
    #2  0x00007fd7be4bc089 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #3  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #4  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #5  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #6  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #7  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 7 (Thread 0x7fd7b83dc700 (LWP 30601)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e44f863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 6 (Thread 0x7fd7b2ffd700 (LWP 30600)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x00007fd7be4b9ac5 in epoll_poll () from /lib64/libev.so.4
    No symbol table info available.
    #2  0x00007fd7be4bc089 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #3  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #4  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #5  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #6  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #7  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 5 (Thread 0x7fd7b845d700 (LWP 30599)) ###########

    #0  0x00007fd7bca37563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e44f863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 4 (Thread 0x7fd7b37fe700 (LWP 30598)) ###########

    #0  0x00007fd7bd9ba46d in raise () from /lib64/libpthread.so.0
    No symbol table info available.
    #1  0x0000556b4e3f33ea in Passenger::abortHandler(int, siginfo_t*, void*) ()
    No symbol table info available.
    #2  <signal handler called>
    No symbol table info available.
    #3  0x0000556b4e4cfd82 in Passenger::ApplicationPool2::Group::createProcessObject(Json::Value const&) ()
    No symbol table info available.
    #4  0x0000556b4e4d051d in Passenger::ApplicationPool2::Group::initialize() ()
    No symbol table info available.
    #5  0x0000556b4e4d3893 in Passenger::ApplicationPool2::Pool::createGroup(Passenger::ApplicationPool2::Options const&) ()
    No symbol table info available.
    #6  0x0000556b4e4e7161 in Passenger::ApplicationPool2::Pool::createGroupAndAsyncGetFromIt(Passenger::ApplicationPool2::Options const&, Passenger::ApplicationPool2::GetCallback const&, boost::container::vector<boost::function<void ()>, boost::container::new_allocator<boost::function<void ()> > >&) ()
    No symbol table info available.
    #7  0x0000556b4e4e8597 in Passenger::ApplicationPool2::Pool::asyncGet(Passenger::ApplicationPool2::Options const&, Passenger::ApplicationPool2::GetCallback const&, bool, Passenger::UnionStation::StopwatchLog**) ()
    No symbol table info available.
    #8  0x0000556b4e517dd0 in Passenger::Core::Controller::asyncGetFromApplicationPool(Passenger::Core::Request*, Passenger::ApplicationPool2::GetCallback) ()
    No symbol table info available.
    #9  0x0000556b4e5324f6 in Passenger::Core::Controller::checkoutSession(Passenger::Core::Client*, Passenger::Core::Request*) ()
    No symbol table info available.
    #10 0x0000556b4e533d24 in Passenger::Core::Controller::onRequestBegin(Passenger::Core::Client*, Passenger::Core::Request*) ()
    No symbol table info available.
    #11 0x0000556b4e551e29 in Passenger::ServerKit::HttpServer<Passenger::Core::Controller, Passenger::Core::Client>::processClientDataWhenParsingHeaders(Passenger::Core::Client*, Passenger::Core::Request*, Passenger::MemoryKit::mbuf const&, int) ()
    No symbol table info available.
    #12 0x0000556b4e552878 in Passenger::ServerKit::HttpServer<Passenger::Core::Controller, Passenger::Core::Client>::onClientDataReceived(Passenger::Core::Client*, Passenger::MemoryKit::mbuf const&, int) ()
    No symbol table info available.
    #13 0x0000556b4e4a9a96 in Passenger::ServerKit::BaseServer<Passenger::Core::Controller, Passenger::Core::Client>::_onClientDataReceived(Passenger::ServerKit::Channel*, Passenger::MemoryKit::mbuf const&, int) ()
    No symbol table info available.
    #14 0x0000556b4e419a33 in Passenger::ServerKit::Channel::callDataCallbackWithoutRefGuard() ()
    No symbol table info available.
    #15 0x0000556b4e41a3b5 in Passenger::ServerKit::Channel::feedWithoutRefGuard(Passenger::MemoryKit::mbuf&&) ()
    No symbol table info available.
    #16 0x0000556b4e423c8c in Passenger::ServerKit::FdSourceChannel::onReadableWithoutRefGuard() ()
    No symbol table info available.
    #17 0x0000556b4e4b4dd7 in Passenger::ServerKit::BaseServer<Passenger::Core::Controller, Passenger::Core::Client>::feedNewClients(int const*, unsigned int) ()
    No symbol table info available.
    #18 0x0000556b4e4b5577 in Passenger::ServerKit::AcceptLoadBalancer<Passenger::Core::Controller>::feedNewClient(Passenger::Core::Controller*, int) ()
    No symbol table info available.
    #19 0x0000556b4e486b87 in Passenger::SafeLibev::asyncHandler(ev_loop*, ev_async*, int) ()
    No symbol table info available.
    #20 0x00007fd7be4b9113 in ev_invoke_pending () from /lib64/libev.so.4
    No symbol table info available.
    #21 0x00007fd7be4bc708 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #22 0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #23 0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #24 0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #25 0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #26 0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 3 (Thread 0x7fd7be866700 (LWP 30596)) ###########

    #0  0x00007fd7bd9b6809 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
       from /lib64/libpthread.so.0
    No symbol table info available.
    #1  0x0000556b4e4e4f15 in Passenger::ApplicationPool2::Pool::garbageCollect(boost::shared_ptr<Passenger::ApplicationPool2::Pool>) ()
    No symbol table info available.
    #2  0x0000556b4e4f3527 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<Passenger::ApplicationPool2::Pool>), boost::_bi::list1<boost::_bi::value<boost::shared_ptr<Passenger::ApplicationPool2::Pool> > > >, void>::invoke(boost::detail::function::function_buffer&) ()
    No symbol table info available.
    #3  0x0000556b4e5029fb in boost::function0<void>::operator()() const ()
    No symbol table info available.
    #4  0x0000556b4e506c39 in oxt::dynamic_thread_group::thread_main(oxt::dynamic_thread_group::thread_func_data*) ()
    No symbol table info available.
    #5  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #6  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #7  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #8  0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #9  0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 2 (Thread 0x7fd7be8a7700 (LWP 30595)) ###########

    #0  0x00007fd7bd9b9c7d in nanosleep () from /lib64/libpthread.so.0
    No symbol table info available.
    #1  0x0000556b4e5e4ecb in oxt::syscalls::nanosleep(timespec const*, timespec*)
        ()
    No symbol table info available.
    #2  0x0000556b4e5e5123 in oxt::syscalls::usleep(unsigned int) ()
    No symbol table info available.
    #3  0x0000556b4e4ebbf7 in Passenger::ApplicationPool2::Pool::collectAnalytics(boost::shared_ptr<Passenger::ApplicationPool2::Pool>) ()
    No symbol table info available.
    #4  0x0000556b4e4f3527 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<Passenger::ApplicationPool2::Pool>), boost::_bi::list1<boost::_bi::value<boost::shared_ptr<Passenger::ApplicationPool2::Pool> > > >, void>::invoke(boost::detail::function::function_buffer&) ()
    No symbol table info available.
    #5  0x0000556b4e5029fb in boost::function0<void>::operator()() const ()
    No symbol table info available.
    #6  0x0000556b4e506c39 in oxt::dynamic_thread_group::thread_main(oxt::dynamic_thread_group::thread_func_data*) ()
    No symbol table info available.
    #7  0x0000556b4e5e13b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #8  0x0000556b4e40d020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #9  0x0000556b4e5b0246 in thread_proxy ()
    No symbol table info available.
    #10 0x00007fd7bd9b06ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #11 0x00007fd7bca36f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 1 (Thread 0x7fd7be8a8840 (LWP 30593)) ###########

    #0  0x00007fd7bca2cd63 in select () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x0000556b4e5e42d3 in oxt::syscalls::select(int, fd_set*, fd_set*, fd_set*, timeval*) ()
    No symbol table info available.
    #2  0x0000556b4e45ac6d in waitForExitEvent() ()
    No symbol table info available.
    #3  0x0000556b4e45d07c in runCore() ()
    No symbol table info available.
    #4  0x0000556b4e45f162 in coreMain(int, char**) ()
    No symbol table info available.
    #5  0x0000556b4e3b9af5 in main ()
    No symbol table info available.
[ 2016-11-27 16:43:29.8087 30042/7f15f9b6f700 age/Wat/AgentWatcher.cpp:97 ]: Passenger core (pid=30593) crashed with signal SIGSEGV, restarting it...
[ 2016-11-27 16:43:29.8146 30913/7f30d797a840 age/Cor/CoreMain.cpp:982 ]: Starting Passenger core...
[ 2016-11-27 16:43:29.8148 30913/7f30d797a840 age/Cor/CoreMain.cpp:235 ]: Passenger core running in multi-application mode.
[ 2016-11-27 16:43:29.8320 30913/7f30d797a840 age/Cor/CoreMain.cpp:732 ]: Passenger core online, PID 30913
OnixGH commented 7 years ago

Hi, thanks for reporting. The crash appears to have happened in C++, in Thread 4:

#0  0x00007fd7bd9ba46d in raise () from /lib64/libpthread.so.0
..
#3  0x0000556b4e4cfd82 in Passenger::ApplicationPool2::Group::createProcessObject(Json::Value const&) ()
#4  0x0000556b4e4d051d in Passenger::ApplicationPool2::Group::initialize() ()

I installed redmine on a clean Fedora 24 and I cannot reproduce your crash.

So we need more information to see what about your system is triggering this.

Note: if you are an Enterprise customer we can provide support by email.

Tcharl commented 7 years ago

Hi OnixGH,

It worked perfectly on F24, I upgraded my system to Fedora 25... I installed passenger with dnf, and the crash always happens.

I'll try to test on a VirtualBox image when I'll have some times, but now it's on my own raw home server.

Regards,

OnixGH commented 7 years ago

I'm not familiar with dnf, but did you try to install using the gem or tarball method?

Either way it would be great if you could reproduce it on a fresh VM.

OnixGH commented 7 years ago

Tried again on Fedora 25 (clean image from DigitalOcean, gem install as the pages I linked to describe), still works without errors or crashing.

jbeima commented 7 years ago

I am seeing this as well.

Under Fedora 24 I had 7 Ruby on Rails apps that ran just fine.

Now that I have upgraded to Fedora 25, I get this from passenger:

[ pid=16620, timestamp=1481136126 ] Process aborted! signo=SIGSEGV(11), reason=SEGV_MAPERR, si_addr=0x400, randomSeed=1481135860
[ pid=16620 ] Crash log dumped to /var/tmp/passenger-crash-log.1481136126
[ pid=16620 ] Date, uname and ulimits:
Wed Dec  7 11:42:06 MST 2016
Linux 4.8.11-300.fc25.x86_64 #1 SMP Mon Nov 28 18:24:51 UTC 2016 x86_64 x86_64
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7924
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7924
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[ pid=16620 ] Phusion Passenger version: 5.0.30
[ pid=16620 ] libc backtrace available!
--------------------------------------
[ pid=16620 ] Backtrace with 26 frames:
Using backtrace sanitizer.
Passenger core(+0x61a7b)[0x55ebfae1fa7b]
Passenger core(+0x28d09)[0x55ebfade6d09]
Passenger core(+0x63bee)[0x55ebfae21bee]
/lib64/libpthread.so.0(?? at ??:0; +0x115c0) [0x7f5829f4c5c0]
Passenger core(+0x13fd82)[0x55ebfaefdd82]
Passenger core(+0x14051d)[0x55ebfaefe51d]
Passenger core(+0x143893)[0x55ebfaf01893]
Passenger core(+0x157161)[0x55ebfaf15161]
Passenger core(+0x158597)[0x55ebfaf16597]
Passenger core(+0x187dd0)[0x55ebfaf45dd0]
Passenger core(+0x1a24f6)[0x55ebfaf604f6]
Passenger core(+0x1a3d24)[0x55ebfaf61d24]
Passenger core(+0x1c1e29)[0x55ebfaf7fe29]
Passenger core(+0x1c2878)[0x55ebfaf80878]
Passenger core(+0x119a96)[0x55ebfaed7a96]
Passenger core(+0x89a33)[0x55ebfae47a33]
Passenger core(+0x8a3b5)[0x55ebfae483b5]
Passenger core(+0x93c8c)[0x55ebfae51c8c]
Passenger core(+0x9406e)[0x55ebfae5206e]
/lib64/libev.so.4(?? at ??:0; ev_invoke_pending+0x53) [0x7f582aa4b113]
/lib64/libev.so.4(?? at ??:0; ev_run+0x968) [0x7f582aa4e708]
Passenger core(+0x2513b5)[0x55ebfb00f3b5]
Passenger core(+0x7d020)[0x55ebfae3b020]
Passenger core(+0x220246)[0x55ebfafde246]
/lib64/libpthread.so.0(?? at ??:0; +0x76ca) [0x7f5829f426ca]
/lib64/libc.so.6(?? at ??:0; clone+0x5f) [0x7f5828fc8f6f]
--------------------------------------
[ pid=16620 ] Dumping additional diagnostical information...
--------------------------------------
### Backtraces
Thread 'Main thread' (0x7f582ae55840, LWP 16620):
     in 'void waitForExitEvent()' (CoreMain.cpp:875)
     in 'void mainLoop()' (CoreMain.cpp:768)
     in 'int runCore()' (CoreMain.cpp:995)

Thread 'Pool garbage collector' (0x7f582adf2700, LWP 16620):
     in 'static void Passenger::ApplicationPool2::Pool::garbageCollect(Passenger::ApplicationPool2::PoolPtr)' (GarbageCollection.cpp:62)

Thread 'Pool analytics collector' (0x7f582ae33700, LWP 16620):
     in 'static void Passenger::ApplicationPool2::Pool::collectAnalytics(Passenger::ApplicationPool2::PoolPtr)' (AnalyticsCollection.cpp:64)

Thread 'Main event loop: thread 1: libuv poller' (0x7f582ad67700, LWP 16620):
     (empty)

Thread 'Main event loop: thread 1' (0x7f5824316700, LWP 16620):
     in 'Passenger::ServerKit::Channel::Result Passenger::ServerKit::HttpServer<DerivedServer, Client>::onClientDataReceived(Client*, const Passenger::MemoryKit::mbuf&, int) [with DerivedServer = Pas
senger::Core::Controller; Client = Passenger::Core::Client]' (HttpServer.h:708) -- Client 1-1

Thread 'API event loop: libuv poller' (0x7f5823314700, LWP 16620):
     (empty)

Thread 'API event loop' (0x7f5823b15700, LWP 16620):
     (empty)

### Request handler state (thread 1)
{
        "active_client_count" : 1,
        "active_clients" : 
        {
                "1-1" : 
                {
                        "connected_at" : 
                        {
                                "local" : "Wed Dec  7 11:42:06 2016",
                                "relative" : "0s ago",
                                "relative_timestamp" : -0.00042581558227539062,
                                "timestamp" : 1481136126.9557619
                        },
                        "connection_state" : "ACTIVE",
                        "current_request" : 
                        {
                                "app_sink_state" : 
                                {
                                        "callback_in_progress" : true,
                                        "end_acked" : true,
                                        "ended" : true,
                                        "initialized" : false,
                                        "io_watcher_active" : false
                                },
                                "app_source_state" : 
                                {
                                        "callback_in_progress" : true,
                                        "end_acked" : true,
                                        "ended" : true,
                                        "initialized" : false,
                                        "io_watcher_active" : false
                                },
                                "flags" : 
                                {
                                        "dechunk_response" : true,
                                        "https" : false,
                                        "request_body_buffering" : false
                                },
                                "host" : "ala-editor-devel.alocalagent.com",
                                "http_major" : 1,
                                "http_minor" : 1,
                                "http_state" : "COMPLETE",
                                "last_data_receive_time" : 
                                {
                                        "local" : "Wed Dec  7 11:42:06 2016",
                                        "relative" : "0s ago",
                                        "relative_timestamp" : 0,
                                        "timestamp" : 1481136126.9559519
                                },
                                "last_data_send_time" : null,
                                "method" : "GET",
                                "path" : "/",
                                "refcount" : 3,
                                "request_body_already_read" : 0,
                                "request_body_fully_read" : true,
                                "request_body_type" : "NO_BODY",
                                "response_begun" : false,
                                "session_checkout_try" : 0,
                                "started_at" : 
                                {
                                        "local" : "Wed Dec  7 11:42:06 2016",
                                        "relative" : "0s ago",
                                        "relative_timestamp" : 0,
                                        "timestamp" : 1481136126.9561379
                                },
                                "state" : "CHECKING_OUT_SESSION",
                                "sticky_session" : false,
                                "want_keep_alive" : false
                        },
                        "lingering_request_count" : 0,
                        "name" : "1-1",
                        "number" : 1,
                        "output_channel_state" : 
                        {
                                "bytes_buffered" : 
                                {
                                        "bytes" : 0,
                                        "human_readable" : "0 bytes"
                                },
                                "callback_in_progress" : false,
                                "mode" : "IN_MEMORY_MODE",
                                "nbuffers" : 0,
                                "reader_state" : "RS_INACTIVE"
                        },
                        "refcount" : 3,
                        "requests_begun" : 1
                }
        },
        "client_accept_speed" :
        {
                "1h" : 
                {
                        "averaged_over" : "1 hour",
                        "per" : "minute",
                        "value" : 0
                },
                "1m" : 
                {
                        "averaged_over" : "1 minute",
                        "per" : "minute",
                        "value" : 0
                }
        },
        "disconnected_client_count" : 0,
        "disconnected_clients" : {},
        "free_client_count" : 127,
        "free_request_count" : 0,
        "mbuf_pool" : 
        {
                "active_blocks" : 1,
                "active_memory" : 
                {
                        "bytes" : 4096,
                        "human_readable" : "4.0 KB"
                },
                "chunk_size" : 4096,
                "free_blocks" : 0,
                "offset" : 4048,
                "spare_memory" : 
                {
                        "bytes" : 0,
                        "human_readable" : "0 bytes"
                }
        },
        "peak_active_client_count" : 1,
        "pid" : 16620,
        "request_begin_speed" : 
        {
                "1h" : 
                {
                        "averaged_over" : "1 hour",
                        "per" : "minute",
                        "value" : 0
                },
                "1m" : 
                {
                        "averaged_over" : "1 minute",
                        "per" : "minute",
                        "value" : 0
                }
        },
        "server_state" : "ACTIVE",
        "total_bytes_consumed" : 0,
        "total_clients_accepted" : 1,
        "total_requests_begun" : 1,
        "turbocaching" : 
        {
                "fetches" : 0,
                "hit_ratio" : null,
                "hits" : 0,
                "store_success_ratio" : null,
                "store_successes" : 0,
                "stores" : 0
        }
}
### Request handler config (thread 1)
{
        "accept_burst_count" : 32,
        "client_freelist_limit" : 1024,
        "data_buffer_dir" : "/tmp",
        "min_spare_clients" : 128,
        "request_freelist_limit" : 1024,
        "show_version_in_header" : true,
        "single_app_mode" : false,
        "start_reading_after_accept" : true,
        "stat_throttle_rate" : 10
}
### Pool state (simple)
----------- General information -----------
Max pool size : 6
App groups    : 0
Processes     : 0
Requests in top-level queue : 0

----------- Application groups -----------

### mbuf stats

nfree_mbuf_blockq  : 0
nactive_mbuf_blockq: 1
mbuf_block_chunk_size: 4096

### Pool state (XML)
<?xml version="1.0" encoding="iso8859-1" ?>
<info version="3"><passenger_version>5.0.30</passenger_version><group_count>0</group_count><process_count>0</process_count><max>6</max><capacity_used>0</capacity_used><get_wait_list_size>0</get_wait_
list_size><get_wait_list></get_wait_list><supergroups></supergroups></info>

--------------------------------------
[ pid=16620 ] Open files and file descriptors:
COMMAND     PID USER   FD      TYPE             DEVICE SIZE/OFF    NODE NAME
Passenger 16620 root  cwd       DIR              252,2     4096       2 /
Passenger 16620 root  rtd       DIR              252,2     4096       2 /
Passenger 16620 root  txt       REG              252,2  3193584 2361829 /usr/lib64/passenger/support-binaries/PassengerAgent
Passenger 16620 root  mem       REG              252,2    11368 1311316 /usr/lib64/libfreebl3.so
Passenger 16620 root  mem       REG              252,2   471584 1316143 /usr/lib64/libpcre.so.1.2.7
Passenger 16620 root  mem       REG              252,2    37208 1318165 /usr/lib64/libcrypt-nss-2.24.so
Passenger 16620 root  mem       REG              252,2   153184 1312417 /usr/lib64/libselinux.so.1
Passenger 16620 root  mem       REG              252,2   120136 1313859 /usr/lib64/libsasl2.so.3.0.0
Passenger 16620 root  mem       REG              252,2   114680 1315384 /usr/lib64/libresolv-2.24.so
Passenger 16620 root  mem       REG              252,2    15128 1315107 /usr/lib64/libkeyutils.so.1.5
Passenger 16620 root  mem       REG              252,2    61232 1318208 /usr/lib64/libkrb5support.so.0.1
Passenger 16620 root  mem       REG              252,2  1240184 1312026 /usr/lib64/libunistring.so.2.0.0
Passenger 16620 root  mem       REG              252,2  2485224 1318202 /usr/lib64/libcrypto.so.1.0.2j
Passenger 16620 root  mem       REG              252,2   467696 1318203 /usr/lib64/libssl.so.1.0.2j
Passenger 16620 root  mem       REG              252,2   333488 1318330 /usr/lib64/libldap-2.4.so.2.10.7
Passenger 16620 root  mem       REG              252,2    60808 1313405 /usr/lib64/liblber-2.4.so.2.10.7
Passenger 16620 root  mem       REG              252,2    15416 1311353 /usr/lib64/libcom_err.so.2.1
Passenger 16620 root  mem       REG              252,2   201296 1318206 /usr/lib64/libk5crypto.so.3.1
Passenger 16620 root  mem       REG              252,2   951440 1313562 /usr/lib64/libkrb5.so.3.3
Passenger 16620 root  mem       REG              252,2   320832 1312745 /usr/lib64/libgssapi_krb5.so.2.2
Passenger 16620 root  mem       REG              252,2   253136 1314285 /usr/lib64/libnspr4.so
Passenger 16620 root  mem       REG              252,2    19720 1316714 /usr/lib64/libplc4.so
Passenger 16620 root  mem       REG              252,2    15576 1316894 /usr/lib64/libplds4.so
Passenger 16620 root  mem       REG              252,2   188424 1316951 /usr/lib64/libnssutil3.so
Passenger 16620 root  mem       REG              252,2  1229696 1317516 /usr/lib64/libnss3.so
Passenger 16620 root  mem       REG              252,2   164040 1318322 /usr/lib64/libsmime3.so
Passenger 16620 root  mem       REG              252,2   314456 1318325 /usr/lib64/libssl3.so
Passenger 16620 root  mem       REG              252,2    51984 1318170 /usr/lib64/libpsl.so.5.0.1
Passenger 16620 root  mem       REG              252,2   185360 1314874 /usr/lib64/libssh2.so.1.0.1
Passenger 16620 root  mem       REG              252,2   225208 1315028 /usr/lib64/libidn2.so.0.0.11
Passenger 16620 root  mem       REG              252,2   157416 1313991 /usr/lib64/libnghttp2.so.14.9.0
Passenger 16620 root  mem       REG              252,2   112432 1312467 /usr/lib64/libnsl-2.24.so
Passenger 16620 root  mem       REG              252,2  2115832 1312027 /usr/lib64/libc-2.24.so
Passenger 16620 root  mem       REG              252,2    92872 1311913 /usr/lib64/libgcc_s-6.2.1-20160916.so.1
Passenger 16620 root  mem       REG              252,2  1169416 1314541 /usr/lib64/libm-2.24.so
Passenger 16620 root  mem       REG              252,2  1590720 1316312 /usr/lib64/libstdc++.so.6.0.22
Passenger 16620 root  mem       REG              252,2    19736 1312079 /usr/lib64/libdl-2.24.so
Passenger 16620 root  mem       REG              252,2    44056 1315510 /usr/lib64/librt-2.24.so
Passenger 16620 root  mem       REG              252,2   151112 1315382 /usr/lib64/libpthread-2.24.so
Passenger 16620 root  mem       REG              252,2    89520 1313658 /usr/lib64/libz.so.1.2.8
Passenger 16620 root  mem       REG              252,2   523896 1316429 /usr/lib64/libcurl.so.4.4.0
Passenger 16620 root  mem       REG              252,2   212376 1316730 /usr/lib64/libjsoncpp.so.1.7.7
Passenger 16620 root  mem       REG              252,2   155344 1313591 /usr/lib64/libuv.so.1.0.0
Passenger 16620 root  mem       REG              252,2    56584 1315286 /usr/lib64/libev.so.4.0.0
Passenger 16620 root  mem       REG              252,2   182608 1311273 /usr/lib64/ld-2.24.so
Passenger 16620 root    0r      CHR                1,3      0t0    7146 /dev/null
Passenger 16620 root    1w     FIFO               0,10      0t0  131018 pipe
Passenger 16620 root    2w     FIFO               0,10      0t0  131018 pipe
Passenger 16620 root    3u     unix 0xffff9fe4ba830400      0t0  130697 type=STREAM
Passenger 16620 root    5w     FIFO               0,10      0t0  131018 pipe
Passenger 16620 root    8r     FIFO               0,10      0t0  130740 pipe
Passenger 16620 root    9w     FIFO               0,10      0t0  130740 pipe
Passenger 16620 root   10r     FIFO               0,10      0t0  130741 pipe
Passenger 16620 root   11w     FIFO               0,10      0t0  130741 pipe
Passenger 16620 root   12r     FIFO               0,10      0t0  130742 pipe
Passenger 16620 root   13w     FIFO               0,10      0t0  130742 pipe
Passenger 16620 root   14u     unix 0xffff9fe4ba832c00      0t0  130743 /tmp/passenger.OXbNxHR/agents.s/core type=STREAM
Passenger 16620 root   15u     unix 0xffff9fe4ba832800      0t0  130745 /tmp/passenger.OXbNxHR/agents.s/core_api type=STREAM
Passenger 16620 root   16r      CHR                1,9      0t0    7151 /dev/urandom
Passenger 16620 root   17u  a_inode               0,11        0    7141 [eventpoll]
Passenger 16620 root   18u  a_inode               0,11        0    7141 [eventfd]
Passenger 16620 root   19r     FIFO               0,10      0t0  130748 pipe
Passenger 16620 root   20w     FIFO               0,10      0t0  130748 pipe
Passenger 16620 root   21u  a_inode               0,11        0    7141 [eventpoll]
Passenger 16620 root   22r     FIFO               0,10      0t0  130749 pipe
Passenger 16620 root   23w     FIFO               0,10      0t0  130749 pipe
Passenger 16620 root   24u  a_inode               0,11        0    7141 [eventfd]
Passenger 16620 root   25u  a_inode               0,11        0    7141 [eventpoll]
Passenger 16620 root   26u  a_inode               0,11        0    7141 [eventfd]
Passenger 16620 root   27u  a_inode               0,11        0    7141 [eventpoll]
Passenger 16620 root   28u  a_inode               0,11        0    7141 [eventfd]
Passenger 16620 root   29u  a_inode               0,11        0    7141 [eventpoll]
Passenger 16620 root   30r     FIFO               0,10      0t0  130750 pipe
Passenger 16620 root   31w     FIFO               0,10      0t0  130750 pipe
Passenger 16620 root   32u  a_inode               0,11        0    7141 [eventfd]
Passenger 16620 root   33u     unix 0xffff9fe4ba830800      0t0  131016 /tmp/passenger.OXbNxHR/agents.s/core type=STREAM
--------------------------------------
[ pid=16620 ] Dumping a more detailed backtrace with crash-watch...
Found gdb at: /usr/bin/gdb
*******************************************************
*
*    Current thread (1) backtrace
*
*******************************************************

    #0  0x00007f5828fbed63 in select () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x000055ebfb0122d3 in oxt::syscalls::select(int, fd_set*, fd_set*, fd_set*, timeval*) ()
    No symbol table info available.
    #2  0x000055ebfae88c6d in waitForExitEvent() ()
    No symbol table info available.
    #3  0x000055ebfae8b07c in runCore() ()
    No symbol table info available.
    #4  0x000055ebfae8d162 in coreMain(int, char**) ()
    No symbol table info available.
    #5  0x000055ebfade7af5 in main ()
    No symbol table info available.

*******************************************************
*
*    All thread backtraces
*
*******************************************************

########### Thread 8 (Thread 0x7f582ace6700 (LWP 16678)) ###########

    #0  0x00007f5828fc8f41 in clone () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x00007f5829f42600 in ?? () from /lib64/libpthread.so.0
    No symbol table info available.
    #2  0x00007f582ace6700 in ?? ()
    No symbol table info available.
    #3  0x0000000000000000 in ?? ()
    No symbol table info available.

########### Thread 7 (Thread 0x7f5823314700 (LWP 16629)) ###########

    #0  0x00007f5828fc9563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x000055ebfae7d863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x000055ebfb00f3b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x000055ebfae3b020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value
<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x000055ebfafde246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007f5829f426ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007f5828fc8f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 6 (Thread 0x7f5823b15700 (LWP 16628)) ###########

    #0  0x00007f5828fc9563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x00007f582aa4bac5 in epoll_poll () from /lib64/libev.so.4
    No symbol table info available.
    #2  0x00007f582aa4e089 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #3  0x000055ebfb00f3b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #4  0x000055ebfae3b020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value
<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #5  0x000055ebfafde246 in thread_proxy ()
    No symbol table info available.
    #6  0x00007f5829f426ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #7  0x00007f5828fc8f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 5 (Thread 0x7f582ad67700 (LWP 16627)) ###########

    #0  0x00007f5828fc9563 in epoll_wait () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x000055ebfae7d863 in Passenger::pollLibuv(Passenger::BackgroundEventLoop*) ()
    No symbol table info available.
    #2  0x000055ebfb00f3b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #3  0x000055ebfae3b020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value
<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #4  0x000055ebfafde246 in thread_proxy ()
    No symbol table info available.
    #5  0x00007f5829f426ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #6  0x00007f5828fc8f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 4 (Thread 0x7f5824316700 (LWP 16626)) ###########

    #0  0x00007f5829f4c46d in raise () from /lib64/libpthread.so.0
    No symbol table info available.
    #1  0x000055ebfae213ea in Passenger::abortHandler(int, siginfo_t*, void*) ()
    No symbol table info available.
    #2  <signal handler called>
    No symbol table info available.
    #3  0x000055ebfaefdd82 in Passenger::ApplicationPool2::Group::createProcessObject(Json::Value const&) ()
    No symbol table info available.
    #4  0x000055ebfaefe51d in Passenger::ApplicationPool2::Group::initialize() ()
    No symbol table info available.
    #5  0x000055ebfaf01893 in Passenger::ApplicationPool2::Pool::createGroup(Passenger::ApplicationPool2::Options const&) ()
    No symbol table info available.
    #6  0x000055ebfaf15161 in Passenger::ApplicationPool2::Pool::createGroupAndAsyncGetFromIt(Passenger::ApplicationPool2::Options const&, Passenger::ApplicationPool2::GetCallback const&, boost::cont
ainer::vector<boost::function<void ()>, boost::container::new_allocator<boost::function<void ()> > >&) ()
    No symbol table info available.
    #7  0x000055ebfaf16597 in Passenger::ApplicationPool2::Pool::asyncGet(Passenger::ApplicationPool2::Options const&, Passenger::ApplicationPool2::GetCallback const&, bool, Passenger::UnionStation::
StopwatchLog**) ()
    No symbol table info available.
    #8  0x000055ebfaf45dd0 in Passenger::Core::Controller::asyncGetFromApplicationPool(Passenger::Core::Request*, Passenger::ApplicationPool2::GetCallback) ()
    No symbol table info available.
    #9  0x000055ebfaf604f6 in Passenger::Core::Controller::checkoutSession(Passenger::Core::Client*, Passenger::Core::Request*) ()
    No symbol table info available.
    #10 0x000055ebfaf61d24 in Passenger::Core::Controller::onRequestBegin(Passenger::Core::Client*, Passenger::Core::Request*) ()
    No symbol table info available.
    #11 0x000055ebfaf7fe29 in Passenger::ServerKit::HttpServer<Passenger::Core::Controller, Passenger::Core::Client>::processClientDataWhenParsingHeaders(Passenger::Core::Client*, Passenger::Core::Re
quest*, Passenger::MemoryKit::mbuf const&, int) ()
    No symbol table info available.
    #12 0x000055ebfaf80878 in Passenger::ServerKit::HttpServer<Passenger::Core::Controller, Passenger::Core::Client>::onClientDataReceived(Passenger::Core::Client*, Passenger::MemoryKit::mbuf const&,
 int) ()
    No symbol table info available.
    #13 0x000055ebfaed7a96 in Passenger::ServerKit::BaseServer<Passenger::Core::Controller, Passenger::Core::Client>::_onClientDataReceived(Passenger::ServerKit::Channel*, Passenger::MemoryKit::mbuf 
const&, int) ()
    No symbol table info available.
    #14 0x000055ebfae47a33 in Passenger::ServerKit::Channel::callDataCallbackWithoutRefGuard() ()
    No symbol table info available.
    #15 0x000055ebfae483b5 in Passenger::ServerKit::Channel::feedWithoutRefGuard(Passenger::MemoryKit::mbuf&&) ()
    No symbol table info available.
    #16 0x000055ebfae51c8c in Passenger::ServerKit::FdSourceChannel::onReadableWithoutRefGuard() ()
    No symbol table info available.
    #17 0x000055ebfae5206e in Passenger::ServerKit::FdSourceChannel::_onReadable(ev_loop*, ev_io*, int) ()
    No symbol table info available.
    #18 0x00007f582aa4b113 in ev_invoke_pending () from /lib64/libev.so.4
    No symbol table info available.
    #19 0x00007f582aa4e708 in ev_run () from /lib64/libev.so.4
    No symbol table info available.
    #20 0x000055ebfb00f3b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #21 0x000055ebfae3b020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value
<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #22 0x000055ebfafde246 in thread_proxy ()
    No symbol table info available.
    #23 0x00007f5829f426ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #24 0x00007f5828fc8f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 3 (Thread 0x7f582adf2700 (LWP 16624)) ###########

    #0  0x00007f5829f48809 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
       from /lib64/libpthread.so.0
    No symbol table info available.
    #1  0x000055ebfaf12f15 in Passenger::ApplicationPool2::Pool::garbageCollect(boost::shared_ptr<Passenger::ApplicationPool2::Pool>) ()
    No symbol table info available.
    #2  0x000055ebfaf21527 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<Passenger::ApplicationPool2::Pool>), boost::_bi::list1<boost::_bi
::value<boost::shared_ptr<Passenger::ApplicationPool2::Pool> > > >, void>::invoke(boost::detail::function::function_buffer&) ()
    No symbol table info available.
    #3  0x000055ebfaf309fb in boost::function0<void>::operator()() const ()
    No symbol table info available.
    #4  0x000055ebfaf34c39 in oxt::dynamic_thread_group::thread_main(oxt::dynamic_thread_group::thread_func_data*) ()
    No symbol table info available.
    #5  0x000055ebfb00f3b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #6  0x000055ebfae3b020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value
<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #7  0x000055ebfafde246 in thread_proxy ()
    No symbol table info available.
    #8  0x00007f5829f426ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #9  0x00007f5828fc8f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 2 (Thread 0x7f582ae33700 (LWP 16623)) ###########

    #0  0x00007f5829f4bc7d in nanosleep () from /lib64/libpthread.so.0
    No symbol table info available.
    #1  0x000055ebfb012ecb in oxt::syscalls::nanosleep(timespec const*, timespec*)
        ()
    No symbol table info available.
    #2  0x000055ebfb013123 in oxt::syscalls::usleep(unsigned int) ()
    No symbol table info available.
    #3  0x000055ebfaf19bf7 in Passenger::ApplicationPool2::Pool::collectAnalytics(boost::shared_ptr<Passenger::ApplicationPool2::Pool>) ()
    No symbol table info available.
    #4  0x000055ebfaf21527 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<Passenger::ApplicationPool2::Pool>), boost::_bi::list1<boost::_bi
::value<boost::shared_ptr<Passenger::ApplicationPool2::Pool> > > >, void>::invoke(boost::detail::function::function_buffer&) ()
    No symbol table info available.
    #5  0x000055ebfaf309fb in boost::function0<void>::operator()() const ()
    No symbol table info available.
    #6  0x000055ebfaf34c39 in oxt::dynamic_thread_group::thread_main(oxt::dynamic_thread_group::thread_func_data*) ()
    No symbol table info available.
    #7  0x000055ebfb00f3b5 in oxt::thread::thread_main(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>) ()
    No symbol table info available.
    #8  0x000055ebfae3b020 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value
<boost::function<void ()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() ()
    No symbol table info available.
    #9  0x000055ebfafde246 in thread_proxy ()
    No symbol table info available.
    #10 0x00007f5829f426ca in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #11 0x00007f5828fc8f6f in clone () from /lib64/libc.so.6
    No symbol table info available.

########### Thread 1 (Thread 0x7f582ae55840 (LWP 16620)) ###########

    #0  0x00007f5828fbed63 in select () from /lib64/libc.so.6
    No symbol table info available.
    #1  0x000055ebfb0122d3 in oxt::syscalls::select(int, fd_set*, fd_set*, fd_set*, timeval*) ()
    No symbol table info available.
    #2  0x000055ebfae88c6d in waitForExitEvent() ()
    No symbol table info available.
    #3  0x000055ebfae8b07c in runCore() ()
    No symbol table info available.
    #4  0x000055ebfae8d162 in coreMain(int, char**) ()
    No symbol table info available.
    #5  0x000055ebfade7af5 in main ()
    No symbol table info available.
[ 2016-12-07 11:42:07.7881 16620/7f582ace6700 age/Cor/Spa/PipeWatcher.h:95 ]: Cannot read from process 0 stdout: Bad file descriptor (errno=9)
jbeima commented 7 years ago

All 7 rails applications worked just fine before I upgraded my Fedora 24 -> 25.

Now not one of them works.

HOWEVER

The Rails console/irb works fine.

There is nothing in the rails logs at all. It is as if passenger never even gets to the point of looked at the Rails app.

jbeima commented 7 years ago

I have even tried creating a new app and the new app does not work either. Same error.

jbeima commented 7 years ago

Just in case this helps, these are the installed rpms with Fedora 25.

Interestingly enough Fedora now longer provides the gem "The Ruby Racer".

ruby-2.3.3-61.1.fc25.x86_64
ruby-devel-2.3.3-61.1.fc25.x86_64
rubygem-actioncable-5.0.0.1-1.fc25.noarch
rubygem-actionmailer-5.0.0.1-1.fc25.noarch
rubygem-actionpack-5.0.0.1-2.fc25.noarch
rubygem-actionview-5.0.0.1-2.fc25.noarch
rubygem-activejob-5.0.0.1-1.fc25.noarch
rubygem-activemodel-5.0.0.1-1.fc25.noarch
rubygem-activerecord-5.0.0.1-1.fc25.noarch
rubygem-activesupport-5.0.0.1-1.fc25.noarch
rubygem-ansi-1.5.0-3.fc24.noarch
rubygem-arel-7.0.0-1.fc25.noarch
rubygem-bcrypt-3.1.10-4.fc24.x86_64
rubygem-bigdecimal-1.2.8-61.1.fc25.x86_64
rubygem-binding_of_caller-0.7.2-4.fc24.noarch
rubygem-builder-3.2.2-5.fc24.noarch
rubygem-bundler-1.12.5-1.fc25.noarch
rubygem-byebug-9.0.6-1.fc25.x86_64
rubygem-capybara-2.4.1-3.fc24.noarch
rubygem-climate_control-0.0.3-5.fc21.noarch
rubygem-cliver-0.3.2-4.fc24.noarch
rubygem-cocaine-0.5.7-3.fc24.noarch
rubygem-coffee-rails-4.2.1-1.fc25.noarch
rubygem-coffee-script-2.4.1-3.fc25.noarch
rubygem-coffee-script-source-1.10.0-2.fc24.noarch
rubygem-concurrent-ruby-1.0.2-1.fc25.noarch
rubygem-daemon_controller-1.2.0-4.fc24.noarch
rubygem-debug_inspector-0.0.2-5.fc24.x86_64
rubygem-did_you_mean-1.0.0-61.1.fc25.x86_64
rubygem-erubis-2.7.0-11.fc25.noarch
rubygem-execjs-2.7.0-1.fc25.noarch
rubygem-ffi-1.9.10-3.fc24.x86_64
rubygem-font-awesome-rails-4.7.0.0-1.fc25.noarch
rubygem-globalid-0.3.6-1.fc25.noarch
rubygem-hike-2.1.3-2.fc24.noarch
rubygem-i18n-0.7.0-3.fc24.noarch
rubygem-io-console-0.4.5-61.1.fc25.x86_64
rubygem-jbuilder-2.5.0-1.fc25.noarch
rubygem-jquery-rails-4.0.4-3.fc25.noarch
rubygem-json-1.8.3-103.fc25.x86_64
rubygem-kaminari-0.16.1-3.fc24.noarch
rubygem-listen-3.1.5-1.fc25.noarch
rubygem-loofah-2.0.3-2.fc24.noarch
rubygem-mail-2.6.4-1.fc25.noarch
rubygem-method_source-0.8.2-3.fc24.noarch
rubygem-mime-types-3.1-1.fc25.noarch
rubygem-mime-types-data-3.2016.0521-1.fc25.noarch
rubygem-minitest-5.8.5-61.1.fc25.noarch
rubygem-multi_json-1.10.1-3.fc24.noarch
rubygem-mysql2-0.4.4-1.fc25.x86_64
rubygem-net-http-persistent-2.9.4-6.fc24.noarch
rubygem-nio4r-1.2.1-2.fc25.x86_64
rubygem-nokogiri-1.6.8.1-1.fc25.x86_64
rubygem-orm_adapter-0.5.0-2.fc21.noarch
rubygem-pkg-config-1.1.7-2.fc24.noarch
rubygem-polyglot-0.3.4-4.fc24.noarch
rubygem-psych-2.1.0-61.1.fc25.x86_64
rubygem-puma-3.6.0-3.fc25.x86_64
rubygem-rack-2.0.1-1.fc25.noarch
rubygem-rack-test-0.6.3-3.fc24.noarch
rubygem-rails-5.0.0.1-1.fc25.noarch
rubygem-rails-deprecated_sanitizer-1.0.3-4.fc24.noarch
rubygem-rails-dom-testing-2.0.1-1.fc25.noarch
rubygem-rails-html-sanitizer-1.0.3-2.fc24.noarch
rubygem-railties-5.0.0.1-2.fc25.noarch
rubygem-rake-10.4.2-100.fc24.1.noarch
rubygem-rb-inotify-0.9.7-1.fc25.noarch
rubygem-rdoc-4.2.2-2.fc25.noarch
rubygem-ref-1.0.5-4.fc24.noarch
rubygems-2.5.2-61.1.fc25.noarch
rubygem-sass-3.4.22-1.fc25.noarch
rubygem-sass-rails-5.0.6-1.fc25.noarch
rubygem-slim-2.0.2-1.fc21.noarch
rubygem-spring-1.7.2-1.fc25.noarch
rubygem-spring-watcher-listen-2.0.0-1.fc25.noarch
rubygem-sprockets-3.7.0-1.fc25.noarch
rubygem-sprockets-rails-3.1.1-1.fc25.noarch
rubygem-sqlite3-1.3.11-2.fc24.x86_64
rubygem-temple-0.6.7-3.fc24.noarch
rubygem-term-ansicolor-1.3.0-5.fc24.noarch
rubygem-thor-0.19.1-5.fc25.noarch
rubygem-thread_safe-0.3.5-3.fc24.noarch
rubygem-tilt-2.0.4-1.fc25.noarch
rubygem-tins-1.0.0-6.fc24.noarch
rubygem-treetop-1.6.3-2.fc24.noarch
rubygem-turbolinks-5.0.0-1.fc25.noarch
rubygem-turbolinks-source-5.0.0-1.fc25.noarch
rubygem-tzinfo-1.2.2-3.fc24.noarch
rubygem-uglifier-3.0.0-1.fc25.noarch
rubygem-web-console-3.3.1-1.fc25.noarch
rubygem-websocket-driver-0.6.4-1.fc25.x86_64
rubygem-websocket-extensions-0.1.2-1.fc25.noarch
rubygem-will_paginate-3.1.5-1.fc25.noarch
rubygem-xpath-2.0.0-4.fc25.noarch
ruby-irb-2.3.3-61.1.fc25.noarch
ruby-libs-2.3.3-61.1.fc25.x86_64
rubypick-1.1.1-5.fc24.noarch
jbeima commented 7 years ago

Just in case this makes a difference:

source 'https://rubygems.org'

# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '5.0.0.1'
# Use mysql as the database for Active Record
gem 'mysql2', '0.4.4'
# Use SCSS for stylesheets
gem 'sass-rails', '5.0.6'
# Use Uglifier as compressor for JavaScript assets
gem 'uglifier', '3.0.0'
# Use CoffeeScript for .js.coffee assets and views
gem 'coffee-rails', '4.2.1'
# See https://github.com/sstephenson/execjs#readme for more supported runtimes
gem 'therubyracer',  '0.11.0', platforms: :ruby

# Use jquery as the JavaScript library
gem 'jquery-rails', '4.0.4'
# Turbolinks makes following links in your web application faster. Read more: https://github.com/rails/turbolinks
gem 'turbolinks', '2.5.3'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
# gem 'jbuilder', '~> 2.0'
# bundle exec rake doc:rails generates the API under doc/api.
# gem 'sdoc', '~> 0.4.0',          group: :doc

# Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
# gem 'spring',        group: :development

# Use ActiveModel has_secure_password
gem 'bcrypt', '3.1.10'

# Use unicorn as the app server
# gem 'unicorn'

# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development

# Use debugger
# gem 'debugger', group: [:development, :test]

# Use Kaminari
gem 'kaminari', '0.16.1'

# Use Bootsrap
gem 'bootstrap', '4.0.0.alpha5'
gem 'autoprefixer-rails', '6.4.0.2'
gem 'capybara', '2.4.1'
gem 'chunky_png', '1.2.7'
gem 'compass', '1.0.1'
gem 'compass-core', '1.0.1'
gem 'ffi', '1.9.10'
gem 'minitest-reporters'
gem 'multi_json', '1.10.1'
gem 'poltergeist', '1.10.0'
gem 'rb-inotify', '0.9.7'
gem 'slim', '2.0.2'
gem 'slim-rails', '0.2.1'
gem 'temple', '0.6.7'
gem 'term-ansicolor', '1.3.0'
gem 'websocket-driver', '0.6.4'

# Use Font Awesome - Rails
#gem 'font-awesome-rails', '4.7.0.0'
gem 'font-awesome-rails'

# Editor - Bootsy
gem 'bootsy'

# Editor - TinyMCE
gem 'tinymce-rails'

# Editor - CKEditor
#gem 'ckeditor'
#gem 'cocaine', '0.5.7'
#gem 'orm_adapter', '0.5.0'
#gem 'websocket-extensions', '0.1.2'

# Editors - Floala
gem 'wysiwyg-rails'

# Fix all the auto-updated Gems
gem 'arel', '7.0.0'
gem 'concurrent-ruby', '1.0.2'
gem 'execjs', '2.7.0'
gem 'globalid', '0.3.6'
gem 'json', '1.8.3'
gem 'listen', '3.1.5'
gem 'mail', '2.6.4'
gem 'mime-types', '3.1'
gem 'minitest', '5.8.5'
gem 'puma', '3.6.0'
gem 'rack', '2.0.1'
gem 'rails-dom-testing', '2.0.1'
gem 'rake', '10.4.2'
gem 'ref', '1.0.5'
gem 'sprockets', '3.7.0'
gem 'sprockets-rails', '3.1.1'
gem 'sqlite3', '1.3.11'
gem 'thor', '0.19.1'
gem 'tilt', '2.0.4'
gem 'tins', '1.0.0'

rails console and irb both launch and appear to work just fine.

I did have to install the rubygem-listen, since Rails is dependent on it, but it is not included in the dependencies.

OnixGH commented 7 years ago

@jbeima Thanks for reporting. The raise() in your logs confirms the same crash location.

jbeima commented 7 years ago

@OnixGH

Fedora now has an extremely reliable upgrade process. Pretty much three (four) steps:

01) dnf upgrade --refresh 02) reboot - Optional if kernel was updated 03) dnf system-upgrade download --refresh --releasever=25 04) dnf system-upgrade reboot

This development server started out as Fedora 21 and was upgraded through the above mentioned steps to 22 to 23 to 24 to 25.

In all cases Apache, Passenger, Ruby, and Rails worked just fine, with the exception of the upgrade to 25.

I have ZERO interest in compiling ANYTHING. Whenever possible I have even avoided gems to be able to use RedHat/Fedora provided ones.

You can't have a development environment that freely updates anything at any time.

Plus I have no interest in riddling my servers with devel rpms and srpms.

Passenger crashes on a brand new just created app withnothing yet done to it. Can not even get the "Splash" screen so to speak.

I have not and will not look at source tarballs.

Given this, are you saying that the rpm that is included with Fedora is broken, since if that is the case this should be submitted to the FedoraProject and not here. Unless it is broken because of something your project has done/caused.

jbeima commented 7 years ago

@OnixGH

The above message aside. I have stripped out all version numbers from the gems and allowed the system to install all gems from source via the bundle install command. Nothing.

Please remember in the app.log directory nothing is added to any log file.

Only the Apache general error.log file is written to. It is as if Passenger never even gets to the point of trying to load the app.

Also rails console works just fine, so it is not Ruby, Rails, or the gems.

Honestly gents, I am happy to try anything that does not involve compiling anything.

I have an extremely simply virtual host file:

<VirtualHost 1.2.3.4:80>
   ServerName app-01.domain.com
   ErrorLog logs/app-01.domain.com-error_log
   CustomLog logs/app-01.domain.com-access_log combined
   DocumentRoot /home/rails-devel/development/rails/app-01/public
   PassengerAppRoot /home/rails-devel/development/rails/app-01
   RailsEnv development
   RailsBaseURI /
   <Directory /home/rails-devel/development/rails/app-01/public>
      # Relax Apache security settings
      AllowOverride all
      # MultiViews must be turned off
      Options -MultiViews +FollowSymLinks
      Require all granted
   </Directory>
</VirtualHost>

Same for each of the 7 projects with obvious updates.

I did add the +FollowSymLinks to this setup with Fedora 25, since I have read it is needed, and I never needed it before.

root     28325  0.0  0.5 266100 12136 ?        Ss   Dec07   0:04 /usr/sbin/httpd -DFOREGROUND
root     30236  0.0  0.4 442596  8348 ?        Ssl  03:41   0:00 Passenger watchdog
nobody   30245  0.0  0.4 453260  9112 ?        Sl   03:41   0:00 Passenger ust-router
apache   30256  0.0  0.4 481380  9680 ?        Sl   03:41   0:04 /usr/sbin/httpd -DFOREGROUND
apache   30259  0.0  0.4 284708  9728 ?        Sl   03:41   0:04 /usr/sbin/httpd -DFOREGROUND
apache   30262  0.0  0.4 284708  9728 ?        Sl   03:41   0:04 /usr/sbin/httpd -DFOREGROUND
apache   30263  0.0  0.4 284708  9728 ?        Sl   03:41   0:04 /usr/sbin/httpd -DFOREGROUND
apache   30278  0.0  0.4 284708  9728 ?        Sl   03:41   0:04 /usr/sbin/httpd -DFOREGROUND
apache   30551  0.0  0.4 284708  9728 ?        Sl   08:53   0:01 /usr/sbin/httpd -DFOREGROUND
root     30937  0.0  0.4 588756  9288 ?        Sl   08:53   0:00 Passenger core

This is what I see as a process list.

OnixGH commented 7 years ago

@jbeima we're not affiliated with the dnf packages so we can't fix anything there..

Passenger supports such a wide variety of systems and technologies that many times the problem does not actually lie with Passenger. But we're always going out of our way to make sure of that and point our users in the right direction.

That's why I've been asking you (both) how exactly you've installed Passenger so I can see if I can at least reproduce the crash and maybe see what's wrong with the install. I'm still waiting for an answer because a simple dnf install passenger doesn't appear sufficient (it complains about a nonexistent passenger-devel-5.0.30 package when I run passenger -- no idea what that's supposed to be).

Btw. although passenger provides very friendly compilation guidance (it's real easy!), installing Passenger standalone via source tarball should not require any compilation on Fedora (auto-download of compatible precompiled binaries from our servers).

If you crash-test that with one of your apps, we'd have the important clue whether it's related to our Passenger binary or the dnf version.

jbeima commented 7 years ago

@OnixGH

I am not prepared to compromise my development platform by riddling it with compilers and source code that are not needed.

NOW given that, I don't mind helping in setting up a test bed. If you have a VPS I can have access to, or I can set a temporary one up.

You should not need the devel rpm, not sure why installing passenger would even ask for it.

devel rpms simply carry the header files. So the .h files. They are only needed when something needs to be compiled referencing Passenger's libraries/headers.

Here are the rpms that you would need and should not require anything else:

httpd-2.4.23-4.fc25.x86_64 httpd-filesystem-2.4.23-4.fc25.noarch httpd-tools-2.4.23-4.fc25.x86_64 http-parser-2.7.1-3.fc25.x86_64 mod_passenger-5.0.30-1.fc25.x86_64 passenger-5.0.30-1.fc25.x86_64

A simple:

dnf install httpd mod_passenger mod_ssl rubygem-rails

Should get you what you need.

I apologise for the delay in the response I have been pulled in several directions today.

Part of this might be done quicker in email.

jbeima commented 7 years ago

By the way your response was correct there is no devel rpm, not sure why Passenger would even need one, I don't think anything compiles against it?

jbeima commented 7 years ago

This could be an important piece of the puzzle?

I am not running Passenger directly, I am running mod_passenger through Apache.

Not sure if that makes any difference to all this or not.

OnixGH commented 7 years ago

@jbeima Installed Passenger 5.0.30 using your commands and reproduced the same crash with the rails helloworld app.

The application works fine on the same server if I point Apache to the Passenger 5.0.30 install as recommended / supported by Phusion.

In conclusion there are two solutions to this issue: 1) Use one of our supported installation methods 2) Fix Fedora package

We can't really do anything about 2), as far as I could see the Apache integration module from the Fedora package was OK, the app still worked if I pointed Apache to the Fedora package mod_passenger.so, but with the self-installed passenger root so the problem is more likely to be in the compiled PassengerAgent they supply.

jbeima commented 7 years ago

I have isolated and confirmed where this bug is.

If you download the source tarball of Passenger from here:

https://www.phusionpassenger.com/latest_stable_tarball

The untar it and do the following:

01) cd passenger-5.0.30 02) ./bin/passenger-install-apache2-module 03) cp buildout/support-binaries/PassengerAgent /usr/lib64/passenger/support-binaries 04) systemnctl restart|start httpd

Passenger then works fine.

This one file is broken. There might be more but this at least brings all of the offline down systems back online.

NOTES: When running passenger-install-apache2-module select only Ruby when your asked about the 4 languages. Unless you want the other 3. Seems pointless for a Rails server. I am unaware of what the default choices are for Fedora.

Tcharl commented 7 years ago

Thank @jbeima ! will create a ticket on fedora issuetracker if there isn't

FooBarWidget commented 7 years ago

@Tcharl If you have created or found a ticket, could you link to that ticket from here? I'm curious about any analysis the packager may have.

@jbeima: In another you thread you said this:

If your already doing a fix compile for Cent & RHEL, why not do a release for Fedora or at least help us to get it fixed.

I understand your frustration. I ask for your understanding w.r.t. our situation as well. We support a ton of different distributions and platforms, and it is very challenging to package for all of them. Since time is a limited resource, we have to carefully balance our time between packaging, maintenance and feature development.

Fedora is a very active target and has (compared to Debian, Ubuntu, CentOS and RHEL) relatively few users. Analysis has shown that, given the resources we have at our disposal, packaging for Fedora and updating these packages on every Fedora release cycle would place too much of a burden on our team for too little gain. If we have unlimited resources, we would have gladly helped you.

The next best things we can offer you are:

  1. A set of source RPMs which you are free to modify for Fedora compatibility.
  2. A tarball which makes use of precompiled generic Linux binaries.
Tcharl commented 7 years ago

Sure, here's the fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1403569

OlegGirko commented 7 years ago

Looks like this is still a fault of Passenger to some degree.

Look at this bug and its description.

Of course, it's Fedora's fault that unbundling of jsoncpp went bad. But it's Passenger's fault that it bundled this library in the first place and then patched it incompatible way without upstreaming this patch.

I could understand that you may feel an urge to bundle some essential libraries to make development easier (less dependencies to install). But please resist this temptation.

Remember, that users of your software are not end-users who do make install (or any equivalent of it) on their system. There are people like that, but they are a minority. Real users of your software are packagers that package your software for Linux distributions. This is how your software reaches most end-users.

We (end-users) are not interested in building and installing software from source. What we want is just to install your software with dnf install (or apt-get install or any GUI equivalent), so it not only starts running on our systems, but also is included in automatic update system.

I have hundreds of packages on my Fedora system, and Passenger is just one of them. I don't want to check manually for security patches for all of these packages, so I prefer to trust Fedora packagers to push updates to Fedora package repos and keep my system updated this way. Hence, your recommended way to install Passenger is actually harmful and not recommended because it leads to outdated and vulnerable software installed and never updated.

Returning back to library bundling, all sane Linux distribution try hard to unbundle as much libraries as possible. Bundled libraries lead to not only wasted disk and memory (several packages using the same library), but also security vulnerabilities. If there is a bug found in some library, it's enough to update just the package containing this library to fix this bug in all packages using it. But bundling this library with your software is a way to ensure that the fix will be missed.

You may argue that bundling a library makes life easier for people using systems that have no proper package manager (Windows, MacOS) and where it's difficult to install all dependencies. Then you should provide an easy way to choose whether to build your software with bundled version or with the one provided by the system.

And what you should never ever do is patch bundled library without upstreaming your changes. This makes life more difficult for everybody, including you: you have to either keep your change and rebase it on every new version of library, or stick with old library version forever. And it makes life extremely difficult for packagers who try to unbundle this library.

So, this particular bug is caused by botched attempt to unbundle jsoncpp by Fedora packager, but the root cause is patched bundled library. Never patch bundled libraries. Instead, provide a workaround in your own source code. And make unbundling easy.

HerbertKoelman commented 6 years ago

Does anyone know if this Fedora 25 packaging issue still exists in fedora 26 or higher ?

Tcharl commented 6 years ago

The last time I tried it was always here, 26 and even for 27. The hack consists in manually installing passenger on the machine, without using dnf/yum

HerbertKoelman commented 6 years ago

The hack did indeed do the trick 😊

Thanks

Herbert

Le 18 mars 2018 à 19:27, Charlie Mordant notifications@github.com a écrit :

The last time I tried it was always here, 26 and even for 27. The hack consists in manually installing passenger on the machine, without using dnf/yum

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.