Check failed: may_start_threads

[ellisonpatterson]

I have tested this with the recommended build of ngx_pagespeed, the latest build, with nginx 1.4.0, 1.4.1,1.5.1 and it keeps happening and I'm giving up :( it happens with my filters disabled, memcached server disabled and whatnot. Here is the error logs:

2013/06/18 00:46:17 [alert] 13630#0: aborting 2013/06/18 00:46:17 [alert] 13626#0: aborting 2013/06/18 00:46:17 [alert] 13629#0: open socket #30 left in connection 2 2013/06/18 00:46:17 [alert] 13629#0: aborting 2013/06/18 00:46:17 [alert] 13624#0: [ngx_pagespeed 1.5.27.3-3005] [0618/004617:FATAL:ngx_thread_system.cc(43)] Check failed: may_startthreads. Backtrace: nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4af82a] nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4a4463] nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4ac869] nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4a253c] nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x60634e] /lib64/libpthread.so.0() [0x327f407851] /lib64/libc.so.6(clone+0x6d) [0x327ece890d] 2013/06/18 00:47:01 [alert] 13666#0: open socket #24 left in connection 2 2013/06/18 00:47:01 [alert] 13666#0: aborting 2013/06/18 00:47:01 [alert] 13669#0: open socket #28 left in connection 2 2013/06/18 00:47:01 [alert] 13669#0: aborting 2013/06/18 00:47:01 [alert] 13671#0: open socket #32 left in connection 2 2013/06/18 00:47:01 [alert] 13671#0: aborting 2013/06/18 00:47:04 [alert] 13667#0: open socket #26 left in connection 2 2013/06/18 00:47:04 [alert] 13667#0: aborting 2013/06/18 00:47:06 [alert] 13670#0: open socket #30 left in connection 2 2013/06/18 00:47:06 [alert] 13670#0: aborting 2013/06/18 00:47:06 [alert] 13665#0: [ngx_pagespeed 1.5.27.3-3005] [0618/004706:FATAL:ngx_thread_system.cc(43)] Check failed: may_startthreads. Backtrace: nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4af82a] nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4a4463] nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4ac869] nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4a253c] nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x60634e] /lib64/libpthread.so.0() [0x327f407851] /lib64/libc.so.6(clone+0x6d) [0x327ece890d] 2013/06/18 00:48:02 [alert] 13760#0: open socket #30 left in connection 2 2013/06/18 00:48:02 [alert] 13760#0: aborting 2013/06/18 00:48:02 [alert] 13758#0: open socket #26 left in connection 2 2013/06/18 00:48:02 [alert] 13758#0: aborting 2013/06/18 00:48:02 [alert] 13761#0: open socket #32 left in connection 2 2013/06/18 00:48:02 [alert] 13761#0: aborting 2013/06/18 00:48:02 [alert] 13759#0: open socket #28 left in connection 2 2013/06/18 00:48:02 [alert] 13759#0: aborting 2013/06/18 00:48:04 [alert] 13757#0: open socket #24 left in connection 2 2013/06/18 00:48:04 [alert] 13757#0: aborting

[jeffkaufman]

Are you using any other modules? Could you paste your ./configure line? Could you paste your config?

[ellisonpatterson]

./configure --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-cc-opt='-O2 -g' --with-http_geoip_module --with-http_spdy_module --with-http_gunzip_module

Config:

worker_processes 5;
worker_rlimit_nofile 30000;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections 30000;
    multi_accept on;
    use epoll;
}

http {
    resolver 8.8.8.8;

    include /etc/nginx/mime.types;

    limit_req_zone $binary_remote_addr zone=antiddosphp:10m rate=1r/s;
    limit_req_zone $binary_remote_addr zone=antiddos:10m rate=10r/s;

    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
            '$status $body_bytes_sent "$http_referer" '
            '"$http_user_agent" "$http_x_forwarded_for"';

    access_log off;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    disable_symlinks if_not_owner;

    server_tokens off;

    keepalive_timeout 15;
    client_header_timeout 10;
    client_body_timeout 10;
    send_timeout 10;
    reset_timedout_connection on;
    connection_pool_size 256;

    open_file_cache max=200000 inactive=20s;
    open_file_cache_valid 0;
    open_file_cache_min_uses 2;
    open_file_cache_errors off;

    fastcgi_cache_path /var/cache/nginx/fastcgi_cache levels=1:2 keys_zone=NAME:10m inactive=5m;
    fastcgi_cache_key "$scheme$request_method$host$request_uri";

    client_body_buffer_size 128K;
    client_header_buffer_size 4k;
    client_max_body_size 128m;
    large_client_header_buffers 2 1k;

    #pagespeed on;

    #pagespeed MemcachedServers "127.0.0.1:11211";
    #pagespeed MemcachedThreads 1;
    #pagespeed ModPagespeedMemcachedTimeoutUs 100000;
    #pagespeed UseNativeFetcher on;
    #pagespeed ModPagespeedInPlaceResourceOptimization on;
    #pagespeed CustomFetchHeader Accept-Encoding gzip;

    #pagespeed Domain https://test.org;
    #pagespeed Domain https://wiki.test.org;
    #pagespeed Domain https://min1.test.ing;
    #pagespeed Domain https://img1.test.ing;
    #pagespeed Domain https://img2.test.ing;
    #pagespeed Domain https://bans.test.org;
    #pagespeed Domain https://motd.test.org;
    #pagespeed LoadFromFile https://min1.test.ing /home/test/public_html/xenforo/js/;
    #pagespeed LoadFromFile https://img2.test.ing /home/test/public_html/xenforo/data/;
    #pagespeed LoadFromFile https://img2.test.ing /home/test/public_html/xenforo/styles/;
    #pagespeed LoadFromFile https://bans.test.org /home/test/public_html/sourcebans/;
    #pagespeed LoadFromFileRuleMatch Disallow \.php$;
    #pagespeed LoadFromFileRuleMatch Disallow \.config$;
    #pagespeed LoadFromFileRuleMatch Disallow \.dtd$;
    #pagespeed LoadFromFileRuleMatch Disallow \.xsl$;

    #pagespeed MapOriginDomain http://test.org https://test.org;
    #pagespeed MapOriginDomain http://wiki.test.org https://wiki.test.org;
    #pagespeed MapOriginDomain http://ajax.googleapis.com https://ajax.googleapis.com;
    #pagespeed MapOriginDomain http://img1.test.ing https://img1.test.ing;
    #pagespeed MapOriginDomain http://img2.test.ing https://img2.test.ing;
    #pagespeed MapOriginDomain http://min1.test.ing https://min1.test.ing;
    #pagespeed MapOriginDomain http://bans.test.org https://bans.test.org;
    #pagespeed MapOriginDomain http://motd.test.org https://motd.test.org;

    #pagespeed EnableFilters inline_javascript,inline_images,extend_cache,combine_css,rewrite_css,collapse_whitespace,rewrite_images,rewrite_javascript,combine_heads,combine_javascript,defer_javascript;
    #pagespeed FileCachePath /var/cache/nginx/pagespeed;

    set_real_ip_from 204.93.240.0/24;
    set_real_ip_from 204.93.177.0/24;
    set_real_ip_from 199.27.128.0/21;
    set_real_ip_from 173.245.48.0/20;
    set_real_ip_from 103.22.200.0/22;
    set_real_ip_from 141.101.64.0/18;
    set_real_ip_from 108.162.192.0/18;
    set_real_ip_from 190.93.240.0/20;
    set_real_ip_from 188.114.96.0/20;
    set_real_ip_from 198.41.128.0/17;
    real_ip_header CF-Connecting-IP;

    set_real_ip_from 199.83.128.0/21;
    set_real_ip_from 198.143.32.0/19;
    set_real_ip_from 149.126.72.0/21;
    set_real_ip_from 103.28.248.0/22;
    set_real_ip_from 185.11.124.0/22;
    real_ip_header X-Forwarded-For;

    gzip on;
    gzip_static off;
    gzip_http_version 1.1;
    gzip_min_length 0;
    gzip_vary on;
    gzip_comp_level 9;
    gzip_proxied any;
    gzip_buffers 16 8k;

    gzip_types image/jpeg image/gif image/png image/tiff image/x-icon image/x-ms-bmp text/plain text/css application/x-httpd-php text/php application/x-javascript text/xml application/xml application/rss+xml application/atom+xml application/rdf+xml text/javascript application/xhtml+xml application/vnd.ms-fontobject application/x-font-ttf font/opentype application/x-shockwave-flash application/json;

    geoip_country  /usr/local/share/GeoIP/GeoIP.dat; # the country IP database
    geoip_city     /usr/local/share/GeoIP/GeoLiteCity.dat; # the city IP database

    ssl_session_cache    shared:SSL:10m;
    ssl_session_timeout  10m;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    error_page 404 403 https://test.org/error.php;

    #glitch in firefox: spdy won't send gzip headers, keep that disabled! This is the supposes fix but didn't work for me :(

        #map $spdy $spdy_ae {
        #   default $http_accept_encoding;
        #   2 "gzip, deflate";
        #}

    #include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

[ellisonpatterson]

Any updates on this because I really want to use the module!

[oschaaf]

@xenogamers I tried to reproduce this using ubuntu 12.04 x64 using the ./configure line you provided (plus --add-module=../ngx_pagespeed - which was missing) - but no luck yet.

Does this always happen immediately on the first request?

[ellisonpatterson]

It does not happen until I restart it for the first time after compiling. Also I'm using centos if that matters and get zero issues with a vanilla nginx.

.\

On Jun 23, 2013, at 10:01 AM, Otto van der Schaaf notifications@github.com wrote:

@xenogamers I tried to reproduce this using ubuntu 12.04 x64 using the ./configure line you provided (plus --add-module=../ngx_pagespeed - which was missing) - but no luck yet.

Does this always happen immediately on the first request?

— Reply to this email directly or view it on GitHub.

[oschaaf]

@xenogamers so, just to make sure: when running nginx + ngx_pagespeed for the first time it is running fine. until you restart it after which trouble begins? That does sound like https://github.com/pagespeed/ngx_pagespeed/issues/364 , but you have got memcached disabled right?

[oschaaf]

Testing restarts a few times with gdb I caught this backtrace, which might be it - looking in to it:

Program received signal SIGSEGV, Segmentation fault.
0x000000000041be8e in ngx_clean_old_cycles (ev=0xef79e0) at src/core/ngx_cycle.c:1336
1336                if (cycle[i]->connections[n].fd != (ngx_socket_t) -1) {
(gdb) bt
#0  0x000000000041be8e in ngx_clean_old_cycles (ev=0xef79e0) at src/core/ngx_cycle.c:1336
#1  0x0000000000425d7c in ngx_event_expire_timers () at src/event/ngx_event_timer.c:149
#2  0x0000000000425954 in ngx_process_events_and_timers (cycle=0x1143010) at src/event/ngx_event.c:265
#3  0x000000000042cc88 in ngx_single_process_cycle (cycle=0x1143010) at src/os/unix/ngx_process_cycle.c:315
#4  0x000000000040eaf2 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:409

[ellisonpatterson]

Yes but it happens when enabled as well.

and yes it is fine after compiling but as soon as I restart the socket leaks start happening (i think that's what my error log suggests)

On Jun 24, 2013, at 8:37 AM, Otto van der Schaaf notifications@github.com wrote:

@xenogamers so, just to make sure: when running nginx + ngx_pagespeed for the first time it is running fine. until you restart it after which trouble begins? That does sound like #364 , but you have got memcached disabled right?

— Reply to this email directly or view it on GitHub.

[vbtechsupport]

Found similar error in my logs with Nginx 1.4.2, 1.6.29.3 beta without memcached (disabled) using CentOS 6.4 32bit on 512MB ram OpenVZ based VPS with 1 cpu thread.

Initially got this error with Nginx compiled with PCRE 8.32

2013/07/19 08:50:04 [alert] 29049#0: [ngx_pagespeed 1.6.29.3-3270]         [0719/085004:FATAL:ngx_thread_system.cc(43)] Check failed: may_start_threads_. 
Backtrace:
        nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf() [0x80dbe2d]
        nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf() [0x80bbe63]
        nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf() [0x80d83d0]
        nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf() [0x80ca931]
        nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf() [0x826ac95]
        /lib/libpthread.so.0(+0x6a49) [0xb7738a49]
        /lib/libc.so.6(clone+0x5e) [0xb74cfaee]

Then updated to PCRE 8.33 the only change so configure

nginx version: nginx/1.4.2 built by gcc 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) TLS SNI support enabled configure arguments: --sbin-path=/usr/local/sbin --conf-path=/usr/local/nginx/conf/nginx.conf --add-module=../ngx_pagespeed-release-1.6.29.3-beta --with-http_spdy_module --with-http_ssl_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_sub_module --with-http_addition_module --with-http_secure_link_module --with-http_flv_module --with-http_realip_module --with-libatomic --with-pcre=../pcre-8.33 --with-pcre-jit --with-http_dav_module --add-module=../nginx-dav-ext-module-0.0.2 --add-module=../ngx-fancyindex-ngx-fancyindex --add-module=../ngx_cache_purge-2.1 --add-module=../nginx-accesskey-2.0.3 --with-openssl=../openssl-1.0.1e --add-module=../nginx-http-concat-master --with-openssl-opt=enable-tlsext

Now after PCRE 8.32 to 8.33 update (http://www.pcre.org/changelog.txt), the only error in logs on Nginx restart differs and only logs

2013/07/19 09:04:43 [alert] 29093#0: worker process 29096 exited on signal 11
2013/07/19 09:06:47 [alert] 14715#0: worker process 14718 exited on signal 11
2013/07/19 09:12:34 [alert] 14796#0: worker process 14798 exited on signal 11

Not sure if that helps.

[vnevremeni]

Confirming this. I switched to nginx, all is fine, but ngx_pagespeed is TOTALLY UNUSABLE for now. I tried it with latest versions, 1.4.2 (stable), 1.5.3 (mainline) and also tried to compile all from source (including psol). Always same error. Nginx stops responding after second+ ab or siege test. Let's try: ab -n 1000 -c 10 http://artactivator.com/ for first time, all goes fine. 50 req/s for drupal index.php page but second time, there are timeouts, ab stops working. Ok, trying siege. Req/s slowing slowing down each test, until nginx will give 500 error

In nginx's error log I receive such errors randomly: Check failed: may_startthreads NgxBaseFetch::RequestCollection: Broken pipe

Disabling ngx_pagespeed shows about 50 req/s every stresstest on every page and nginx working fine. But it's better to recompile nginx without pagespeed support, because sometimes it's working strange when off.... I want to use it, but don't want to switch back to apache....

[jeffkaufman]

This doesn't seem to have come up again in almost a year; reopen this if it's still a problem.