tengine 2.3.2 limit_req模块问题

yourchanges commented 5 years ago

Ⅰ. Issue Description

生产上正常运行的是2.2.3, limit_req相关配置如下:

in http块:

geo $white_ip {
                  default     0;
                  116.xx.xx.0/24        1;                  
    }
limit_req_zone $binary_remote_addr $request_uri zone=one:10m rate=1r/s;
limit_req_whitelist geo_var_name=white_ip geo_var_value=1;

在server 块(很多站点):

    limit_req_whitelist geo_var_name=white_ip geo_var_value=1;
    limit_req zone=one burst=3;

        location ~* ^/m/app/ {  
                proxy_read_timeout       600s;
            proxy_pass http://web2480;  
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header    Host $host; 
            proxy_set_header    X-Real-IP       $remote_addr;
            proxy_set_header    X-Forwarded-For $remote_addr;
            proxy_cache_key $host$uri$is_args$args$ckey;
            #proxy_cache_valid  200 304  3m; 
            add_header  Cache-status $upstream_cache_status;
            add_header  Cache-status1 $uri$is_args$args$ckey'ajax';

            limit_req_whitelist geo_var_name=white_ip geo_var_value=1;
            limit_req zone=one burst=2;
    }

运行若干年都正常,

升级到最新 2.3.2 , 配置未调整, 上线几分钟后, 线上报大量url 503, error日志也是大量 503. 只能回滚到2.2.3 一切正常.

我看2.3.0 是大升级, limit_req 模块也是有调整, 请问需要做什么调整?

Ⅵ. Environment:

Tengine version (use sbin/nginx -V):

nginx -V
Tengine version: Tengine/2.3.2
nginx version: nginx/1.17.3
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC) 
built with OpenSSL 1.0.2s  28 May 2019
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_gzip_static_module --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.0.2s --with-pcre=../pcre-8.43/ --with-zlib=../zlib-1.2.11 --with-http_lua_module --with-luajit-lib=/usr/local/lib/ --with-luajit-inc=/usr/local/include/luajit-2.0/ --with-lua-inc=/usr/local/include/luajit-2.0/ --with-lua-lib=/usr/local/lib/ --with-ld-opt=-Wl,-rpath, --add-module=modules/ngx_http_concat_module --add-module=modules/ngx_http_upstream_session_sticky_module --add-module=modules/ngx_http_reqstat_module --add-module=modules/ngx_http_upstream_check_module --add-module=modules/ngx_http_trim_filter_module

[root@localhost ~]# /usr/local/nginx/sbin/nginx -m
Tengine version: Tengine/2.3.2
nginx version: nginx/1.17.3
nginx: loaded modules:
nginx:     ngx_core_module (static)
nginx:     ngx_errlog_module (static)
nginx:     ngx_conf_module (static)
nginx:     ngx_openssl_module (static)
nginx:     ngx_regex_module (static)
nginx:     ngx_events_module (static)
nginx:     ngx_event_core_module (static)
nginx:     ngx_epoll_module (static)
nginx:     ngx_procs_module (static)
nginx:     ngx_proc_core_module (static)
nginx:     ngx_http_module (static)
nginx:     ngx_http_core_module (static)
nginx:     ngx_http_log_module (static)
nginx:     ngx_http_upstream_module (static)
nginx:     ngx_http_v2_module (static)
nginx:     ngx_http_static_module (static)
nginx:     ngx_http_gzip_static_module (static)
nginx:     ngx_http_autoindex_module (static)
nginx:     ngx_http_index_module (static)
nginx:     ngx_http_mirror_module (static)
nginx:     ngx_http_try_files_module (static)
nginx:     ngx_http_auth_request_module (static)
nginx:     ngx_http_auth_basic_module (static)
nginx:     ngx_http_access_module (static)
nginx:     ngx_http_limit_conn_module (static)
nginx:     ngx_http_limit_req_module (static)
nginx:     ngx_http_geo_module (static)
nginx:     ngx_http_map_module (static)
nginx:     ngx_http_split_clients_module (static)
nginx:     ngx_http_referer_module (static)
nginx:     ngx_http_rewrite_module (static)
nginx:     ngx_http_ssl_module (static)
nginx:     ngx_http_proxy_module (static)
nginx:     ngx_http_fastcgi_module (static)
nginx:     ngx_http_uwsgi_module (static)
nginx:     ngx_http_scgi_module (static)
nginx:     ngx_http_grpc_module (static)
nginx:     ngx_http_memcached_module (static)
nginx:     ngx_http_empty_gif_module (static)
nginx:     ngx_http_browser_module (static)
nginx:     ngx_http_upstream_hash_module (static)
nginx:     ngx_http_upstream_ip_hash_module (static)
nginx:     ngx_http_upstream_least_conn_module (static)
nginx:     ngx_http_upstream_random_module (static)
nginx:     ngx_http_upstream_keepalive_module (static)
nginx:     ngx_http_upstream_zone_module (static)
nginx:     ngx_http_stub_status_module (static)
nginx:     ngx_http_reqstat_module (static)
nginx:     ngx_http_upstream_check_module (static)
nginx:     ngx_http_write_filter_module (static)
nginx:     ngx_http_header_filter_module (static)
nginx:     ngx_http_chunked_filter_module (static)
nginx:     ngx_http_v2_filter_module (static)
nginx:     ngx_http_range_header_filter_module (static)
nginx:     ngx_http_gzip_filter_module (static)
nginx:     ngx_http_postpone_filter_module (static)
nginx:     ngx_http_ssi_filter_module (static)
nginx:     ngx_http_charset_filter_module (static)
nginx:     ngx_http_userid_filter_module (static)
nginx:     ngx_http_headers_filter_module (static)
nginx:     ngx_http_concat_module (static)
nginx:     ngx_http_upstream_session_sticky_module (static)
nginx:     ngx_http_trim_filter_module (static)
nginx:     ngx_http_lua_module (static)
nginx:     ngx_http_copy_filter_module (static)
nginx:     ngx_http_range_body_filter_module (static)
nginx:     ngx_http_not_modified_filter_module (static)
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

OS (e.g. from /etc/os-release):

[root@linux1 tengine]# cat /etc/redhat-release 
CentOS release 6.10 (Final)

Kernel (e.g. uname -a):

[root@linux1 tengine]# uname -a
Linux linux1 2.6.32-696.6.3.el6.x86_64 #1 SMP Wed Jul 12 14:17:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Others:

yourchanges commented 5 years ago

我看官方2.3.0 release notes 里有:

5. limit_req的请求计数逻辑和官方保持一致，去除limit_req_zone中任何一个变量值为空跳过请求计数的逻辑；

是否是之前是按照远程ip+ url进行计数, 新版现在一下失效, 只基于远程IP 计数了

wangfakang commented 5 years ago

@yourchanges Tengine-2.3.x中的limit_req的逻辑当前和Nginx官方保持一致，即当limit_req_zone中配置的变量（如你上文提供的配置变量：$binary_remote_addr $request_uri）任何一个值为空时，在之前的版本（2.3.0版本之前）中都是不会统计的，而新版本中则会统计的。所以在某个变量值为空的时候相当于统计key的计数增加的就快，即容易触发限流。

yourchanges commented 5 years ago

@yourchanges Tengine-2.3.x中的limit_req的逻辑当前和Nginx官方保持一致，即当limit_req_zone中配置的变量（如你上文提供的配置变量：$binary_remote_addr $request_uri）任何一个值为空时，在之前的版本（2.3.0版本之前）中都是不会统计的，而新版本中则会统计的。所以在某个变量值为空的时候相当于统计key的计数增加的就快，即容易触发限流。

@wangfakang 那请问如何避免这种行为或者调优这个呢? $binary_remote_addr 和$ $request_uri 看起来理论上不会为空啊.

yourchanges commented 5 years ago

@wangfakang 我今天又把http端的limit_req_zone定义调大, 结果和之前一样, 一上去就海量503

zone=one:10m rate=1r/s ===> zone=one:100m rate=100r/s

limit_req_zone $binary_remote_addr $request_uri zone=one:100m rate=100r/s;

yourchanges commented 5 years ago

限制放大100倍还是无效

yourchanges commented 5 years ago

然后, 我对比了2.2.3 和2.3.2 的模块加载情况, 有如下差异:

2.2.3 特有
ngx_dso_module
ngx_http_user_agent_module

ngx_http_upstream_consistent_hash_module 
ngx_http_upstream_dynamic_module

ngx_http_footer_filter_module

2.3.2 特有
ngx_http_grpc_module
ngx_http_upstream_zone_module
ngx_http_mirror_module
ngx_http_try_files_module

wangfakang commented 5 years ago

@yourchanges 你可以先尝试做如下修改（即多个变量之间不要有空格）： limit_req_zone $binary_remote_addr $request_uri zone=one:100m rate=100r/s; 修改为 limit_req_zone $binary_remote_addr$request_uri zone=one:100m rate=100r/s;

yourchanges commented 5 years ago

@wangfakang 重新编译2.3.2 添加了 4个模块

ngx_http_user_agent_module

ngx_http_upstream_consistent_hash_module 
ngx_http_upstream_dynamic_module

ngx_http_footer_filter_module

然后按照你给的建议调整定义配置:

limit_req_zone $binary_remote_addr$request_uri zone=one:30m rate=3r/s;

目前跑了10多分钟, 看起来正常. 后续继续观察 ($binary_remote_addr$request_uri 这种配置只能在2.3.2 上跑, 2.2.3 config test 失败)

那内部如果修复, 麻烦告知下

yourchanges commented 5 years ago

@wangfakang 另外还有一个问题, 就是执行reload 后, 老的work进程一直无法正常退出

[root@linux1 ~]# ps aux | grep nginx
root     12146  0.0  0.0 5320916 44324 ?       Ss   11:49   0:05 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
nginx    12147  2.1  0.3 5575216 208704 ?      S    11:49   2:55 nginx: worker process is shutting down                         
nginx    12148  1.2  0.2 5434720 181940 ?      S    11:49   1:47 nginx: worker process is shutting down                         
nginx    12149  1.7  0.2 5424532 177964 ?      S    11:49   2:25 nginx: worker process is shutting down                         
nginx    12150  1.1  0.2 5447736 171324 ?      S    11:49   1:34 nginx: worker process is shutting down                         

nginx    15495  0.1  0.2 5417056 145056 ?      S    12:09   0:07 nginx: worker process is shutting down                         
nginx    15498  0.0  0.2 5409188 149288 ?      S    12:09   0:06 nginx: worker process is shutting down                         

nginx    15738  3.0  0.2 5440320 172832 ?      S    12:10   3:36 nginx: worker process is shutting down                         
nginx    15739  2.7  0.2 5418800 181428 ?      S    12:10   3:13 nginx: worker process is shutting down                         
nginx    15740  5.1  0.3 5467748 204968 ?      S    12:10   6:06 nginx: worker process is shutting down                         
nginx    15741  4.1  0.3 5434668 196108 ?      S    12:10   4:53 nginx: worker process is shutting down                         
nginx    22097  0.0  0.2 5409088 135744 ?      S    12:49   0:02 nginx: worker process is shutting down                         
nginx    22099  0.0  0.2 5410100 150064 ?      S    12:49   0:04 nginx: worker process is shutting down                         
nginx    22218  0.1  0.2 5409168 148552 ?      S    12:50   0:04 nginx: worker process is shutting down                         
nginx    22221  0.0  0.2 5409168 151164 ?      S    12:50   0:04 nginx: worker process is shutting down                         
nginx    22363  0.0  0.2 5409964 138972 ?      S    12:50   0:00 nginx: worker process is shutting down                         
nginx    22364  0.0  0.2 5409964 139292 ?      S    12:50   0:00 nginx: worker process is shutting down                         
nginx    22441  0.0  0.2 5408976 141132 ?      S    12:50   0:04 nginx: worker process is shutting down                         
nginx    22442  0.0  0.2 5409104 142932 ?      S    12:50   0:04 nginx: worker process is shutting down                         
nginx    22553  0.0  0.2 5409088 136668 ?      S    12:51   0:00 nginx: worker process is shutting down                         
nginx    22554  0.0  0.2 5409964 139548 ?      S    12:51   0:01 nginx: worker process is shutting down                         
nginx    22555  0.0  0.2 5409964 144704 ?      S    12:51   0:01 nginx: worker process is shutting down                         
nginx    22605  0.0  0.2 5409964 143856 ?      S    12:51   0:01 nginx: worker process is shutting down                         
nginx    22606  0.0  0.2 5409072 143360 ?      S    12:51   0:04 nginx: worker process is shutting down                         
nginx    22675  7.6  0.3 5434868 187896 ?      S    12:51   5:52 nginx: worker process                                          
nginx    22676  8.2  0.3 5444992 196300 ?      S    12:51   6:19 nginx: worker process                                          
nginx    22677 11.3  0.3 5472176 205432 ?      R    12:51   8:43 nginx: worker process                                          
nginx    22678 14.5  0.3 5474604 204476 ?      S    12:51  11:14 nginx: worker process                                          
nginx    22679  0.0  0.1 5320788 80468 ?       S    12:51   0:02 nginx: cache manager process

之前 2.2.3 是通过配置最顶层添加:

force_exit 60s;

升级到2.3.2 这个就无法用了, 有什么方法绕过, 或者修复方法么?

wangfakang commented 5 years ago

@yourchanges 详细参考文档: http://tengine.taobao.org/document/core.html

yourchanges commented 5 years ago

@wangfakang 刚看到, 谢谢

wangfakang commented 5 years ago

@yourchanges 这个问题在这个PR中已经修复了 https://github.com/alibaba/tengine/pull/1364 ，你可以本地把修复的代码同步下，然后编译，使用之前的配置（多个变量之间可以支持空格）就可以兼容的。

yourchanges commented 5 years ago

@wangfakang 好非常感谢, 稍后我本地合并了试试

webnginx886 commented 5 years ago

否是之前是按照远程ip+ url进行计数, 新版现在一下失效, 只基于远程IP 计数了

我现在Tengine/2.3.1，发现远程ip+ url进行计数，没有生效，只是对 IP限流

wangfakang commented 5 years ago

@webnginx886 新版本（Tengine-2.3.x）中需要多个变量之间不能有空格，正如上面讨论：

另外这个问题已经在 #1364 修复了，你可以本地同步一下这个修复代码或者使用上述方法（去掉变量之间的空格）进行规避。

yourchanges commented 5 years ago

@wangfakang 运行了十多天, 突然整个进程都崩溃了, dmesg 有大量错误日志:

__ratelimit: 376 callbacks suppressed
nginx[45589]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[45591]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[45590]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[45595]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[45596]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[45597]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[45601]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[45602]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[45603]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[45607]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 340 callbacks suppressed
nginx[46301]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[46303]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[46304]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[46305]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[46309]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[46311]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[46312]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[46314]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[46317]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[46319]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 429 callbacks suppressed
nginx[47189]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[47191]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[47192]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[47194]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[47197]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[47199]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[47200]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[47202]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[47205]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[47207]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 429 callbacks suppressed
nginx[48078]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48081]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48083]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48084]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48086]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48089]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48091]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48092]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48094]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48097]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 374 callbacks suppressed
nginx[48857]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48862]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48860]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48863]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48866]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48868]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48871]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48872]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48873]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[48876]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 260 callbacks suppressed
nginx[569]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[573]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[575]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[576]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[577]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[581]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[582]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[584]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[586]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[589]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 432 callbacks suppressed
nginx[1499]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[1501]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[1504]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[1505]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[1507]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[1509]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[1512]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[1513]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[1515]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[1518]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 430 callbacks suppressed
nginx[2402]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[2403]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[2404]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[2405]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[2410]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[2411]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[2412]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[2413]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[2418]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[2419]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 371 callbacks suppressed
nginx[3195]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3196]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3198]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3200]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3202]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3204]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3206]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3207]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3210]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3212]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 170 callbacks suppressed
nginx[3569]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3570]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3572]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3574]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3577]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3578]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3580]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3583]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3585]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[3586]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 428 callbacks suppressed
nginx[4457]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[4458]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[4459]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[4460]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[4465]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[4466]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[4467]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[4468]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[4472]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[4473]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 430 callbacks suppressed
nginx[5360]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[5362]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[5364]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[5367]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[5369]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[5370]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[5372]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[5375]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[5376]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[5378]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 372 callbacks suppressed
nginx[6146]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6148]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6150]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6151]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6155]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6156]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6159]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6160]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6162]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6164]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 334 callbacks suppressed
nginx[6847]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6850]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6851]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6853]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6856]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6857]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6858]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6862]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6864]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[6865]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 433 callbacks suppressed
nginx[7850]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[7852]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[7855]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[7857]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[7858]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[7860]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[7863]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[7865]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[7866]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[7868]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 435 callbacks suppressed
nginx[8761]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[8763]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[8764]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[8766]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[8769]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[8771]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[8772]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[8776]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[8779]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[8781]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 367 callbacks suppressed
nginx[9526]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[9530]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[9531]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[9532]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[9537]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[9539]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[9541]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[9543]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[9545]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[9546]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 254 callbacks suppressed
nginx[10084]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[10085]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[10087]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[10092]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[10094]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[10096]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[10097]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[10099]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[10102]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[10103]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 435 callbacks suppressed
nginx[10998]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11001]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11003]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11004]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11006]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11009]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11011]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11012]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11014]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11016]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 432 callbacks suppressed
nginx[11900]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11901]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11906]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4
nginx[11908]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000] in nginx[400000+398000]

nginx[11909]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11910]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11915]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11916]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11917]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
nginx[11920]: segfault at 10 ip 00000000004e67b0 sp 00007ffda66e6c68 error 4 in nginx[400000+398000]
__ratelimit: 260 callbacks suppressed
nginx[40710] general protection ip:49f28e sp:7ffe8b7cf140 error:0 in nginx[400000+398000]
nginx[44771]: segfault at 58 ip 000000000049f28e sp 00007ffe8b7cf170 error 4 in nginx[400000+398000]

yourchanges commented 5 years ago

Sep 25 16:45:58 linux1 abrt[11939]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11942]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11944]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11946]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11947]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11949]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11952]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11954]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11956]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11957]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11959]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11962]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11964]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11965]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11968]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11970]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11972]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11973]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11976]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11978]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11979]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11981]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11984]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11986]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11987]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11989]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11992]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11994]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11995]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[11997]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12000]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12002]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12003]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12006]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12009]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12008]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12011]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12014]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12016]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12018]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12019]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12021]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12025]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12026]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12027]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12030]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12033]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12035]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12036]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12039]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12041]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12043]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12044]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12046]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12049]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12051]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12053]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12054]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12057]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12059]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12061]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12062]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12064]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12067]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12069]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:58 linux1 abrt[12070]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12072]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12075]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12078]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12079]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12081]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12085]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12087]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12088]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12090]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12094]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12095]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12096]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12098]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12101]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12103]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12104]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12106]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12109]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12111]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12112]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12114]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12117]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12119]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12121]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12122]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12124]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12127]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12129]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12130]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12132]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12135]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12137]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12138]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12141]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12143]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12145]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12146]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12148]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12151]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12153]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12154]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12156]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12159]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12161]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12162]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12164]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12167]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12169]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12170]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12172]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12175]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12177]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12178]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12181]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12183]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12184]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12186]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12189]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12191]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12193]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12194]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12197]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12199]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12200]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12203]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12205]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12206]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12209]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12211]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12212]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12214]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12217]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12219]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12220]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12222]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12225]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12227]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12229]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12230]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12232]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12235]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12237]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12239]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12240]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12242]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12245]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12247]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:45:59 linux1 abrt[12248]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12253]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12255]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12258]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12260]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12262]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12264]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12266]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12268]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12270]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12272]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12274]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12276]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12278]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12280]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12282]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12284]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12286]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12288]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12291]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12292]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12294]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12296]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12298]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12300]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12302]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12304]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12307]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12308]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12310]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12314]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12312]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12317]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12318]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12319]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12323]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12324]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12326]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12328]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12330]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12332]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12334]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12336]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12338]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12340]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12342]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12344]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12346]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12348]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12350]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12353]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12355]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12356]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12357]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12361]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12362]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12364]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12366]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12368]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12370]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12372]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12376]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12375]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:00 linux1 abrt[12379]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12380]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12382]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12384]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12388]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12390]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12392]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12395]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12396]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12397]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12401]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12402]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12404]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12406]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12410]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12408]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12413]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12415]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12416]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12418]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12421]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12424]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12426]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12430]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12428]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12432]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12434]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12436]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12438]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12441]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12442]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12443]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12447]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12448]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12451]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:01 linux1 abrt[12461]: Not saving repeating crash in '/usr/local/nginx/sbin/nginx'
Sep 25 16:46:07 linux1 abrt[12250]: Saved core dump of pid 12246 (/usr/local/nginx/sbin/nginx) to /var/spool/abrt/ccpp-2019-09-25-16:46:00-12246 (1048576000 bytes)
Sep 25 16:46:07 linux1 abrtd: Directory 'ccpp-2019-09-25-16:46:00-12246' creation detected
Sep 25 16:46:07 linux1 abrtd: Executable '/usr/local/nginx/sbin/nginx' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 25 16:46:07 linux1 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2019-09-25-16:46:00-12246' exited with 1
Sep 25 16:46:07 linux1 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2019-09-25-16:46:00-12246'
Sep 25 22:37:14 linux1 kernel: __ratelimit: 260 callbacks suppressed
Sep 25 22:37:14 linux1 kernel: nginx[40710] general protection ip:49f28e sp:7ffe8b7cf140 error:0 in nginx[400000+398000]
Sep 25 22:37:22 linux1 abrt[22133]: Saved core dump of pid 40710 (/usr/local/nginx/sbin/nginx) to /var/spool/abrt/ccpp-2019-09-25-22:37:14-40710 (1048576000 bytes)
Sep 25 22:37:22 linux1 abrtd: Directory 'ccpp-2019-09-25-22:37:14-40710' creation detected
Sep 25 22:37:22 linux1 abrtd: Executable '/usr/local/nginx/sbin/nginx' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 25 22:37:22 linux1 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2019-09-25-22:37:14-40710' exited with 1
Sep 25 22:37:22 linux1 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2019-09-25-22:37:14-40710'
Sep 26 16:17:08 linux1 kernel: nginx[44771]: segfault at 58 ip 000000000049f28e sp 00007ffe8b7cf170 error 4 in nginx[400000+398000]
Sep 26 16:17:16 linux1 abrt[1724]: Saved core dump of pid 44771 (/usr/local/nginx/sbin/nginx) to /var/spool/abrt/ccpp-2019-09-26-16:17:08-44771 (1048576000 bytes)
Sep 26 16:17:16 linux1 abrtd: Directory 'ccpp-2019-09-26-16:17:08-44771' creation detected
Sep 26 16:17:16 linux1 abrtd: Executable '/usr/local/nginx/sbin/nginx' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 26 16:17:16 linux1 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2019-09-26-16:17:08-44771' exited with 1
Sep 26 16:17:16 linux1 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2019-09-26-16:17:08-44771'

yourchanges commented 5 years ago

有问题是应该是这个:

__ratelimit: xxx callbacks suppressed

wangfakang commented 5 years ago

@yourchanges 看看是否可以把coredump文件打印出来，然后具体gdb看看是那个地方有问题。你可以在该机器上执行如下命令看看：

sudo gdb /usr/local/nginx/sbin/nginx 
然后在gdb窗口里执行如下命令
disassemble 0x00000000004e67b0

yourchanges commented 5 years ago

抓到了:

[root@linux1 coredump]# gdb /usr/local/nginx/sbin/nginx core.31835 
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/nginx/sbin/nginx...done.
BFD: Warning: /home/coredump/core.31835 is truncated: expected core file size >= 5686394880, found: 1048576000.
[New Thread 31835]

warning: Error reading shared library list entry at 0x7efbf850d658

warning: Error reading shared library list entry at 0x7efbf850d658

warning: Error reading shared library list entry at 0x7efbf850d658
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2

warning: Error reading shared library list entry at 0x7efbf850d658
Failed to read a valid object file image from memory.
Core was generated by `nginx: worker process is shutting down                         '.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000004a6517 in ngx_http_upstream_handler (ev=0x7efab3645270) at src/http/ngx_http_upstream.c:1337
1337        ngx_http_set_log_request(c->log, r);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.212.el6_10.3.x86_64
(gdb)

yourchanges commented 5 years ago

(gdb) disassemble 0x00000000004a6517
Dump of assembler code for function ngx_http_upstream_handler:
   0x00000000004a64e9 <+0>: mov    %rbx,-0x20(%rsp)
   0x00000000004a64ee <+5>: mov    %rbp,-0x18(%rsp)
   0x00000000004a64f3 <+10>:    mov    %r12,-0x10(%rsp)
   0x00000000004a64f8 <+15>:    mov    %r13,-0x8(%rsp)
   0x00000000004a64fd <+20>:    sub    $0x28,%rsp
   0x00000000004a6501 <+24>:    mov    %rdi,%rbx
   0x00000000004a6504 <+27>:    mov    (%rdi),%rax
   0x00000000004a6507 <+30>:    mov    (%rax),%rbp
   0x00000000004a650a <+33>:    mov    0x48(%rbp),%r13
   0x00000000004a650e <+37>:    mov    0x8(%rbp),%r12
   0x00000000004a6512 <+41>:    mov    0x58(%r12),%rax
=> 0x00000000004a6517 <+46>:    mov    0x28(%rax),%rax
   0x00000000004a651b <+50>:    mov    %rbp,0x10(%rax)
   0x00000000004a651f <+54>:    mov    0x58(%r12),%rsi
   0x00000000004a6524 <+59>:    mov    (%rsi),%rax
   0x00000000004a6527 <+62>:    test   $0x1,%ah
   0x00000000004a652a <+65>:    je     0x4a6553 <ngx_http_upstream_handler+106>
   0x00000000004a652c <+67>:    lea    0x3f0(%rbp),%r9
   0x00000000004a6533 <+74>:    lea    0x3e0(%rbp),%r8
   0x00000000004a653a <+81>:    mov    $0x6cbfd8,%ecx
   0x00000000004a653f <+86>:    mov    $0x0,%edx
   0x00000000004a6544 <+91>:    mov    $0x8,%edi
   0x00000000004a6549 <+96>:    mov    $0x0,%eax
   0x00000000004a654e <+101>:   callq  0x44f108 <ngx_log_error_core>
   0x00000000004a6553 <+106>:   movzbl 0x9(%rbx),%eax
   0x00000000004a6557 <+110>:   and    $0x14,%eax
   0x00000000004a655a <+113>:   cmp    $0x14,%al
   0x00000000004a655c <+115>:   jne    0x4a6562 <ngx_http_upstream_handler+121>
   0x00000000004a655e <+117>:   andb   $0xeb,0x9(%rbx)
   0x00000000004a6562 <+121>:   testb  $0x1,0x8(%rbx)
   0x00000000004a6566 <+125>:   je     0x4a6574 <ngx_http_upstream_handler+139>
   0x00000000004a6568 <+127>:   mov    %r13,%rsi
   0x00000000004a656b <+130>:   mov    %rbp,%rdi
   0x00000000004a656e <+133>:   callq  *0x8(%r13)
   0x00000000004a6572 <+137>:   jmp    0x4a657e <ngx_http_upstream_handler+149>
   0x00000000004a6574 <+139>:   mov    %r13,%rsi
   0x00000000004a6577 <+142>:   mov    %rbp,%rdi
   0x00000000004a657a <+145>:   callq  *0x0(%r13)
   0x00000000004a657e <+149>:   mov    %r12,%rdi
   0x00000000004a6581 <+152>:   callq  0x490de1 <ngx_http_run_posted_requests>
   0x00000000004a6586 <+157>:   mov    0x8(%rsp),%rbx
   0x00000000004a658b <+162>:   mov    0x10(%rsp),%rbp
   0x00000000004a6590 <+167>:   mov    0x18(%rsp),%r12
   0x00000000004a6595 <+172>:   mov    0x20(%rsp),%r13
   0x00000000004a659a <+177>:   add    $0x28,%rsp
   0x00000000004a659e <+181>:   retq   
End of assembler dump.
(gdb)

yourchanges commented 5 years ago

@wangfakang 麻烦有空看看, 谢谢

yourchanges commented 5 years ago

@wangfakang 有陆续收集了十几个coredump文件, 大部分还是在 src/http/ngx_http_upstream.c:1337 这里, 其他看起来问题也是相关, 具体日志如下:

[root@linux1 coredump]# gdb /usr/local/nginx/sbin/nginx core.26857 
Reading symbols from /usr/local/nginx/sbin/nginx...done.
BFD: Warning: /home/coredump/core.26857 is truncated: expected core file size >= 5534490624, found: 104857600.
[New Thread 26857]

warning: Error reading shared library list entry at 0x7f13aff9a658

warning: Error reading shared library list entry at 0x7f13aff9a658

warning: Error reading shared library list entry at 0x7f13aff9a658
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2

warning: Error reading shared library list entry at 0x7f13aff9a658
Failed to read a valid object file image from memory.
Core was generated by `nginx: worker process                                          '.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000004a657a in ngx_http_upstream_handler (ev=0x7f126afd9bf0) at src/http/ngx_http_upstream.c:1351
1351            u->read_event_handler(r, u);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.212.el6_10.3.x86_64
(gdb) 

[root@linux1 coredump]# gdb /usr/local/nginx/sbin/nginx core.36830 
Reading symbols from /usr/local/nginx/sbin/nginx...done.
BFD: Warning: /home/coredump/core.36830 is truncated: expected core file size >= 5556793344, found: 104857600.
[New Thread 36830]

warning: Error reading shared library list entry at 0x7f13aff9a658

warning: Error reading shared library list entry at 0x7f13aff9a658

warning: Error reading shared library list entry at 0x7f13aff9a658
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2

warning: Error reading shared library list entry at 0x7f13aff9a658
Failed to read a valid object file image from memory.
Core was generated by `nginx: worker process                                          '.
Program terminated with signal 11, Segmentation fault.
#0  ngx_http_upstream_process_upstream (r=0x34303e0, u=0x3c53140) at src/http/ngx_http_upstream.c:4086
4086        rev = c->read;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.212.el6_10.3.x86_64
(gdb) 

[root@linux11 coredump]# gdb /usr/local/nginx/sbin/nginx core.20807 
Reading symbols from /usr/local/nginx/sbin/nginx...done.
BFD: Warning: /home/coredump/core.20807 is truncated: expected core file size >= 9782812672, found: 104857600.
[New LWP 20807]
Cannot access memory at address 0x7f5dcd38f128
Cannot access memory at address 0x7f5dcd38f120
Failed to read a valid object file image from memory.
Core was generated by `nginx: worker process                                          '.
Program terminated with signal 11, Segmentation fault.
#0  ngx_http_upstream_process_upstream (r=0x2ffbbe0, 
    r@entry=<error reading variable: Cannot access memory at address 0x7ffe1dc0c788>, u=0x1b47f08, 
    u@entry=<error reading variable: Cannot access memory at address 0x7ffe1dc0c788>) at src/http/ngx_http_upstream.c:4086
4086        rev = c->read;
(gdb)

wangfakang commented 5 years ago

@yourchanges 看看具体的堆栈（使用 bt命令），另外提供下编译参数以及最小可复现的配置（可以看看请求特征什么的 p r 里面会有的）。

yourchanges commented 5 years ago

编译参数

[root@linux1 ~]# /usr/local/nginx/sbin/nginx -V
Tengine version: Tengine/2.3.2
nginx version: nginx/1.17.3
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC) 
built with OpenSSL 1.0.2s  28 May 2019
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_gzip_static_module --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.0.2s --with-pcre=../pcre-8.43/ --with-zlib=../zlib-1.2.11 --with-http_lua_module --with-luajit-lib=/usr/local/lib/ --with-luajit-inc=/usr/local/include/luajit-2.1/ --with-lua-inc=/usr/local/include/luajit-2.1/ --with-lua-lib=/usr/local/lib/ --with-ld-opt=-Wl,-rpath, --add-module=modules/ngx_http_concat_module --add-module=modules/ngx_http_upstream_session_sticky_module --add-module=modules/ngx_http_reqstat_module --add-module=modules/ngx_http_upstream_check_module --add-module=modules/ngx_http_trim_filter_module --add-module=modules/ngx_http_footer_filter_module --add-module=modules/ngx_http_upstream_consistent_hash_module --add-module=modules/ngx_http_upstream_dynamic_module --add-module=modules/ngx_http_user_agent_module --add-module=modules/ngx_http_upstream_dyups_module --add-module=modules/ngx_http_upstream_vnswrr_module --with-debug

yourchanges commented 5 years ago

里面配置站点太多, 不知道是那个导致的, 具体我在看看堆栈是哪个域名下崩的(能看到么)

yourchanges commented 5 years ago

bt也获取不到更多信息

Reading symbols from /usr/local/nginx/sbin/nginx...done.
BFD: Warning: /home/coredump/core.14018 is truncated: expected core file size >= 5688664064, found: 104857600.
[New Thread 14018]
Cannot access memory at address 0x3e4d422168
Cannot access memory at address 0x3e4d422168
Cannot access memory at address 0x3e4d422168
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Failed to read a valid object file image from memory.
Core was generated by `nginx: worker process                                          '.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000004a6517 in ngx_http_upstream_handler (ev=0x7f126aff3750) at src/http/ngx_http_upstream.c:1337
1337        ngx_http_set_log_request(c->log, r);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.212.el6_10.3.x86_64
(gdb) bt
#0  0x00000000004a6517 in ngx_http_upstream_handler (ev=0x7f126aff3750) at src/http/ngx_http_upstream.c:1337
Cannot access memory at address 0x7ffd6620d028
(gdb)

yourchanges commented 5 years ago

https://github.com/alibaba/tengine/issues/1363#issuecomment-535818182 这个不是调用栈么?

wangfakang commented 5 years ago

打印的core file比较大，被截断了，所以你执行bt命令的时候没有看到调用栈。你可以通过在Tengine配置里面加上 worker_rlimit_core 100000000000; (注意是控制打印core file的大小的，为避免磁盘问题，建议在单台测试机器上配置)。另外r变量里面存储了该请求的几乎全部特征，比如r->uri 查看请求uri ，r->headers_in.host.value.data 查看请求域名等信息。

yourchanges commented 5 years ago

抓到了, 感觉是一个攻击请求一样, url 特别长, 45MB? 然后host 看不到:

Reading symbols from /usr/local/nginx/sbin/nginx...done.
[New Thread 43170]
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/local/lib/libluajit-5.1.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libluajit-5.1.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `nginx: worker process                                          '.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000004a6517 in ngx_http_upstream_handler (ev=0x7fa18cef9410) at src/http/ngx_http_upstream.c:1337
1337        ngx_http_set_log_request(c->log, r);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.212.el6_10.3.x86_64 libgcc-4.4.7-23.el6.x86_64 nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64
(gdb) bt
#0  0x00000000004a6517 in ngx_http_upstream_handler (ev=0x7fa18cef9410) at src/http/ngx_http_upstream.c:1337
#1  0x000000000047bb03 in ngx_epoll_process_events (cycle=0x4119fe0, timer=<value optimized out>, flags=<value optimized out>) at src/event/modules/ngx_epoll_module.c:973
#2  0x000000000046f50b in ngx_process_events_and_timers (cycle=0x4119fe0) at src/event/ngx_event.c:255
#3  0x0000000000479a7b in ngx_worker_process_cycle (cycle=0x4119fe0, data=<value optimized out>) at src/os/unix/ngx_process_cycle.c:811
#4  0x0000000000475eb0 in ngx_spawn_process (cycle=0x4119fe0, proc=0x4799e8 <ngx_worker_process_cycle>, data=0x0, name=0x6c4d3c "worker process", respawn=-4)
    at src/os/unix/ngx_process.c:199
#5  0x0000000000478df2 in ngx_start_worker_processes (cycle=0x4119fe0, n=4, type=-4) at src/os/unix/ngx_process_cycle.c:395
#6  0x000000000047a8c2 in ngx_master_process_cycle (cycle=0x4119fe0) at src/os/unix/ngx_process_cycle.c:268
#7  0x000000000044e6f9 in main (argc=<value optimized out>, argv=<value optimized out>) at src/core/nginx.c:418
(gdb) p r

(gdb) p r->headers_in.host
$4 = (ngx_table_elt_t *) 0x0
(gdb) p r->uri
$5 = {len = 45939304, data = 0x2bd03a8 "\016`\324.\016\215\066$\f"}
(gdb) p r->args
$6 = {len = 4, data = 0x0}
(gdb) p r->request_line
$7 = {len = 0, data = 0x3d46530 "@\201\374\002"}

(gdb) p r->headers_in
$12 = {headers = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, host = 0x0, connection = 0x0, if_modified_since = 0x0, 
  if_unmodified_since = 0x0, if_match = 0xffffffffffffffff, if_none_match = 0xc8000000c2, user_agent = 0x0, referer = 0x5ee57e0, content_length = 0x0, content_range = 0x203033202c646557, 
  content_type = 0x393130322074634f, range = 0x343a30333a363020, if_range = 0x616d00544d472034, transfer_encoding = 0x35323d6567612d78, te = 0x3030303239, expect = 0x9c63a0, 
  upgrade = 0x2e77777700000000, accept_encoding = 0x2f6d6f632e696a39, via = 0x3933312f74736964, authorization = 0x313264653932322e, keep_alive = 0x6361666362383736, x_forwarded_for = {
    elts = 0x736a2e6563326330, nelts = 8314605285628212007, size = 69694067, nalloc = 4459745558579569649, pool = 0xe2fab2802564482c}, accept = 0xffffffffffffffff, 
  accept_language = 0xffffffffffffffff, user = {len = 18446744073709551615, data = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>}, passwd = {len = 18446744073709551615, 
    data = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>}, cookies = {elts = 0xffffffffffffffff, nelts = 18446744073709551615, size = 0, nalloc = 0, pool = 0x0}, server = {
    len = 0, data = 0x0}, content_length_n = 0, keep_alive_n = 0, connection_type = 0, chunked = 0, msie = 0, msie6 = 0, opera = 0, gecko = 0, chrome = 0, safari = 0, konqueror = 0}
(gdb) p r->headers_in.cookies
$13 = {elts = 0xffffffffffffffff, nelts = 18446744073709551615, size = 0, nalloc = 0, pool = 0x0}
(gdb) p r->headers_in.host
$14 = (ngx_table_elt_t *) 0x0
(gdb)

yourchanges commented 5 years ago

@wangfakang

yourchanges commented 5 years ago

@wangfakang 有什么思路么?

wangfakang commented 5 years ago

@yourchanges 从你发的消息看起来是有踩内存发生，你可以使用valgrind工具进行检测下具体的踩内存起始位置。valgrind使用方法可以google搜索下，比如： valgrind --trace-children=yes --log-file=/${check_log_file_path}/check.log --tool=memcheck --leak-check=full /${nginx_bain} -c /${nginx_conf} -p /${nginx_work_path}

yourchanges commented 5 years ago

@wangfakang 换用jemalloc 会不会可以避免, 你们推荐用哪个版本的jemalloc, 3.6.0还是4.5.0? 我们目前没有使用jemalloc

wangfakang commented 5 years ago

@yourchanges jemalloc是可以用来避免ptmalloc的伪内存泄漏，但是没法避免踩内存问题的。我们目前使用的是jemalloc-4.0.x版本,你可以尝试使用新版本的。

alibaba / tengine