Closed Kernal-GH closed 1 year ago
hi @Kernal-GH
could you give more details, I want to know how "crash" mean?
BTW, if you can privide nginx debug log during crash, this can help me resolved this problem ASAP.
delay to reproduce this in windows by meself, wait more details of debug info.
I build windows version with nginx-1.14.2, nginx crashed when i run example for curl,Is it a problem? This is the logs
2019/08/07 17:17:40 [debug] 4576#30852: worker cycle 2019/08/07 17:17:40 [debug] 4576#30852: select event: fd:580 wr:0 2019/08/07 17:17:40 [debug] 4576#30852: select timer: 500 2019/08/07 17:17:41 [debug] 4576#30852: select ready 1 2019/08/07 17:17:41 [debug] 4576#30852: select read 580 2019/08/07 17:17:41 [debug] 4576#30852: post event 012A5D10 2019/08/07 17:17:41 [debug] 4576#30852: timer delta: 342 2019/08/07 17:17:41 [debug] 4576#30852: posted event 012A5D10 2019/08/07 17:17:41 [debug] 4576#30852: delete posted event 012A5D10 2019/08/07 17:17:41 [debug] 4576#30852: accept on 0.0.0.0:3128, ready: 0 2019/08/07 17:17:41 [debug] 4576#30852: malloc: 01258BE8:256 2019/08/07 17:17:41 [debug] 4576#30852: 1 accept: 127.0.0.1:53118 fd:368 2019/08/07 17:17:41 [debug] 4576#30852: 1 event timer add: 368: 60000:1801425531 2019/08/07 17:17:41 [debug] 4576#30852: 1 reusable connection: 1 2019/08/07 17:17:41 [debug] 4576#30852: 1 select add event fd:368 ev:0 2019/08/07 17:17:41 [debug] 4576#30852: worker cycle 2019/08/07 17:17:41 [debug] 4576#30852: select event: fd:580 wr:0 2019/08/07 17:17:41 [debug] 4576#30852: select event: fd:368 wr:0 2019/08/07 17:17:41 [debug] 4576#30852: select timer: 500 2019/08/07 17:17:41 [debug] 4576#30852: select ready 1 2019/08/07 17:17:41 [debug] 4576#30852: select read 368 2019/08/07 17:17:41 [debug] 4576#30852: 1 post event 012A5D60 2019/08/07 17:17:41 [debug] 4576#30852: timer delta: 14 2019/08/07 17:17:41 [debug] 4576#30852: posted event 012A5D60 2019/08/07 17:17:41 [debug] 4576#30852: 1 delete posted event 012A5D60 2019/08/07 17:17:41 [debug] 4576#30852: 1 http wait request handler 2019/08/07 17:17:41 [debug] 4576#30852: 1 malloc: 01278850:1024 2019/08/07 17:17:41 [debug] 4576#30852: 1 WSARecv: fd:368 rc:0 112 of 1024 2019/08/07 17:17:41 [debug] 4576#30852: 1 reusable connection: 0 2019/08/07 17:17:41 [debug] 4576#30852: 1 malloc: 01259DD0:4096 2019/08/07 17:17:41 [debug] 4576#30852: 1 http process request line 2019/08/07 17:17:41 [debug] 4576#30852: *1 http request line: "CONNECT github.com:443 HTTP/1.1" 2019/08/07 17:17:43 [debug] 25272#28324: master WaitForMultipleObjects: 4 2019/08/07 17:17:43 [debug] 25272#28324: reap worker 2019/08/07 17:17:43 [notice] 25272#28324: worker process 4576 exited with code C0000005 2019/08/07 17:17:43 [debug] 25272#28324: GetModuleFileName: "D:\06_OpenSource\httpproxy\nginx-1.14.2\nginx.exe" 2019/08/07 17:17:43 [notice] 25272#28324: start worker process 26280 2019/08/07 17:17:43 [debug] 26280#13352: bind() 0.0.0.0:3128 #204 2019/08/07 17:17:43 [debug] 26280#13352: malloc: 01114800:30 2019/08/07 17:17:43 [debug] 26280#13352: counter: 2EFE0080, 2 2019/08/07 17:17:43 [notice] 26280#13352: nginx/1.14.2 2019/08/07 17:17:43 [info] 26280#13352: OS: 260200 build:9200, "", suite:100, type:1 2019/08/07 17:17:43 [debug] 26280#13352: worker started 2019/08/07 17:17:43 [debug] 25272#28324: WaitForMultipleObjects: 0 2019/08/07 17:17:43 [notice] 26280#13352: create thread 11556 2019/08/07 17:17:43 [debug] 25272#28324: process: 0 26280 00000294 e:0 j:0 2019/08/07 17:17:43 [notice] 26280#13352: create thread 25132 2019/08/07 17:17:43 [notice] 26280#13352: create thread 29840 2019/08/07 17:17:43 [debug] 26280#11556: malloc: 01143E80:8192 2019/08/07 17:17:43 [debug] 26280#11556: malloc: 0116DD78:131072 2019/08/07 17:17:43 [debug] 26280#11556: malloc: 0118DD80:81920 2019/08/07 17:17:43 [debug] 26280#11556: malloc: 011A1D88:81920 2019/08/07 17:17:43 [debug] 26280#25132: cache manager WaitForMultipleObjects: 129 2019/08/07 17:17:43 [debug] 26280#11556: select add event fd:204 ev:0 2019/08/07 17:17:43 [debug] 26280#11556: worker cycle
hi @loory
Not familiar how to debug windows system, could you try some debug tool to get crash stack tracing info ( like linux gdb backtrace)?
Also check your nginx version with right patch.
From https://stackoverflow.com/questions/17168982/exception-error-c0000005-in-vc.
Exception code c0000005 is the code for an access violation.
That means that your program is accessing (either reading or writing)
a memory address to which it does not have rights. Most commonly this is caused by: ...
Only if the memory manager has returned the memory to the system
do you get an access violation.
I build with nginx 1.14.2,and use patch proxy_connect_1014.patch
This is the stack, and in fun ngx_write_fd(ngx_fd_t fd, void *buf, size_t size),the size is 0.
ntdll.dll!NtWriteFile() + 12 bytes Unknown [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll] KERNELBASE.dll!WriteFile() + 88 bytes Unknown nginx.exe!ngx_write_fd(void fd, void buf, unsigned int size) Line 182 C ntdll.dll!NtDeviceIoControlFile() + 12 bytes Unknown KERNELBASE.dll!GetConsoleOutputCP() + 298 bytes Unknown ntdll.dll!NtDeviceIoControlFile() + 12 bytes Unknown ntdll.dll!NtDuplicateObject() + 12 bytes Unknown ntdll.dll!NtWaitForSingleObject() + 12 bytes Unknown ntdll.dll!NtDeviceIoControlFile() + 12 bytes Unknown mswsock.dll!7293c5a2() Unknown mswsock.dll!7293c82f() Unknown nginx.exe!ngx_http_process_request_line(ngx_event_s rev) Line 997 C nginx.exe!ngx_http_wait_request_handler(ngx_event_s rev) Line 510 C nginx.exe!ngx_event_process_posted(ngx_cycle_s cycle, ngx_queue_s posted) Line 33 C nginx.exe!ngx_process_events_and_timers(ngx_cycle_s cycle) Line 259 C nginx.exe!ngx_worker_thread(void data) Line 793 C
hi @loory
It will be close to locate this crash issue.
From your backtrack info, I guess the crash may occurs for some error logging operation( it may call ngx_write_fd to write error message to log file).
Show your source code part of this line ngx_http_process_request_line(ngx_event_s * rev) Line 997 C
?
My source code has something different with yours.
nginx.exe!ngx_write_fd(void * fd, void * buf, unsigned int size) Line 182 C
ntdll.dll!NtDeviceIoControlFile() + 12 bytes Unknown
KERNELBASE.dll!GetConsoleOutputCP() + 298 bytes Unknown
ntdll.dll!NtDeviceIoControlFile() + 12 bytes Unknown
ntdll.dll!NtDuplicateObject() + 12 bytes Unknown
ntdll.dll!NtWaitForSingleObject() + 12 bytes Unknown
ntdll.dll!NtDeviceIoControlFile() + 12 bytes Unknown
mswsock.dll!7293c5a2() Unknown
mswsock.dll!7293c82f() Unknown
nginx.exe!ngx_http_process_request_line(ngx_event_s * rev) Line 997 C
^^^
ngx_http_process_request_line >> windows call >> ngx_write_fd
??
r->request_line.len = r->request_end - r->request_start;
r->request_line.data = r->request_start;
r->request_length = r->header_in->pos - r->request_start;
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
"http request line: \"%V\"", &r->request_line);
r->method_name.len = r->method_end - r->request_start + 1;
r->method_name.data = r->request_line.data;
if (r->http_protocol.data) {
r->http_protocol.len = r->request_end - r->http_protocol.data;
}
if (ngx_http_process_request_uri(r) != NGX_OK) { //Line 997
return;
}
#if (NGX_HTTP_PROXY_CONNECT)
...
#endif
After load more symbols, the stack changed to bellow,and in the function
ngx_event_process_posted
, the param cycle is NULL
ntdll.dll!_NtWriteFile@36() Unknown KERNELBASE.dll!WriteFile() Unknown nginx.exe!ngx_write_fd(void fd, void buf, unsigned int size) Line 184 C ntdll.dll!_NtDeviceIoControlFile@40() Unknown ntdll.dll!_NtDeviceIoControlFile@40() Unknown ntdll.dll!_NtDeviceIoControlFile@40() Unknown nginx.exe!ngx_event_process_posted(ngx_cycle_s cycle, ngx_queue_s posted) Line 33 C kernel32.dll!@BaseThreadInitThunk@12() Unknown
Could you show the buf
string of nginx.exe!ngx_write_fd(void * fd, void * buf, unsigned int size) Line 184 C
and show *r
variable.
Also you can try one-step trace via vscode.
I found the following log_error in ngx_http_process_request_uri.
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
"http uri: \"%V\"", &r->uri);
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
"http args: \"%V\"", &r->args);
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
"http exten: \"%V\"", &r->exten);
Sorry, I'm not familiar with markdown. The stack info is wrong because of visual stdio,I change the debug tool to windbg,the crash stack is
00 035afa1c 00a5158e nginx!ngx_http_process_request_uri+0x1d6 [\ngx_http_request.c @ 1239]
01 035afa3c 00a53721 nginx!ngx_http_process_request_line+0x1ce [\ngx_http_request.c @ 997]
02 035afa54 00a33204 nginx!ngx_http_wait_request_handler+0x231 [ngx_http_request.c @ 510]
03 035afa68 00a32edc nginx!ngx_event_process_posted+0x74 [ngx_event_posted.c @ 33]
04 035afa80 00a3b35b nginx!ngx_process_events_and_timers+0xec [ngx_event.c @ 259]
05 035afa90 77810419 nginx!ngx_worker_thread+0x9b [ngx_process_cycle.c @ 793]
06 035afaa0 77cb662d KERNEL32!BaseThreadInitThunk+0x19
07 035afafc 77cb65fd ntdll!__RtlUserThreadStart+0x2f
08 035afb0c 00000000 ntdll!_RtlUserThreadStart+0x1b
And the crash code in funcation ngx_http_process_request_uri
is
p = r->uri.data + r->uri.len - 1;
while (p > r->uri.data) {
if (*p == ' ') { //line 1239
p--;
continue;
}
if (*p == '.') {
p--;
continue;
}
break;
}
The log_error in ngx_http_process_request_uri can not print out becase is has been carshed upline.
I change the log position to the head of function ngx_http_process_request_uri
,the log is follow
2019/08/08 11:39:51 [debug] 14472#34724: 1 http uri: "" 2019/08/08 11:39:51 [debug] 14472#34724: 1 http args: "" 2019/08/08 11:39:51 [debug] 14472#34724: *1 http exten: ""
Because of the uri is "",and the r->uri.data
is null, so i add some code in ngx_http_process_request_uri
to judge the r->uri.data, it works.
Thank you very much for your replay.
@chobits I build with nginx 1.22.1,and use patch proxy_connect_rewrite_102101.patch in windows with myys2
After the build is successful, use curl https://www.baidu.com -v -x 127.0.0.1:3128
to test the same problem, because r->uri.data is null
.
Using the action of #143 , skipping ngx_http_process_request_uri
works fine when r->method == NGX_HTTP_CONNECT
.
Could you please use this operation to solve the problem.
or just skip the ngx_http_request.c.ngx_http_process_request_uri
part of the code
#if (NGX_WIN32)
{
u_char *p, *last;
if(r->uri.data){
p = r->uri.data;
last = r->uri.data + r->uri.len;
// ...
p = r->uri.data + r->uri.len - 1;
while (p > r->uri.data) {
if (*p == ' ') {
p--;
continue;
}
// ...
}
// ...
}
#endif
ty for ur tips, ill update it in patch
enable NGX_WIN32 macro to trigger sigfault in gdb:
Program received signal SIGSEGV, Segmentation fault.
ngx_http_process_request_uri (r=r@entry=0x56203f79da80) at src/http/ngx_http_request.c:1346 1346 if (*p == ' ') { (gdb) bt #0 ngx_http_process_request_uri (r=r@entry=0x56203f79da80)
at src/http/ngx_http_request.c:1346
#1 0x000056203d7d616f in ngx_http_process_request_line (
rev=rev@entry=0x56203f7e2a40) at src/http/ngx_http_request.c:1103
#2 0x000056203d7d6664 in ngx_http_wait_request_handler (
rev=0x56203f7e2a40) at src/http/ngx_http_request.c:501
#3 0x000056203d7b9b8d in ngx_epoll_process_events (cycle=0x56203f7a2c60,
timer=<optimized out>, flags=<optimized out>)
at src/event/modules/ngx_epoll_module.c:901
(gdb) p p
$17 = (u_char *) 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>
(gdb) f 0
#0 ngx_http_process_request_uri (r=r@entry=0x56203f79da80)
at src/http/ngx_http_request.c:1346
1346 if (*p == ' ') {
(gdb) l
1341
1342 p = r->uri.data + r->uri.len - 1;
1343
1344 while (p > r->uri.data) {
1345
1346 if (*p == ' ') {
This issue has been fixed in https://github.com/chobits/ngx_http_proxy_connect_module/pull/248. Note that only nginx-1.17.x ~ nginx-1.23.2 have been fixed. Their corresponding patch is proxy_connect_rewrite_1018.patch and proxy_connect_rewrite_102101.patch.
thanks,it works.✌️
I am using the HTTPS forward proxy after the patch below Windows will crash, can you support this Windows system?