search

Sysinternals / ProcDump-for-Linux

A Linux version of the ProcDump Sysinternals tool
MIT License
2.95k stars 306 forks source link

Segmentation fault when taking proc dump #53

Closed gnuhpc closed 5 years ago

gnuhpc commented 5 years ago

Expected behavior

Get the proc dump snapshot

Actual behavior

gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ sudo ./procdump -p 3151

ProcDump v1.0.1 - Sysinternals process dump utility Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license. Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi Monitors a process and writes a dump file when the process exceeds the specified criteria.

[13:58:02 - ERROR]: Error getting process name. Process: (null) (3151) CPU Threshold: n/a Commit Threshold: n/a Threshold Seconds: 10 Number of Dumps: 1

Press Ctrl-C to end monitoring without terminating the process.

Segmentation fault

Steps to reproduce the behavior

  1. Start a redis server (empty for test purpose only) and obtain the pid of the server and redis-cli client:

gnuhpc@gnuhpc-pc:~/app/redis-5.0.4/src$ ps -ef |grep redis gnuhpc 2947 2839 0 13:54 pts/11 00:00:00 redis-cli gnuhpc 3151 2714 0 13:57 pts/10 00:00:00 ./redis-server 127.0.0.1:6379

  1. proc dump for the server and client

gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ sudo ./procdump -p 3151

ProcDump v1.0.1 - Sysinternals process dump utility Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license. Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi Monitors a process and writes a dump file when the process exceeds the specified criteria.

[13:58:02 - ERROR]: Error getting process name. Process: (null) (3151) CPU Threshold: n/a Commit Threshold: n/a Threshold Seconds: 10 Number of Dumps: 1

Press Ctrl-C to end monitoring without terminating the process.

Segmentation fault gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ sudo ./procdump -p 2947

ProcDump v1.0.1 - Sysinternals process dump utility Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license. Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi Monitors a process and writes a dump file when the process exceeds the specified criteria.

Process: redis-cli (2947) CPU Threshold: n/a Commit Threshold: n/a Threshold Seconds: 10 Number of Dumps: 1

Press Ctrl-C to end monitoring without terminating the process.

14:01:12 - INFO: Core dump 1 generated: redis_cli_time_2019-04-25_14:01:12.2947

System information (e.g., distro, kernel version, etc.)

Linux distro:

gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ cat /etc/release DISTRIB_ID=LinuxMint DISTRIB_RELEASE=19 DISTRIB_CODENAME=tara DISTRIB_DESCRIPTION="Linux Mint 19 Tara" NAME="Linux Mint" VERSION="19 (Tara)" ID=linuxmint ID_LIKE=ubuntu PRETTY_NAME="Linux Mint 19" VERSION_ID="19" HOME_URL="https://www.linuxmint.com/" SUPPORT_URL="https://forums.ubuntu.com/" BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/" PRIVACY_POLICY_URL="https://www.linuxmint.com/" VERSION_CODENAME=tara UBUNTU_CODENAME=bionic

ProcDump for Linux: using the master branch from the github and make the source code by myself

Kernel version:

gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ uname -a Linux gnuhpc-pc 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

gnuhpc commented 5 years ago

Try ProcDumping ProcDump.

@JL2210 What do you mean by saying this?

nikkitan commented 5 years ago

Expected behavior Get the proc dump snapshot Actual behavior gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ sudo ./procdump -p 3151 ProcDump v1.0.1 - Sysinternals process dump utility Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license. Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi Monitors a process and writes a dump file when the process exceeds the specified criteria. [13:58:02 - ERROR]: Error getting process name. Process: (null) (3151) CPU Threshold: n/a Commit Threshold: n/a Threshold Seconds: 10 Number of Dumps: 1 Press Ctrl-C to end monitoring without terminating the process. Segmentation fault Steps to reproduce the behavior

Start a redis server (empty for test purpose only) and obtain the pid of the server and redis-cli client:

gnuhpc@gnuhpc-pc:~/app/redis-5.0.4/src$ ps -ef |grep redis gnuhpc 2947 2839 0 13:54 pts/11 00:00:00 redis-cli gnuhpc 3151 2714 0 13:57 pts/10 00:00:00 ./redis-server 127.0.0.1:6379

proc dump for the server and client

gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ sudo ./procdump -p 3151 ProcDump v1.0.1 - Sysinternals process dump utility Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license. Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi Monitors a process and writes a dump file when the process exceeds the specified criteria. [13:58:02 - ERROR]: Error getting process name. Process: (null) (3151) CPU Threshold: n/a Commit Threshold: n/a Threshold Seconds: 10 Number of Dumps: 1 Press Ctrl-C to end monitoring without terminating the process. Segmentation fault gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ sudo ./procdump -p 2947 ProcDump v1.0.1 - Sysinternals process dump utility Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license. Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi Monitors a process and writes a dump file when the process exceeds the specified criteria. Process: redis-cli (2947) CPU Threshold: n/a Commit Threshold: n/a Threshold Seconds: 10 Number of Dumps: 1 Press Ctrl-C to end monitoring without terminating the process. 14:01:12 - INFO: Core dump 1 generated: redis_cli_time_2019-04-25_14:01:12.2947

System information (e.g., distro, kernel version, etc.) Linux distro:

gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ cat /etc/release DISTRIB_ID=LinuxMint DISTRIB_RELEASE=19 DISTRIB_CODENAME=tara DISTRIB_DESCRIPTION="Linux Mint 19 Tara" NAME="Linux Mint" VERSION="19 (Tara)" ID=linuxmint ID_LIKE=ubuntu PRETTY_NAME="Linux Mint 19" VERSION_ID="19" HOME_URL="https://www.linuxmint.com/" SUPPORT_URL="https://forums.ubuntu.com/" BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/" PRIVACY_POLICY_URL="https://www.linuxmint.com/" VERSION_CODENAME=tara UBUNTU_CODENAME=bionic

ProcDump for Linux: using the master branch from the github and make the source code by myself Kernel version:

gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ uname -a Linux gnuhpc-pc 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

@gnuhpc Can you check if you have the file of /proc/{YOUR-REDIS-SERVER-PID}/cmdline when your redis-server is running?

gnuhpc commented 5 years ago

I tried and found the cmd in my redis server pid directory of proc.

I think this issue (maybe not a problem) could be reproduced since the redis server can be easily compiled and started. Could you give it a try in your env?

Thanks!

15 10:13 cwd -> /home/gnuhpc/app/redis-5.0.4/src

nikkitan commented 5 years ago

@gnuhpc Actually I have only been able to reproduce part of your issue. I could reproduce the crash by forcing a null process name and this is easy to fix. But I haven't been able to reproduce the absence of the process name. Can you run procdump again in the same env but with the -d option for diagnostic logs and share the logs here?

gnuhpc commented 5 years ago

I tried but failed to obtain any further logs for this issue.

gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ ps -ef |grep redis-server gnuhpc 27679 10050 0 09:55 pts/0 00:00:00 ./redis-server *:6379 gnuhpc 27880 27367 0 09:57 pts/1 00:00:00 grep --color=auto redis-server

gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ sudo ./procdump -p 27679 -d

ProcDump v1.0.1 - Sysinternals process dump utility Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license. Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi Monitors a process and writes a dump file when the process exceeds the specified criteria.

[09:57:41 - ERROR]: Error getting process name. Process: (null) (27679) CPU Threshold: n/a Commit Threshold: n/a Threshold Seconds: 10 Number of Dumps: 1

Press Ctrl-C to end monitoring without terminating the process.

Segmentation fault gnuhpc@gnuhpc-pc:~/app/ProcDump-for-Linux/bin$ sudo ./procdump -d -p 27679

ProcDump v1.0.1 - Sysinternals process dump utility Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license. Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi Monitors a process and writes a dump file when the process exceeds the specified criteria.

[09:57:46 - ERROR]: Error getting process name. Process: (null) (27679) CPU Threshold: n/a Commit Threshold: n/a Threshold Seconds: 10 Number of Dumps: 1

Press Ctrl-C to end monitoring without terminating the process.

Segmentation fault

nikkitan commented 5 years ago

@gnuhpc Thanks for your effort and collaborations so far. Can you also share any logs in syslog? I’m asking this because the function of GetProcessName() logs few errors in the level of “debug” which are sent to syslog instead of stdout. Also according to the code, if you don’t have such errors in syslog, it is possible you don’t have the “cmdline” file under your redis-server’s folder under the /proc filesystem, and if you do have errors from GetProcessName() in syslog, it will be helpful to get more insights of your env. I’ve been running procdump with redis-server on Ubuntu 1804 but I haven’t had null process name. I installed redis via apt-get. I haven’t tried compiling redis.

gnuhpc commented 5 years ago

It seems that the procdump couldn't obtain the cmdline of the redis server pid.

Aug 21 09:20:02 gnuhpc-pc ProcDump[18814]: [09:20:02 - DEBUG]: Failed to extract process name from /proc/PID/cmdline Aug 21 09:20:02 gnuhpc-pc ProcDump[18814]: [09:20:02 - ERROR]: Error getting process name. Aug 21 09:20:02 gnuhpc-pc ProcDump[18814]: [09:20:02 - DEBUG]: GetOpts and initial Configuration finished in src/ProcDumpConfiguration.c, at line 293 Aug 21 09:20:02 gnuhpc-pc ProcDump[18814]: [09:20:02 - DEBUG]: TimerThread: Starting Trigger Thread in src/TriggerThreadProcs.c, at line 128 Aug 21 09:20:02 gnuhpc-pc ProcDump[18814]: [09:20:02 - INFO]: Timed: Aug 21 09:20:02 gnuhpc-pc kernel: [297268.298775] procdump[18815]: segfault at 0 ip 00007fb5868865a1 sp 00007fb5866f6178 error 4 in libc-2.27.so[7fb5866f8000+1e7000] Aug 21 09:20:35 gnuhpc-pc ProcDump[18958]: [09:20:35 - DEBUG]: Failed to extract process name from /proc/PID/cmdline Aug 21 09:20:35 gnuhpc-pc ProcDump[18958]: [09:20:35 - ERROR]: Error getting process name. Aug 21 09:20:35 gnuhpc-pc ProcDump[18958]: [09:20:35 - DEBUG]: GetOpts and initial Configuration finished in src/ProcDumpConfiguration.c, at line 293 Aug 21 09:20:35 gnuhpc-pc ProcDump[18958]: [09:20:35 - DEBUG]: TimerThread: Starting Trigger Thread in src/TriggerThreadProcs.c, at line 128 Aug 21 09:20:35 gnuhpc-pc ProcDump[18958]: [09:20:35 - INFO]: Timed: Aug 21 09:20:35 gnuhpc-pc kernel: [297300.772416] procdump[18959]: segfault at 0 ip 00007f54496725a1 sp 00007f54494e2178 error 4

However, when I checked the proc directory, the cmdline did exist ( and also could be showed using cat command):

gnuhpc@gnuhpc-pc:/proc/27679$ ls -l total 0 dr-xr-xr-x 2 gnuhpc gnuhpc 0 8月 21 09:22 attr -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 autogroup -r-------- 1 gnuhpc gnuhpc 0 8月 21 09:22 auxv -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 cgroup --w------- 1 gnuhpc gnuhpc 0 8月 21 09:22 clear_refs -r--r--r-- 1 gnuhpc gnuhpc 0 8月 18 09:56 cmdline -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 comm -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 coredump_filter -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 cpuset lrwxrwxrwx 1 gnuhpc gnuhpc 0 8月 21 09:22 cwd -> /home/gnuhpc/app/redis-5.0.4/src -r-------- 1 gnuhpc gnuhpc 0 8月 21 09:22 environ lrwxrwxrwx 1 gnuhpc gnuhpc 0 8月 18 09:55 exe -> /home/gnuhpc/app/redis-5.0.4/src/redis-server dr-x------ 2 gnuhpc gnuhpc 0 8月 21 09:22 fd dr-x------ 2 gnuhpc gnuhpc 0 8月 21 09:22 fdinfo -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 gid_map -r-------- 1 gnuhpc gnuhpc 0 8月 21 09:22 io -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 limits -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 loginuid dr-x------ 2 gnuhpc gnuhpc 0 8月 21 09:22 map_files -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 maps -rw------- 1 gnuhpc gnuhpc 0 8月 21 09:22 mem -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 mountinfo -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 mounts -r-------- 1 gnuhpc gnuhpc 0 8月 21 09:22 mountstats dr-xr-xr-x 6 gnuhpc gnuhpc 0 8月 21 09:22 net dr-x--x--x 2 gnuhpc gnuhpc 0 8月 21 09:22 ns -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 numa_maps -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 oom_adj -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 oom_score -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 oom_score_adj -r-------- 1 gnuhpc gnuhpc 0 8月 21 09:22 pagemap -r-------- 1 gnuhpc gnuhpc 0 8月 21 09:22 patch_state -r-------- 1 gnuhpc gnuhpc 0 8月 21 09:22 personality -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 projid_map lrwxrwxrwx 1 gnuhpc gnuhpc 0 8月 21 09:22 root -> / -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 sched -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 schedstat -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 sessionid -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 setgroups -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 smaps -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 smaps_rollup -r-------- 1 gnuhpc gnuhpc 0 8月 21 09:22 stack -r--r--r-- 1 gnuhpc gnuhpc 0 8月 18 09:55 stat -r--r--r-- 1 gnuhpc gnuhpc 0 8月 18 09:56 statm -r--r--r-- 1 gnuhpc gnuhpc 0 8月 18 09:56 status -r-------- 1 gnuhpc gnuhpc 0 8月 21 09:22 syscall dr-xr-xr-x 6 gnuhpc gnuhpc 0 8月 21 09:22 task -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 timers -rw-rw-rw- 1 gnuhpc gnuhpc 0 8月 21 09:22 timerslack_ns -rw-r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 uid_map -r--r--r-- 1 gnuhpc gnuhpc 0 8月 21 09:22 wchan gnuhpc@gnuhpc-pc:/proc/27679$ cat cmdline ./redis-server *:6379gnuhpc@gnuhpc-pc:/proc/27679$

would you try to use the compiled redis-server to reproduce the issue, pls?

nikkitan commented 5 years ago

@gnuhpc The new logs from you actually help a lot. Can you please do a "hexdump -C cmdline" and share the outputs?

gnuhpc commented 5 years ago

gnuhpc@gnuhpc-pc:/proc/27679$ hexdump -C cmdline 00000000 2e 2f 72 65 64 69 73 2d 73 65 72 76 65 72 20 2a |./redis-server *| 00000010 3a 36 33 37 39 |:6379| 0000001

nikkitan commented 5 years ago

@gnuhpc Frome the hexdump output, the content in your cmdline is not null-terminated. I'm working on this to make sure ProcDump always append a null-terminator to the fetched content from cmdline, and once I finish my work I'll ask you to try my branch in your env. Another thing is that I compiled and tried the version 4.0 of redis-server, and I didn't get the null process name. I didn't try 5.0 because it doesn't appear to be stable since it didn't pass all tests from make test.

gnuhpc commented 5 years ago

@nikkitan I haven't given redis 4.x a try due to the fact that the version 5.x is the latest stable version of Redis according to the page: https://redis.io/download. Actually, I skipped the make test before starting the redis server.

Anyway, keep up your good work and also keep in touch if needed.

nikkitan commented 5 years ago

@gnuhpc Please try this branch in your env and keep us updated here: https://github.com/nikkitan/ProcDump-for-Linux/tree/nullify_cmdline_buffer

gnuhpc commented 5 years ago

You did fix the null name isue while dump triggered. Please check the dump file at :https://wetransfer.com/downloads/8338655d5d6ccd54941ad5c98e77aa1820190822143756/3d3290dcda15342e5808df1ca3ba69eb20190822143756/75167d

gnuhpc@gnuhpc-pc:~/app/ProcDump-null/ProcDump-for-Linux/bin$ sudo ./procdump -p 27679

ProcDump v1.0.1 - Sysinternals process dump utility Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license. Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi Monitors a process and writes a dump file when the process exceeds the specified criteria.

Process: redis-server *:6379 (27679) CPU Threshold: n/a Commit Threshold: n/a Threshold Seconds: 10 Number of Dumps: 1

Press Ctrl-C to end monitoring without terminating the process.

[22:35:34 - INFO]: Core dump 1 generated: redis_server_6379_time_2019-08-22_22:35:33.27679 gnuhpc@gnuhpc-pc:~/app/ProcDump-null/ProcDump-for-Linux/bin$ ls -alrt total 35740 drwxrwxr-x 10 gnuhpc gnuhpc 4096 8月 22 22:35 .. -rwxrwxr-x 1 gnuhpc gnuhpc 107232 8月 22 22:35 procdump -rwxrwxr-x 1 gnuhpc gnuhpc 11144 8月 22 22:35 ProcDumpTestApplication drwxrwxr-x 2 gnuhpc gnuhpc 4096 8月 22 22:35 . -rw-r--r-- 1 root root 36463368 8月 22 22:35 redisserver6379_time_2019-08-22_22:35:33.27679

nikkitan commented 5 years ago

@gnuhpc Thanks for the confirmation and I'm glad I can help. I will integrate the fix into master and we will release it. Please don't forget to close this issue whenever you are fully unblocked from this issue by the new release.

gnuhpc commented 5 years ago

Thanks for your tracking! You could close this issue as problem resoloved after the merge to master action.