ProcDump

ProcDump is a Linux reimagining of the classic ProcDump tool from the Sysinternals suite of tools for Windows. ProcDump provides a convenient way for Linux developers to create core dumps of their application based on performance triggers. ProcDump for Linux is part of Sysinternals.

Installation & Usage

Requirements

• Minimum OS:
  • Red Hat Enterprise Linux / CentOS 7
  • Fedora 29
  • Ubuntu 16.04 LTS
• gdb >= 7.6.1

Install ProcDump

Please see installation instructions here.

Build

Please see build instructions here.

Usage

BREAKING CHANGE With the release of ProcDump 1.3 the switches are now aligned with the Windows ProcDump version.

Capture Usage:
   procdump [-n Count]
            [-s Seconds]
            [-c|-cl CPU_Usage]
            [-m|-ml Commit_Usage1[,Commit_Usage2...]]
            [-gcm [<GCGeneration>: | LOH: | POH:]Memory_Usage1[,Memory_Usage2...]]
            [-gcgen Generation]
            [-restrack [nodump]]
            [-sr Sample_Rate]
            [-tc Thread_Threshold]
            [-fc FileDescriptor_Threshold]
            [-sig Signal_Number1[,Signal_Number2...]]
            [-e]
            [-f Include_Filter,...]
            [-fx Exclude_Filter]
            [-mc Custom_Dump_Mask]
            [-pf Polling_Frequency]
            [-o]
            [-log syslog|stdout]
            {
             {{[-w] Process_Name | [-pgid] PID} [Dump_File | Dump_Folder]}
            }

Options:
   -n      Number of dumps to write before exiting.
   -s      Consecutive seconds before dump is written (default is 10).
   -c      CPU threshold above which to create a dump of the process.
   -cl     CPU threshold below which to create a dump of the process.
   -m      Memory commit threshold(s) (MB) above which to create dumps.
   -ml     Memory commit threshold(s) (MB) below which to create dumps.
   -gcm    [.NET] GC memory threshold(s) (MB) above which to create dumps for the specified generation or heap (default is total .NET memory usage).
   -gcgen  [.NET] Create dump when the garbage collection of the specified generation starts and finishes.
   -restrack Enable memory leak tracking (malloc family of APIs). Use the nodump option to prevent dump generation and only produce restrack report(s).
   -sr     Sample rate when using -restrack.
   -tc     Thread count threshold above which to create a dump of the process.
   -fc     File descriptor count threshold above which to create a dump of the process.
   -sig    Comma separated list of signal number(s) during which any signal results in a dump of the process.
   -e      [.NET] Create dump when the process encounters an exception.
   -f      Filter (include) on the content of .NET exceptions (comma separated). Wildcards (*) are supported.
   -fx     Filter (exclude) on the content of -restrack call stacks. Wildcards (*) are supported.
   -mc     Custom core dump mask (in hex) indicating what memory should be included in the core dump. Please see 'man core' (/proc/[pid]/coredump_filter) for available options.
   -pf     Polling frequency.
   -o      Overwrite existing dump file.
   -log    Writes extended ProcDump tracing to the specified output stream (syslog or stdout).
   -w      Wait for the specified process to launch if it's not running.
   -pgid   Process ID specified refers to a process group ID.

Resource Tracking

The -restrack switch activates resource tracking, allowing for the monitoring and reporting of any resource allocations that have not been freed at the time of generating the core dump. The results are saved to a file with a '.restrack' extension. Currently, the following resource allocation/deallocation functions are tracked:

Allocation:

• malloc
• calloc
• realloc
• reallocarray
• mmap

Deallocation:

• free
• munmap

Examples

The following examples all target a process with pid == 1234

The following will create a core dump immediately.

sudo procdump 1234

The following will create 3 core dumps 10 seconds apart.

sudo procdump -n 3 1234

The following will create 3 core dumps 5 seconds apart.

sudo procdump -n 3 -s 5 1234

The following will create a core dump each time the process has CPU usage >= 65%, up to 3 times, with at least 10 seconds between each dump.

sudo procdump -c 65 -n 3 1234

The following will create a core dump each time the process has CPU usage >= 65%, up to 3 times, with at least 5 seconds between each dump.

sudo procdump -c 65 -n 3 -s 5 1234

The following will create a core dump when CPU usage is outside the range [10,65].

sudo procdump -cl 10 -c 65 1234

The following will create a core dump when CPU usage is >= 65% or memory usage is >= 100 MB.

sudo procdump -c 65 -m 100 1234

The following will create a core dump when memory usage is >= 100 MB followed by another dump when memory usage is >= 200MB.

sudo procdump -m 100,200 1234

The following will create a core dump and a memory leak report when memory usage is >= 100 MB

sudo procdump -m 100 -restrack 1234

The following will create a memory leak report (no dumps) when memory usage is >= 100 MB

sudo procdump -m 100 -restrack nodump 1234

The following will create a core dump and a memory leak report when memory usage is >= 100 MB by sampling every 10th memory allocation.

sudo procdump -m 100 -restrack -sr 10 1234

The following will create a core dump and a memory leak report when memory usage is >= 100 MB and exclude any call stacks that contain frames with the string "cache" in them

sudo procdump -m 100 -restrack -fx *cache* 1234

The following will create a core dump when the total .NET memory usage is >= 100 MB followed by another dump when memory usage is >= 200MB.

sudo procdump -gcm 100,200 1234

The following will create a core dump when .NET memory usage for generation 1 is >= 1 MB followed by another dump when memory usage is >= 2MB.

sudo procdump -gcm 1:1,2 1234

The following will create a core dump when .NET Large Object Heap memory usage is >= 100 MB followed by another dump when memory usage is >= 200MB.

sudo procdump -gcm LOH:100,200 1234

The following will create a core dump at the start and end of a .NET generation 1 garbage collection.

sudo procdump -gcgen 1

The following will create a core dump in the /tmp directory immediately.

sudo procdump 1234 /tmp

The following will create a core dump in the current directory with the name dump_0.1234. If -n is used, the files will be named dump_0.1234, dump_1.1234 and so on.

sudo procdump 1234 dump

The following will create a core dump when a SIGSEGV occurs.

sudo procdump -sig 11 1234

The following will create a core dump when a SIGSEGV occures where the core dump contains only anonymous private mappings.

sudo procdump -mc 1 -sig 11 1234

The following will create a core dump when the target .NET application throws a System.InvalidOperationException

sudo procdump -e -f System.InvalidOperationException 1234

The include filter supports partial and wildcard matching, so the following will create a core dump too for a System.InvalidOperationException

sudo procdump -e -f InvalidOperation 1234

or

sudo procdump -e -f "*Invali*Operation*" 1234

All options can also be used with -w, to wait for any process with the given name.

The following waits for a process named my_application and creates a core dump immediately when it is found.

sudo procdump -w my_application

Current Limitations

• Currently will only run on Linux Kernels version 3.5+
• Does not have full feature parity with Windows version of ProcDump, specifically, stay alive functionality, and custom performance counters

Feedback

• Ask a question on StackOverflow (tag with ProcDumpForLinux)
• Request a new feature on GitHub
• Vote for popular feature requests
• File a bug in GitHub Issues

Contributing

If you are interested in fixing issues and contributing directly to the code base, please see the document How to Contribute, which covers the following:

• How to build and run from source
• The development workflow, including debugging and running tests
• Coding Guidelines
• Submitting pull requests

Please see also our Code of Conduct.

License

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT License.

ProcDump for Linux:

• Clones, compiles and statically links against libbpf (https://github.com/libbpf/libbpf)
• Copies symbol resolution source code from BCC (https://github.com/iovisor/bcc).
• Uses eBPF