Troubleshooting Primer

Sections:

Distinguishes between debugging (working with a codebase into which the debugger has full visibility) and troubleshooting (investigating and resolving problems in complex systems where the troubleshooter may not have full visibility or knowledge).

Outlines general approaches to troubleshooting, including defining the problem, understanding the request path, bisecting the problem space, and generating and proving/disproving hypotheses.

Presents two real-world troubleshooting scenarios (Slack Client Crashes and Broken Load Balancers) to illustrate the troubleshooting process and methods.

Lists various system-specific and general-purpose tools that can be used for troubleshooting, such as Linux OS-level tooling, TCP packet dumps, application-specific counters, and logging.

Provides links to additional resources and tutorials related to troubleshooting and the tools mentioned in the document.

The difference between troubleshooting and debugging
General troubleshooting methods, including:
- Defining the problem
- Understanding the request path
- Bisecting the problem space
- Generating and proving/disproving hypotheses
How to examine possible causes and test solutions
The concept of bottlenecks and how they affect system performance
The USE (Utilisation, Saturation, Errors) method for troubleshooting performance problems
How to analyze real-world troubleshooting scenarios to understand the troubleshooting process and methods
Familiarity with basic Linux tooling for troubleshooting, such as:
- perf, strace, ltrace, top, htop, sar, netstat, lsof, kill, df, du, iotop, ps, pstree, /proc/ filesystem, dmesg, system logfiles, journalctl, cat, less, grep, sed, awk, jq
Tools for debugging network or connectivity issues, such as:
- dig (for DNS), traceroute, tcpdump and wireshark, netcat, curl
How to use man pages for finding information and options for common Linux tools
Basic understanding of eBPF (Extended Berkeley Packet Filter) and its role in observability tools