Open keyolk opened 3 years ago
Contents Preface
Acknowledgments
About the Author
1 Introduction
1.1 Systems Performance
1.2 Roles
1.3 Activities
1.4 Perspectives
1.5 Performance Is Challenging
1.5.1 Subjectivity
1.5.2 Complexity
1.5.3 Multiple Causes
1.5.4 Multiple Performance Issues
1.6 Latency
1.7 Observability
1.7.1 Counters, Statistics, and Metrics
1.7.2 Profiling
1.7.3 Tracing
1.8 Experimentation
1.9 Cloud Computing
1.10 Methodologies
1.10.1 Linux Perf Analysis in 60 Seconds
1.11 Case Studies
1.11.1 Slow Disks
1.11.2 Software Change
1.11.3 More Reading
1.12 References
2 Methodologies
2.1 Terminology
2.2 Models
2.2.1 System Under Test
2.2.2 Queueing System
2.3 Concepts
2.3.1 Latency
2.3.2 Time Scales
2.3.3 Trade-Offs
2.3.4 Tuning Efforts
2.3.5 Level of Appropriateness
2.3.6 When to Stop Analysis
2.3.7 Point-in-Time Recommendations
2.3.8 Load vs. Architecture
2.3.9 Scalability
2.3.10 Metrics
2.3.11 Utilization
2.3.12 Saturation
2.3.13 Profiling
2.3.14 Caching
2.3.15 Known-Unknowns
2.4 Perspectives
2.4.1 Resource Analysis
2.4.2 Workload Analysis
2.5 Methodology
2.5.1 Streetlight Anti-Method
2.5.2 Random Change Anti-Method
2.5.3 Blame-Someone-Else Anti-Method
2.5.4 Ad Hoc Checklist Method
2.5.5 Problem Statement
2.5.6 Scientific Method
2.5.7 Diagnosis Cycle
2.5.8 Tools Method
2.5.9 The USE Method
2.5.10 The RED Method
2.5.11 Workload Characterization
2.5.12 Drill-Down Analysis
2.5.13 Latency Analysis
2.5.14 Method R
2.5.15 Event Tracing
2.5.16 Baseline Statistics
2.5.17 Static Performance Tuning
2.5.18 Cache Tuning
2.5.19 Micro-Benchmarking
2.5.20 Performance Mantras
2.6 Modeling
2.6.1 Enterprise vs. Cloud
2.6.2 Visual Identification
2.6.3 Amdahl’s Law of Scalability
2.6.4 Universal Scalability Law
2.6.5 Queueing Theory
2.7 Capacity Planning
2.7.1 Resource Limits
2.7.2 Factor Analysis
2.7.3 Scaling Solutions
2.8 Statistics
2.8.1 Quantifying Performance Gains
2.8.2 Averages
2.8.3 Standard Deviation, Percentiles, Median
2.8.4 Coefficient of Variation
2.8.5 Multimodal Distributions
2.8.6 Outliers
2.9 Monitoring
2.9.1 Time-Based Patterns
2.9.2 Monitoring Products
2.9.3 Summary-Since-Boot
2.10 Visualizations
2.10.1 Line Chart
2.10.2 Scatter Plots
2.10.3 Heat Maps
2.10.4 Timeline Charts
2.10.5 Surface Plot
2.10.6 Visualization Tools
2.11 Exercises
2.12 References
3 Operating Systems
3.1 Terminology
3.2 Background
3.2.1 Kernel
3.2.2 Kernel and User Modes
3.2.3 System Calls
3.2.4 Interrupts
3.2.5 Clock and Idle
3.2.6 Processes
3.2.7 Stacks
3.2.8 Virtual Memory
3.2.9 Schedulers
3.2.10 File Systems
3.2.11 Caching
3.2.12 Networking
3.2.13 Device Drivers
3.2.14 Multiprocessor
3.2.15 Preemption
3.2.16 Resource Management
3.2.17 Observability
3.3 Kernels
3.3.1 Unix
3.3.2 BSD
3.3.3 Solaris
3.4 Linux
3.4.1 Linux Kernel Developments
3.4.2 systemd
3.4.3 KPTI (Meltdown)
3.4.4 Extended BPF
3.5 Other Topics
3.5.1 PGO Kernels
3.5.2 Unikernels
3.5.3 Microkernels and Hybrid Kernels
3.5.4 Distributed Operating Systems
3.6 Kernel Comparisons
3.7 Exercises
3.8 References
3.8.1 Additional Reading
4 Observability Tools
4.1 Tool Coverage
4.1.1 Static Performance Tools
4.1.2 Crisis Tools
4.2 Tool Types
4.2.1 Fixed Counters
4.2.2 Profiling
4.2.3 Tracing
4.2.4 Monitoring
4.3 Observability Sources
4.3.1 /proc
4.3.2 /sys
4.3.3 Delay Accounting
4.3.4 netlink
4.3.5 Tracepoints
4.3.6 kprobes
4.3.7 uprobes
4.3.8 USDT
4.3.9 Hardware Counters (PMCs)
4.3.10 Other Observability Sources
4.4 sar
4.4.1 sar(1) Coverage
4.4.2 sar(1) Monitoring
4.4.3 sar(1) Live
4.4.4 sar(1) Documentation
4.5 Tracing Tools
4.6 Observing Observability
4.7 Exercises
4.8 References
5 Applications
5.1 Application Basics
5.1.1 Objectives
5.1.2 Optimize the Common Case
5.1.3 Observability
5.1.4 Big O Notation
5.2 Application Performance Techniques
5.2.1 Selecting an I/O Size
5.2.2 Caching
5.2.3 Buffering
5.2.4 Polling
5.2.5 Concurrency and Parallelism
5.2.6 Non-Blocking I/O
5.2.7 Processor Binding
5.2.8 Performance Mantras
5.3 Programming Languages
5.3.1 Compiled Languages
5.3.2 Interpreted Languages
5.3.3 Virtual Machines
5.3.4 Garbage Collection
5.4 Methodology
5.4.1 CPU Profiling
5.4.2 Off-CPU Analysis
5.4.3 Syscall Analysis
5.4.4 USE Method
5.4.5 Thread State Analysis
5.4.6 Lock Analysis
5.4.7 Static Performance Tuning
5.4.8 Distributed Tracing
5.5 Observability Tools
5.5.1 perf
5.5.2 profile
5.5.3 offcputime
5.5.4 strace
5.5.5 execsnoop
5.5.6 syscount
5.5.7 bpftrace
5.6 Gotchas
5.6.1 Missing Symbols
5.6.2 Missing Stacks
5.7 Exercises
5.8 References
6 CPUs
6.1 Terminology
6.2 Models
6.2.1 CPU Architecture
6.2.2 CPU Memory Caches
6.2.3 CPU Run Queues
6.3 Concepts
6.3.1 Clock Rate
6.3.2 Instructions
6.3.3 Instruction Pipeline
6.3.4 Instruction Width
6.3.5 Instruction Size
6.3.6 SMT
6.3.7 IPC, CPI
6.3.8 Utilization
6.3.9 User Time/Kernel Time
6.3.10 Saturation
6.3.11 Preemption
6.3.12 Priority Inversion
6.3.13 Multiprocess, Multithreading
6.3.14 Word Size
6.3.15 Compiler Optimization
6.4 Architecture
6.4.1 Hardware
6.4.2 Software
6.5 Methodology
6.5.1 Tools Method
6.5.2 USE Method
6.5.3 Workload Characterization
6.5.4 Profiling
6.5.5 Cycle Analysis
6.5.6 Performance Monitoring
6.5.7 Static Performance Tuning
6.5.8 Priority Tuning
6.5.9 Resource Controls
6.5.10 CPU Binding
6.5.11 Micro-Benchmarking
6.6 Observability Tools
6.6.1 uptime
6.6.2 vmstat
6.6.3 mpstat
6.6.4 sar
6.6.5 ps
6.6.6 top
6.6.7 pidstat
6.6.8 time, ptime
6.6.9 turbostat
6.6.10 showboost
6.6.11 pmcarch
6.6.12 tlbstat
6.6.13 perf
6.6.14 profile
6.6.15 cpudist
6.6.16 runqlat
6.6.17 runqlen
6.6.18 softirqs
6.6.19 hardirqs
6.6.20 bpftrace
6.6.21 Other Tools
6.7 Visualizations
6.7.1 Utilization Heat Map
6.7.2 Subsecond-Offset Heat Map
6.7.3 Flame Graphs
6.7.4 FlameScope
6.8 Experimentation
6.8.1 Ad Hoc
6.8.2 SysBench
6.9 Tuning
6.9.1 Compiler Options
6.9.2 Scheduling Priority and Class
6.9.3 Scheduler Options
6.9.4 Scaling Governors
6.9.5 Power States
6.9.6 CPU Binding
6.9.7 Exclusive CPU Sets
6.9.8 Resource Controls
6.9.9 Security Boot Options
6.9.10 Processor Options (BIOS Tuning)
6.10 Exercises
6.11 References
7 Memory
7.1 Terminology
7.2 Concepts
7.2.1 Virtual Memory
7.2.2 Paging
7.2.3 Demand Paging
7.2.4 Overcommit
7.2.5 Process Swapping
7.2.6 File System Cache Usage
7.2.7 Utilization and Saturation
7.2.8 Allocators
7.2.9 Shared Memory
7.2.10 Working Set Size
7.2.11 Word Size
7.3 Architecture
7.3.1 Hardware
7.3.2 Software
7.3.3 Process Virtual Address Space
7.4 Methodology
7.4.1 Tools Method
7.4.2 USE Method
7.4.3 Characterizing Usage
7.4.4 Cycle Analysis
7.4.5 Performance Monitoring
7.4.6 Leak Detection
7.4.7 Static Performance Tuning
7.4.8 Resource Controls
7.4.9 Micro-Benchmarking
7.4.10 Memory Shrinking
7.5 Observability Tools
7.5.1 vmstat
7.5.2 PSI
7.5.3 swapon
7.5.4 sar
7.5.5 slabtop
7.5.6 numastat
7.5.7 ps
7.5.8 top
7.5.9 pmap
7.5.10 perf
7.5.11 drsnoop
7.5.12 wss
7.5.13 bpftrace
7.5.14 Other Tools
7.6 Tuning
7.6.1 Tunable Parameters
7.6.2 Multiple Page Sizes
7.6.3 Allocators
7.6.4 NUMA Binding
7.6.5 Resource Controls
7.7 Exercises
7.8 References
8 File Systems
8.1 Terminology
8.2 Models
8.2.1 File System Interfaces
8.2.2 File System Cache
8.2.3 Second-Level Cache
8.3 Concepts
8.3.1 File System Latency
8.3.2 Caching
8.3.3 Random vs. Sequential I/O
8.3.4 Prefetch
8.3.5 Read-Ahead
8.3.6 Write-Back Caching
8.3.7 Synchronous Writes
8.3.8 Raw and Direct I/O
8.3.9 Non-Blocking I/O
8.3.10 Memory-Mapped Files
8.3.11 Metadata
8.3.12 Logical vs. Physical I/O
8.3.13 Operations Are Not Equal
8.3.14 Special File Systems
8.3.15 Access Timestamps
8.3.16 Capacity
8.4 Architecture
8.4.1 File System I/O Stack
8.4.2 VFS
8.4.3 File System Caches
8.4.4 File System Features
8.4.5 File System Types
8.4.6 Volumes and Pools
8.5 Methodology
8.5.1 Disk Analysis
8.5.2 Latency Analysis
8.5.3 Workload Characterization
8.5.4 Performance Monitoring
8.5.5 Static Performance Tuning
8.5.6 Cache Tuning
8.5.7 Workload Separation
8.5.8 Micro-Benchmarking
8.6 Observability Tools
8.6.1 mount
8.6.2 free
8.6.3 top
8.6.4 vmstat
8.6.5 sar
8.6.6 slabtop
8.6.7 strace
8.6.8 fatrace
8.6.9 LatencyTOP
8.6.10 opensnoop
8.6.11 filetop
8.6.12 cachestat
8.6.13 ext4dist (xfs, zfs, btrfs, nfs)
8.6.14 ext4slower (xfs, zfs, btrfs, nfs)
8.6.15 bpftrace
8.6.17 Other Tools
8.6.18 Visualizations
8.7 Experimentation
8.7.1 Ad Hoc
8.7.2 Micro-Benchmark Tools
8.7.3 Cache Flushing
8.8 Tuning
8.8.1 Application Calls
8.8.2 ext4
8.8.3 ZFS
8.9 Exercises
8.10 References
9 Disks
9.1 Terminology
9.2 Models
9.2.1 Simple Disk
9.2.2 Caching Disk
9.2.3 Controller
9.3 Concepts
9.3.1 Measuring Time
9.3.2 Time Scales
9.3.3 Caching
9.3.4 Random vs. Sequential I/O
9.3.5 Read/Write Ratio
9.3.6 I/O Size
9.3.7 IOPS Are Not Equal
9.3.8 Non-Data-Transfer Disk Commands
9.3.9 Utilization
9.3.10 Saturation
9.3.11 I/O Wait
9.3.12 Synchronous vs. Asynchronous
9.3.13 Disk vs. Application I/O
9.4 Architecture
9.4.1 Disk Types
9.4.2 Interfaces
9.4.3 Storage Types
9.4.4 Operating System Disk I/O Stack
9.5 Methodology
9.5.1 Tools Method
9.5.2 USE Method
9.5.3 Performance Monitoring
9.5.4 Workload Characterization
9.5.5 Latency Analysis
9.5.6 Static Performance Tuning
9.5.7 Cache Tuning
9.5.8 Resource Controls
9.5.9 Micro-Benchmarking
9.5.10 Scaling
9.6 Observability Tools
9.6.1 iostat
9.6.2 sar
9.6.3 PSI
9.6.4 pidstat
9.6.5 perf
9.6.6 biolatency
9.6.7 biosnoop
9.6.8 iotop, biotop
9.6.9 biostacks
9.6.10 blktrace
9.6.11 bpftrace
9.6.12 MegaCli
9.6.13 smartctl
9.6.14 SCSI Logging
9.6.15 Other Tools
9.7 Visualizations
9.7.1 Line Graphs
9.7.2 Latency Scatter Plots
9.7.3 Latency Heat Maps
9.7.4 Offset Heat Maps
9.7.5 Utilization Heat Maps
9.8 Experimentation
9.8.1 Ad Hoc
9.8.2 Custom Load Generators
9.8.3 Micro-Benchmark Tools
9.8.4 Random Read Example
9.8.5 ioping
9.8.6 fio
9.8.7 blkreplay
9.9 Tuning
9.9.1 Operating System Tunables
9.9.2 Disk Device Tunables
9.9.3 Disk Controller Tunables
9.10 Exercises
9.11 References
10 Network
10.1 Terminology
10.2 Models
10.2.1 Network Interface
10.2.2 Controller
10.2.3 Protocol Stack
10.3 Concepts
10.3.1 Networks and Routing
10.3.2 Protocols
10.3.3 Encapsulation
10.3.4 Packet Size
10.3.5 Latency
10.3.6 Buffering
10.3.7 Connection Backlog
10.3.8 Interface Negotiation
10.3.9 Congestion Avoidance
10.3.10 Utilization
10.3.11 Local Connections
10.4 Architecture
10.4.1 Protocols
10.4.2 Hardware
10.4.3 Software
10.5 Methodology
10.5.1 Tools Method
10.5.2 USE Method
10.5.3 Workload Characterization
10.5.4 Latency Analysis
10.5.5 Performance Monitoring
10.5.6 Packet Sniffing
10.5.7 TCP Analysis
10.5.8 Static Performance Tuning
10.5.9 Resource Controls
10.5.10 Micro-Benchmarking
10.6 Observability Tools
10.6.1 ss
10.6.2 ip
10.6.3 ifconfig
10.6.4 nstat
10.6.5 netstat
10.6.6 sar
10.6.7 nicstat
10.6.8 ethtool
10.6.9 tcplife
10.6.10 tcptop
10.6.11 tcpretrans
10.6.12 bpftrace
10.6.13 tcpdump
10.6.14 Wireshark
10.6.15 Other Tools
10.7 Experimentation
10.7.1 ping
10.7.2 traceroute
10.7.3 pathchar
10.7.4 iperf
10.7.5 netperf
10.7.6 tc
10.7.7 Other Tools
10.8 Tuning
10.8.1 System-Wide
10.8.2 Socket Options
10.8.3 Configuration
10.9 Exercises
10.10 References
11 Cloud Computing
11.1 Background
11.1.1 Instance Types
11.1.2 Scalable Architecture
11.1.3 Capacity Planning
11.1.4 Storage
11.1.5 Multitenancy
11.1.6 Orchestration (Kubernetes)
11.2 Hardware Virtualization
11.2.1 Implementation
11.2.2 Overhead
11.2.3 Resource Controls
11.2.4 Observability
11.3 OS Virtualization
11.3.1 Implementation
11.3.2 Overhead
11.3.3 Resource Controls
11.3.4 Observability
11.4 Lightweight Virtualization
11.4.1 Implementation
11.4.2 Overhead
11.4.3 Resource Controls
11.4.4 Observability
11.5 Other Types
11.6 Comparisons
11.7 Exercises
11.8 References
12 Benchmarking
12.1 Background
12.1.1 Reasons
12.1.2 Effective Benchmarking
12.1.3 Benchmarking Failures
12.2 Benchmarking Types
12.2.1 Micro-Benchmarking
12.2.2 Simulation
12.2.3 Replay
12.2.4 Industry Standards
12.3 Methodology
12.3.1 Passive Benchmarking
12.3.2 Active Benchmarking
12.3.3 CPU Profiling
12.3.4 USE Method
12.3.5 Workload Characterization
12.3.6 Custom Benchmarks
12.3.7 Ramping Load
12.3.8 Sanity Check
12.3.9 Statistical Analysis
12.3.10 Benchmarking Checklist
12.4 Benchmark Questions
12.5 Exercises
12.6 References
13 perf
13.1 Subcommands Overview
13.2 One-Liners
13.3 perf Events
13.4 Hardware Events
13.4.1 Frequency Sampling
13.5 Software Events
13.6 Tracepoint Events
13.7 Probe Events
13.7.1 kprobes
13.7.2 uprobes
13.7.3 USDT
13.8 perf stat
13.8.1 Options
13.8.2 Interval Statistics
13.8.3 Per-CPU Balance
13.8.4 Event Filters
13.8.5 Shadow Statistics
13.9 perf record
13.9.1 Options
13.9.2 CPU Profiling
13.9.3 Stack Walking
13.10 perf report
13.10.1 TUI
13.10.2 STDIO
13.11 perf script
13.11.1 Flame Graphs
13.11.2 Trace Scripts
13.12 perf trace
13.12.1 Kernel Versions
13.13 Other Commands
13.14 perf Documentation
13.15 References
14 Ftrace
14.1 Capabilities Overview
14.2 tracefs (/sys)
14.2.1 tracefs Contents
14.3 Ftrace Function Profiler
14.4 Ftrace Function Tracing
14.4.1 Using trace
14.4.2 Using trace_pipe
14.4.3 Options
14.5 Tracepoints
14.5.1 Filter
14.5.2 Trigger
14.6 kprobes
14.6.1 Event Tracing
14.6.2 Arguments
14.6.3 Return Values
14.6.4 Filters and Triggers
14.6.5 kprobe Profiling
14.7 uprobes
14.7.1 Event Tracing
14.7.2 Arguments and Return Values
14.7.3 Filters and Triggers
14.7.4 uprobe Profiling
14.8 Ftrace function_graph
14.8.1 Graph Tracing
14.8.2 Options
14.9 Ftrace hwlat
14.10 Ftrace Hist Triggers
14.10.1 Single Keys
14.10.2 Fields
14.10.3 Modifiers
14.10.4 PID Filters
14.10.5 Multiple Keys
14.10.6 Stack Trace Keys
14.10.7 Synthetic Events
14.11 trace-cmd
14.11.1 Subcommands Overview
14.11.2 trace-cmd One-Liners
14.11.3 trace-cmd vs. perf(1)
14.11.4 trace-cmd function_graph
14.11.5 KernelShark
14.11.6 trace-cmd Documentation
14.12 perf ftrace
14.13 perf-tools
14.13.1 Tool Coverage
14.13.2 Single-Purpose Tools
14.13.3 Multi-Purpose Tools
14.13.4 perf-tools One-Liners
14.13.5 Example
14.13.6 perf-tools vs. BCC/BPF
14.13.7 Documentation
14.14 Ftrace Documentation
14.15 References
15 BPF
15.1 BCC
15.1.1 Installation
15.1.2 Tool Coverage
15.1.3 Single-Purpose Tools
15.1.4 Multi-Purpose Tools
15.1.5 One-Liners
15.1.6 Multi-Tool Example
15.1.7 BCC vs. bpftrace
15.1.8 Documentation
15.2 bpftrace
15.2.1 Installation
15.2.2 Tools
15.2.3 One-Liners
15.2.4 Programming
15.2.5 Reference
15.2.6 Documentation
15.3 References
16 Case Study
16.1 An Unexplained Win
16.1.1 Problem Statement
16.1.2 Analysis Strategy
16.1.3 Statistics
16.1.4 Configuration
16.1.5 PMCs
16.1.6 Software Events
16.1.7 Tracing
16.1.8 Conclusion
16.2 Additional Information
16.3 References
A USE Method: Linux
B sar Summary
C bpftrace One-Liners
D Solutions to Selected Exercises
E Systems Performance Who’s Who
Glossary
Index
What is
Observarbility vs Experimental Tools
Performance Observability
Role of methodologies & 60 sec checklist
Performance
SRE
Latency, Utilization, Saturation
Trade offs
Workload vs Architecture
Resource vs Analysis
Steetlight anti-method
Random change anti-method
Blame-someone-else anti-method
Ad hoc checklist method
Problem statement
Scientic method
Diagnoise cycle
Tools method
USE method
RED method
Workload characterization
Drill-down analysis
Latency analysis
Method R
Event tracing
Baseline statistics
Static performance tuning
Performance mantras
Queueing theory
Capacity planning
Quantifying performance gains
Performance monitoring
Event | Latency | Scaled |
---|---|---|
1 CPU cycle | 0.3 ns | 1 s |
Level 1 cache access | 0.9 ns | 3 s |
Level 2 cache access | 3 ns | 10 s |
Level 3 cache access | 10 ns | 33 s |
Main memory access (DRAM, from CPU) | 100 ns | 6 min |
Solid-state disk I/O (flash memory) | 10–100 μs | 9–90 hours |
Rotational disk I/O | 1–10 ms | 1–12 months |
Internet: San Francisco to New York | 40 ms | 4 years |
Internet: San Francisco to United Kingdom | 81 ms | 8 years |
Lightweight hardware virtualization boot | 100 ms | 11 years |
Internet: San Francisco to Australia | 183 ms | 19 years |
OS virtualization system boot | < 1 s | 105 years |
TCP timer-based retransmit | 1–3 s | 105–317 years |
SCSI command time-out | 30 s | 3 millennia |
Hardware (HW) virtualization system boot | 40 s | 4 millennia |
Physical system reboot | 5 m | 32 millennia |
Layer | Example Tuning Targets |
---|---|
Application | Application logic, request queue sizes, database queries performed |
Database | Database table layout, indexes, buffering |
System calls | Memory-mapped or read/write, sync or async I/O flags |
File system | Record size, cache size, file system tunables, journaling |
Storage | RAID level, number and type of disks, storage tunables |
Throughput
IOPS
Utilization
Ltency
Overhead
Issues
Requests
Latency
Completion
Throughput
Latency
Section | Methodology | Type |
---|---|---|
2.5.1 | Streetlight anti-method | Observational analysis |
2.5.2 | Random change anti-method | Experimental analysis |
2.5.3 | Blame-someone-else anti-method | Hypothetical analysis |
2.5.4 | Ad hoc checklist method | Observational and experimental analysis |
2.5.5 | Problem statement | Information gathering |
2.5.6 | Scientific method | Observational analysis |
2.5.7 | Diagnosis cycle | Analysis life cycle |
2.5.8 | Tools method | Observational analysis |
2.5.9 | USE method | Observational analysis |
2.5.10 | RED method | Observational analysis |
2.5.11 | Workload characterization | Observational analysis, capacity planning |
2.5.12 | Drill-down analysis | Observational analysis |
2.5.13 | Latency analysis | Observational analysis |
2.5.14 | Method R | Observational analysis |
2.5.15 | Event tracing | Observational analysis |
2.5.16 | Baseline statistics | Observational analysis |
2.5.17 | Static performance tuning | Observational analysis, capacity planning |
2.5.18 | Cache tuning | Observational analysis, tuning |
2.5.19 | Micro-benchmarking | Experimental analysis |
2.5.20 | Performance mantras | Tuning |
2.6.5 | Queueing theory | Statistical analysis, capacity planning |
2.7 | Capacity planning | Capacity planning, tuning |
2.8.1 | Quantifying performance gains | Statistical analysis |
2.9 | Performance monitoring | Observational analysis, capacity planning |
Section | Methodology | Type |
---|---|---|
1.10.1 | Linux performance analysis in 60s | Observational analysis |
5.4.1 | CPU profiling | Observational analysis |
5.4.2 | Off-CPU analysis | Observational analysis |
6.5.5 | Cycle analysis | Observational analysis |
6.5.8 | Priority tuning | Tuning |
6.5.8 | Resource controls | Tuning |
6.5.9 | CPU binding | Tuning |
7.4.6 | Leak detection | Observational analysis |
7.4.10 | Memory shrinking | Experimental analysis |
8.5.1 | Disk analysis | Observational analysis |
8.5.7 | Workload separation | Tuning |
9.5.10 | Scaling | Capacity planning, tuning |
10.5.6 | Packet sniffing | Observational analysis |
10.5.7 | TCP analysis | Observational analysis |
12.3.1 | Passive benchmarking | Experimental analysis |
12.3.2 | Active benchmarking | Observational analysis |
12.3.6 | Custom benchmarks | Software development |
12.3.7 | Ramping load | Experimental analysis |
12.3.8 | Sanity check | Observational analysis |
Resource | Type | Metric |
---|---|---|
CPU | Utilization | CPU utilization (either per CPU or a system-wide average) |
CPU | Saturation | Run queue length, scheduler latency, CPU pressure (Linux PSI) |
Memory | Utilization | Available free memory (system-wide) |
Memory | Saturation | Swapping (anonymous paging), page scanning, out-of-memory events, memory pressure (Linux PSI) |
Network interface | Utilization | Receive throughput/max bandwidth, transmit throughput/max bandwidth |
Storage device I/O | Utilization | Device busy percent |
Storage device I/O | Saturation | Wait queue length, I/O pressure (Linux PSI) |
Storage device I/O | Errors | Device errors (“soft,” “hard”) |
Resource | Type | Metric |
---|---|---|
CPU | Errors | For example, machine check exceptions, CPU cache errors5 |
Memory | Errors | For example, failed malloc()s (although a default Linux kernel configuration makes this rare due to overcommit) |
Network | Saturation | Saturation-related network interface or OS errors, e.g., Linux “overruns” |
Storage controller | Utilization | Depends on the controller; it may have a maximum IOPS or throughput that can be checked against current activity |
CPU interconnect | Utilization | Per-port throughput/maximum bandwidth (CPU performance counters) |
Memory interconnect | Saturation | Memory stall cycles, high cycles per instruction (CPU performance counters) |
I/O interconnect | Utilization | Bus throughput/maximum bandwidth (performance counters may exist on your HW, e.g., Intel “uncore” events) |
Software Resource
Resource Controls
Microservices
http://www.brendangregg.com/systems-performance-2nd-edition-book.html