Continuous profiling for analysis of CPU and memory usage, down to the line number and throughout time. Saving infrastructure cost, improving performance, and increasing reliability.
Normal flow reproducer
Then, execute ClickHouse server:
```
mkdir ch-data && cd ch-data
../clickhouse server
```
Try to profile it with Parca (you will need to compile parca-agent with the latest version of the upstream OTel eBPF profiler to get the fix for [this](https://github.com/open-telemetry/opentelemetry-ebpf-profiler/issues/230)). Also use `--debuginfo-upload-max-size=2244458216` to be able to upload the binary debuginfo. You will observe that stacks are missing symbols in the UI.
Smaller reproducer
Compile and run this as a binary in this repo (pardon my poor Go):
```go
package main
import (
"context"
"debug/elf"
"os"
"github.com/go-kit/log"
"github.com/go-kit/log/level"
"github.com/parca-dev/parca/pkg/symbol/addr2line"
"github.com/parca-dev/parca/pkg/symbol/demangle"
)
func main() {
ctx := context.Background()
path := "clickhouse"
f, err := elf.Open(path)
if err != nil {
print(err)
}
demangler := demangle.NewDemangler("simple", false)
writer := log.NewSyncWriter(os.Stdout)
logger := log.NewLogfmtLogger(writer)
liner, err := addr2line.DWARF(logger, path, f, demangler)
if err != nil {
print(err)
}
lines, err := liner.PCToLines(ctx, 0x1192c860)
if err != nil {
level.Error(logger).Log("err", err)
}
}
```
Error logs
These come ultimately from debug/dwarf/entry.go, method dwarf5Ranges.
Parca:
level=debug name=parca ts=2024-11-15T12:39:31.37757694Z caller=symbolizer.go:172 component=symbolizer msg="failed to get lines" err="liner pctolines: failed to extract dwarf tree: invalid rnglist offset 409180350 (max 636512)"
Also to note, I've checked some of the generated stacks with this addr2line implementation and they seem to make sense, so stack walking seems to be producing valid addresses.
Other comments
Any tips to debug or cross-check this with dwarfdump or other tools are welcome. For example, how could I check if the offsets that the entry offsets the code is parsing at each step are valid?
Parca can't symbolize addresses for ClickHouse binaries.
How to reproduce
First, download and decompress ClickHouse binary:
Normal flow reproducer
Then, execute ClickHouse server: ``` mkdir ch-data && cd ch-data ../clickhouse server ``` Try to profile it with Parca (you will need to compile parca-agent with the latest version of the upstream OTel eBPF profiler to get the fix for [this](https://github.com/open-telemetry/opentelemetry-ebpf-profiler/issues/230)). Also use `--debuginfo-upload-max-size=2244458216` to be able to upload the binary debuginfo. You will observe that stacks are missing symbols in the UI.Smaller reproducer
Compile and run this as a binary in this repo (pardon my poor Go): ```go package main import ( "context" "debug/elf" "os" "github.com/go-kit/log" "github.com/go-kit/log/level" "github.com/parca-dev/parca/pkg/symbol/addr2line" "github.com/parca-dev/parca/pkg/symbol/demangle" ) func main() { ctx := context.Background() path := "clickhouse" f, err := elf.Open(path) if err != nil { print(err) } demangler := demangle.NewDemangler("simple", false) writer := log.NewSyncWriter(os.Stdout) logger := log.NewLogfmtLogger(writer) liner, err := addr2line.DWARF(logger, path, f, demangler) if err != nil { print(err) } lines, err := liner.PCToLines(ctx, 0x1192c860) if err != nil { level.Error(logger).Log("err", err) } } ```Error logs
These come ultimately from
debug/dwarf/entry.go, methoddwarf5Ranges.Parca:
Reproducer bin:
Cross-checking with other tools
Using
gimli'saddr2line(0x1192c860is one of the addresses it fails to symbolize):Also to note, I've checked some of the generated stacks with this
addr2lineimplementation and they seem to make sense, so stack walking seems to be producing valid addresses.Other comments
Any tips to debug or cross-check this with
dwarfdumpor other tools are welcome. For example, how could I check if the offsets that the entry offsets the code is parsing at each step are valid?