google / perfetto

Performance instrumentation and tracing for Android, Linux and Chrome (read-only mirror of https://android.googlesource.com/platform/external/perfetto/)
https://www.perfetto.dev
Apache License 2.0
2.92k stars 362 forks source link

Implement ftrace pid filtering support in traced_probes #932

Open wangjiyang opened 1 week ago

wangjiyang commented 1 week ago

Hi Perfetto team.

I am looking for a ftrace filter feature. Background is I am trying to tracing sched:sched_switch ftrace event of some specified processes for a very very long time, eg, serveral days. Enabling sched_switch events for all processes will produce too much data and may consumes CPU and disk. The overhead is too much for our usecases. So I wonder what's the best practice of tracing limited number of processes?

LalitMaganti commented 1 week ago

We don't support any such feature natively in Perfetto. If you want to setup filtering outside Perfetto (i.e. by creating an ftrace synthetic event or directly writing to the /sys files) you can do that and just use Perfetto as a recording mechanism.

Note that any filtering may lead to traces which don't make sense in trace processor: there are various assumptions in trace processor that if see an event like sched_switch, we are seeing all the events of that type. Having inconsistencies can lead to undefined behaviour.

LalitMaganti commented 1 week ago

To be clear, having this functionality in Perfetto is a good idea and I think something we would be supportive of. It's just quite a niche feature (over 7 years, I've only heard it requested 2 or 3 times) and not the priority for us.

I'll reopen this as if someone wants to contribute this, we'd be supportive of adding it (you can look at how the syscall filtering works for inspiration). But it's probably not high enough priority for us to look at ourselves.

wangjiyang commented 1 week ago

I would use some ebpf features to do such filtering. Eg, calling bpf helpers like bpf_trace_printk, and acts as what atrace done to take advantage of ftrace features to link with perfetto. I am still investigating and not sure if traced_probe already has this kind of raw ftrace buffers collected. Please share some information if you know. Based on this next step would be parsing this raw buffer and emit track information to trace processor. This approach would be compatible with current ftrace importer, but has some challege because that would need to create a new data source and it depends on ebpf.

wangjiyang commented 1 week ago

Seems this is what i needed. Is there any samples?

https://perfetto.dev/docs/reference/trace-config-proto#PerfEvents.Tracepoint

LalitMaganti commented 1 week ago

Eg, calling bpf helpers like bpf_trace_printk, and acts as what atrace done to take advantage of ftrace features to link with perfetto.

If you are doing this, you don't need to do any integration with perf etc. You just need to enable the ftrace events ftrace/print, ftrace/bprint and ftrace/bputs and the data should just show up in Perfetto.

https://perfetto.dev/docs/reference/trace-config-proto#PerfEvents.Tracepoint

This is totally irrelevant, this is about traced_perf stack sampling. You don't need this, just enable the events above in the normal ftrace data source.