ARM-software / lisa

Linux Integrated System Analysis
Apache License 2.0
200 stars 119 forks source link

lisa/trace: Fix dataframe prefiltering for userspace ftrace events #2153

Closed msrasmussen closed 9 months ago

msrasmussen commented 9 months ago

It looks like the appearance of userspace generated ftrace events has changed. What appears to be memory address has been added to the event meta data breaking the prefiltering of trace data in _select_userspace().

The event used contain the keyword 'tracing_mark_write:', but now has the format: 'tracing_mark_write.

:'.

Example: rt-app-21908 [001] 4470442.464974: print: tracing_mark_write.10e87ffbd993ed977ebcbff93a794b6d: rtapp_main: event=end

Fix the event filtering to handle new format without breaking backwards compatibility.

douglas-raillard-arm commented 9 months ago

Thanks, do you know what kernel commit changed that ? It might actually be related to kallsyms content (entries being swapped) rather than a code change. trace-cmd resolves the IP as the first function name that covers the range of address including IP. There can sometimes be overlapping ranges so you actually have an ambiguity in what name to display.

From a speed point of view, using startswith is probably going to be pretty bad compared to equality comparison, assuming we use a categorical dtype for that column (we should, maybe we don't). Not much we can do about it though.

If you can send me a trace.dat with the new format there is some things I'd like to check for the rust parser.

msrasmussen commented 9 months ago

I have shared the trace.dat with you directly.

I can't see an obvious to speed this up as we don't know exactly what we are looking for, so we have look up something that starts with 'tracing_mark_write' first anyway. So we can only switch back to equality comparison after that.

douglas-raillard-arm commented 9 months ago

Yeah I think we will have to live with that. Everything about these "userspace events" is slow anyway so ...

douglas-raillard-arm commented 9 months ago

Can you update the commit message with the Python module name (rather than some sort of file path) and the FIX tag (used for the changelog generation):

lisa.trace: Fix dataframe prefiltering for userspace ftrace events 

FIX 

It looks like the appearance of userspace generated ftrace events has
changed. What appears to be memory address has been added to the event
meta data breaking the prefiltering of trace data in
_select_userspace().

The event used contain the keyword 'tracing_mark_write:', but now has
the format: 'tracing_mark_write.<address>:'.

Example:
   rt-app-21908 [001] 4470442.464974: print:
tracing_mark_write.10e87ffbd993ed977ebcbff93a794b6d: rtapp_main:
event=end

Fix the event filtering to handle new format without breaking backwards
compatibility.