pkg/report: duplicates on different arches/compilers

[dvyukov]

Currently we extract thew function name for the title before symbolization, as the result inlined functions are never considered. Inlining can depend on compiler, arch, potentially config and unrelated code changes (e.g. some function is optimized to be inlinable).

Perhaps we need to consider doing function extraction after symbolization. The main issue is to ensure that we won't report any duplicates for existing bugs during the switch. AltTitles can do this. We could run function extraction twice (with/without "[inline]" lines) and add the second title as AltTitle.

Here is one example:

WARNING in static_key_slow_try_dec
------------[ cut here ]------------
jump label: negative count!
WARNING: CPU: 0 PID: 27466 at kernel/jump_label.c:235 static_key_slow_try_dec+0xca/0xe0 kernel/jump_label.c:235
RIP: 0010:static_key_slow_try_dec+0xca/0xe0 kernel/jump_label.c:235
Call Trace:
<TASK>
__static_key_slow_dec_cpuslocked kernel/jump_label.c:243 [inline]
__static_key_slow_dec kernel/jump_label.c:255 [inline]
static_key_slow_dec+0x5c/0xc0 kernel/jump_label.c:270

vs:

WARNING in __static_key_slow_dec_cpuslocked
------------[ cut here ]------------
jump label: negative count!
WARNING: CPU: 0 PID: 3047 at kernel/jump_label.c:235 static_key_slow_try_dec kernel/jump_label.c:235 [inline]
WARNING: CPU: 0 PID: 3047 at kernel/jump_label.c:235 __static_key_slow_dec_cpuslocked+0x100/0x138 kernel/jump_label.c:243
pc : static_key_slow_try_dec kernel/jump_label.c:235 [inline]
pc : __static_key_slow_dec_cpuslocked+0x100/0x138 kernel/jump_label.c:243
lr : static_key_slow_try_dec kernel/jump_label.c:235 [inline]
lr : __static_key_slow_dec_cpuslocked+0x100/0x138 kernel/jump_label.c:243
Call trace:
static_key_slow_try_dec kernel/jump_label.c:235 [inline]
__static_key_slow_dec_cpuslocked+0x100/0x138 kernel/jump_label.c:243
__static_key_slow_dec kernel/jump_label.c:255 [inline]
static_key_slow_dec+0x40/0x78 kernel/jump_label.c:270

[dvyukov]

Another difference between x86 and arm64 caused by different inlining:

WARNING in move_page_tables

WARNING: CPU: 1 PID: 25634 at mm/mremap.c:154 move_page_tables+0x11d4/0x162c
...
pc : move_page_tables+0x11d4/0x162c
lr : move_page_tables+0x11d4/0x162c
Call trace:
move_page_tables+0x11d4/0x162c
move_vma+0x1f4/0x8f8

WARNING in move_ptes

WARNING: CPU: 1 PID: 7487 at mm/mremap.c:154 move_ptes+0x83e/0x910 mm/mremap.c:154
RIP: 0010:move_ptes+0x83e/0x910 mm/mremap.c:154
Call Trace:
move_page_tables+0xcaf/0x1020
move_vma+0x236/0xaa0

[dvyukov]

After symbolization we could also skip frames based on file name. For example, there are constantly new functions added to lockdep, debug objects, slab, etc. We want to skip all of them. But any new function breaks report parsing and we need to update skip patterns to include it. If we could skip all of kernel/locking/lockdep.c, it would help.

[dvyukov]

More and more duplicates and mis-parsed reports recently. One was attributed to:

mark_held_locks+0x9f/0xe0 kernel/locking/lockdep.c:4236

Either a new function, or it stopped being inlined.