Unable to run against 5.16.0-rc2 vmcore file

[tom-seewald]

System info: Architecture: x86_64 Crash versions: 8.0.0 and crash-7.3.0-3.fc35 Kernel version: 5.16.0-rc2 (locally built mainline) OS: Fedora 35

Scenario: I am attempting to debug a 5.16-rc2 vmcore file using crash. The kernel was built locally and I have enabled debug symbols. I have tried using the version of crash shipped with Fedora as well as the latest upstream version.

$ file linux/vmlinux
linux/vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=23e866fb18e69a034354066f5c15f4d13d8e4707, with debug_info, not stripped

$ grep -i debug_info linux/.config
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
# CONFIG_DEBUG_INFO_BTF is not set

Problem: For both crash 8.0.0 and crash-7.3.0-3.fc35 the utility appears to start however it abruptly ends without any error being logged. Systemd's coredumpctl doesn't show any coredumps being generated by crash either.

Example attempts to debug the vmcore file using crash:

$ sudo crash linux/System.map linux/vmlinux /var/crash/127.0.0.1-2021-11-24-16\:45\:46/vmcore

crash 7.3.0-3.fc35
Copyright (C) 2002-2021  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2021  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
please wait... (patching 126848 gdb minimal_symbol values) 

$

$ sudo ~/crash/crash linux/System.map linux/vmlinux /var/crash/127.0.0.1-2021-11-24-16\:45\:46/vmcore

crash 8.0.0
Copyright (C) 2002-2021  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2021  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

$ 

[k-hagio]

Do you need the System.map file? Please see mapfile on the man page.

(It looks like crash-7.3.0-3.fc35 does not have 68870c83d299603c07785e3530e33c13045c87ef, so cannot be used for 5.16 in the first place.)

[tom-seewald]

Do you need the System.map file? Please see mapfile on the man page.

Well I'm running 5.15.4 right now, as I didn't want to stay on a kernel that is unstable for me. If I use /boot/System.map-5.16.0-rc2 I also get the same behavior described above, if I do not specify a System.map file it tells me there's a kernel version mismatch.

[k-hagio]

if I do not specify a System.map file it tells me there's a kernel version mismatch.

yes, I meant the following command:

$ sudo ~/crash/crash linux/vmlinux /var/crash/127.0.0.1-2021-11-24-16\:45\:46/vmcore

if this tells a version mismatch, what do the following commands print?

# file vmlinux
# crash -d 1 vmlinux vmcore

[tom-seewald]

if this tells a version mismatch, what do the following commands print?

# file vmlinux

That should be in my original message.

# crash -d 1 vmlinux vmcore

The output is over 400 lines, but it looks like there is a build id mismatch. However it is the same config, compiler, and kernel version.

[k-hagio]

if this tells a version mismatch, what do the following commands print? # file vmlinux

That should be in my original message.

oh, sorry, I missed that.

# crash -d 1 vmlinux vmcore

The output is over 400 lines, but it looks like there is a build id mismatch. However it is the same config, compiler, and kernel version.

I'm not sure what does it, but no guarantee that symbol addresses are consistent. Please try the vmlinux having the same id as the vmcore.