What is the appropriate kernel ELF to use with dwarf2json?

[J4NS-R]

I'm trying to create a symbol table for my Debian distro. The example in the Readme doesn't work for Debian (since there is no /usr/lib/debug directory).

I looked up where to find the kernel ELF and landed on this StackOverflow question which gave two answers: /boot/vmlinuz-*, and a script to extract vmlinux.

The first case (/boot/vmlinuz-4.19.0-16-amd64 for me) points to an ELF (according to file), but dwarf2json doesn't like it:

$ ./dwarf2json linux --elf /boot/vmlinuz-4.19.0-16-amd64
Failed linux processing: could not open /boot/vmlinuz-4.19.0-16-amd64: bad magic number '[77 90 234 7]' in record at byte 0x0

The second case (vmlinux produced by the script) is also definitely an ELF (once again, according to file), but once again, dwarf2json rejected it:

$ ./extract-vmlinux.sh /boot/vmlinuz-4.19.0-16-amd64 > vmlinux
$ ./dwarf2json linux --elf vmlinux
Failed linux processing: could not get DWARF from vmlinux: decoding dwarf section info at offset 0x0: too short

For sake of interest, here's what file had to say for the two ELF cases above:

$ file /boot/vmlinuz-4.19.0-16-amd64
/boot/vmlinuz-4.19.0-16-amd64: Linux kernel x86 boot executable bzImage, version 4.19.0-16-amd64 (debiankernel@lists.debian.org) #1 SMP Debian 4.19.181-1 (2021-03-19), RO-rootFS, swap_dev 0x5, Normal VGA

$ file vmlinux
vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=dbfab4e43b7d635bce4ac2cb0420e4dfab8049fd, stripped

The Volatility 3 documentation on this topic has exactly one sentence of wisdom to offer:

Once a kernel with debugging symbols/appropriate DWARF file has been located, dwarf2json will convert it into an appropriate JSON file.

Which doesn't really help.

Where should I get the appropriate kernel ELF for my distro so that I can generate the appropriate ISF for vol3?

[ikelos]

The was a good effort, but unfortunately:

• The first file you chose was the running kernel, which is compressed (which is why file identified it as a bzImage rather than an ELF file)
• The second file is probably the decompressed version of the first file, but neither of these is the debug kernel

Since debug kernels can be 100x the size of a normal kernel, they're not usually installed on most systems, and on popular distros are stored in a completely different package. The kernel you're looking for will likely be about 700Mb uncompressed, and since this looks like a debian kernel, you're probably looking for a package like linux-image-amd64*-dbg*. You'll need to make sure that the version number of that package matches exactly with the kernel you're trying to investigate, otherwise the banner will come out differently and volatility won't be able to match the kernels...

Hopefully that gives you enough to go on (you'll also need the system.map file to make a complete JSON file).

[ikelos]

For debian kernels, here's an example list of packages. Each pair of packages together contain the system.map and the debug kernel that you'd need for that version: https://github.com/volatilityfoundation/volatility3/blob/develop/development/debian-kernels.txt

[J4NS-R]

Thanks for your accurate feedback. The wonderful aptitude package manager sorted me right out. The command

sudo apt install linux-image-amd64 linux-image-amd64-dbg

automatically selected and installed the appropriate packages: linux-image-4.19.0-16-amd64 and linux-image-4.19.0-16-amd64-dbg.

Lo and behold, inside /usr/lib/debug/boot/ there lies the debug kernel.

You were right about the sizes too btw. 722MB download, 524MB after install.

[ikelos]

Hehehe, no problem, thanks for letting us know all the things you tried, it makes it much easier for us to gauge how in-depth our responses need to be (and shows you tried to solve it on your own before resorting to us!). 5:) Glad you got it fixed!

[tbdk]

I had the exact same experience as @Koellewe, but following the advice of @ikelos, I found that there seemed to be no debug kernel available for my Ubuntu/kernel version. I looked into https://github.com/volatilityfoundation/volatility3/blob/develop/development/debian-kernels.txt and also tried to expand my search by going directly to those websites and searching for my kernel (without the 'generic' part) - as well as using apt search linux-image together with fzf to try and find something relevant to me.

a) I imagine that there is probably not a (debug) kernel provided for all versions that exist, but if not, what is the reason for this? What decides which kernel versions exists online and which does not?

b) So my kernel version is 5.15.0-56 and let's assume that the debug kernel does not exist online for it. For the purpose of Volatility(3) and dwarf2json, if I can find a debug version for a kernel that is within the same major version as mine, e.g. 5.4.x-xx or 5.15+.x-xx would those work as well?

Output of lsb_release -a:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:   jammy

Output of uname -r: 5.15.0-56-generic

Thanks in advance :slightly_smiling_face:

[ikelos]

The answer is both "probably" but also "not for volatility". There's two main things we rely on in volatility 3, symbol addresses and structures. The symbol addresses are determined by how the compiler decides to lay things out in memory (and can change with each compiler, possibly with each compile but I'm not certain) and the structures are decided by the config options and source code used for the kernel.

This means that if either the compiler or the config changes for the exact same source code, you might end up with differences that could affect volatility. To get anything to work, we needed some way to uniquely identify the kernels and luckily, so do the big distributions. They tend to change the kernel banner to include what machine and when the kernel was compiled, which almost always guarantees individuality. With Windows and Mac, the owner of the OS tends to keep track of their kernels and there's a reasonably limited number. With Linux, anyone can roll their own and we need to be able to support that, so we provide the tools for people to create their own symbol tables. Since the symbol tables contain both the structures and the locations of well-known structures, volatility 3 can much more accurately make its way through memory.

In volatility 2, we never had the locations and it meant that for the most part, we assumed structures didn't shift around that often, but we had to do a lot of work to find the kernel and get a foothold on the memory image. It also meant that someone could be happily running an image with structures that were slightly wrong and get some results whilst others were invalid. This is generally bad from a forensic perspective, so in volatility 3 we made a conscious decision to require accuracy over "probably works most of the time". This also vastly reduced the burden of having to look into each report of something not working quite right.

The upshot of all this is that it's unlikely the structures will have changed much between X.Y.Z-AA and X.Y.Z-BB, but the symbol locations may well have shifted. The good news is that the symbol locations are normally shipped with the kernel itself, not just with the debug kernel, so that's why I say that you can probably get away with the structures from a very similar kernel (with the matching config), but that volatility will flag it as having a different banner because otherwise all the team's time would be sunk into trying to figure out if people have actually discovered a bug or just have a symbol table that's slightly off, so "not for volatility".

Theoretically you could take a debug kernel that's close in order to replicate the structures, and then apply the System.map to use the exact kernel's symbols. Then you'd just have to trick volatility by changing the kernel banner in the generated JSON. Doing that is somewhat our only safety bar to ensure people know what they're doing when they start frankensteining together symbol tables, so we don't get lots of frivolous bugs that drain our time... 5:)

I hope that explains the situation a little better?

[ghb2311]

Hello - I am confused by a similar issue on CentOS 7. The kernel-debug package is installed, which deposited /boot/vmlinuz-3.10.0-1160.53.1.el7.x86_64.debug. I uncompressed it with extract-vmlinux. Then:

dwarf2json linux --elf ./vmlinux-3.10.0-1160.53.1.el7.x86_64.debug > 3.10.0-1160.53.1.el7.x86_64.json
Failed linux processing: could not get DWARF from ./vmlinux-3.10.0-1160.53.1.el7.x86_64.debug: decoding dwarf section info at offset 0x0: too short

That seems to be telling me that this is not a debug kernel. Possibly telling is that this file is a lot smaller than I expected:

# ls -al vmlinux-3.10.0-1160.53.1.el7.x86_64.debug 
-rw-r--r--. 1 root root 57900928 Nov  8 19:33 vmlinux-3.10.0-1160.53.1.el7.x86_64.debug

What am I missing? Maybe despite what it says, it is not really a debug kernel?

Thank you.

[phillip-at-work]

I have the same issue. In my case, I confirm it's an elf with symbols: file vmlinux vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=995510549c37c0a1498fd305bc6b3585401d89e3, not stripped

But I receive the same error as others: (volatility) indigo-mac-0000:dwarf2json indigo$ ./dwarf2json linux --elf "/Users/indigo/Desktop/tmp/vmlinux" > result.json Failed linux processing: could not get DWARF from /Users/indigo/Desktop/tmp/vmlinux: decoding dwarf section info at offset 0x0: too short

FWIW, gdb also can't see symbols in this elf: # gdb ./vmlinux ... Reading symbols from ./vmlinux... (No debugging symbols found in ./vmlinux)

Any feedback is appreciated.