CONFIG_DEBUG_INFO_BTF and targets with many sections

joe-lawrence commented 3 years ago

Yulia hit this kpatch-build problem with simple kpatch-patch test and an internal kernel build:

...
  BTF     .btf.vmlinux.bin.o
scripts/link-vmlinux.sh: line 127: 718345 Segmentation fault      (core dumped) LLVM_OBJCOPY=${OBJCOPY} ${PAHOLE} -J ${1}
objcopy: --change-section-vma .BTF=0x0000000000000000 never used
objcopy: --change-section-lma .BTF=0x0000000000000000 never used
objcopy: error: the input file '.btf.vmlinux.bin' is empty
Failed to generate BTF for vmlinux
Try to disable CONFIG_DEBUG_INFO_BTF
make: *** [Makefile:1050: vmlinux] Error 1

Jiri Olsa noted the large number of vmlinux ELF sections and that pahole would need to be enhanced to support that:

AFAICS the generated vmlinux file has many sections:

  Number of section headers:         0 (101720)
  Section header string table index: 65535 (101719)

and that requires special handling when reading the elf object,
which pahole does not support at the moment

TODO:

Which upstream kernel versions are affected?
Which distro kernels
Did we only encounter this because we /just/ spilled over the non-extended limit?

euspectre commented 3 years ago

I've also got such error when building a patch for Virtuozzo kernel 4.18.0-240.1.1.vz8.5.4. The kernel is based on 4.18.0-240.1.1.el8_3 from RHEL 8.3, so, I suspect the kernel from RHEL might be affected too.

  LD      vmlinux.o
  MODPOST vmlinux.o
  BTF     .btf.vmlinux.bin.o
scripts/link-vmlinux.sh: line 127: 177631 Segmentation fault      (core dumped) LLVM_OBJCOPY=${OBJCOPY} ${PAHOLE} -J ${1}
objcopy: --change-section-vma .BTF=0x0000000000000000 never used
objcopy: --change-section-lma .BTF=0x0000000000000000 never used
objcopy: error: the input file '.btf.vmlinux.bin' is empty
Failed to generate BTF for vmlinux
Try to disable CONFIG_DEBUG_INFO_BTF
make: *** [Makefile:1065: vmlinux] Error 1

joe-lawrence commented 3 years ago

@euspectre : thanks for the report. We haven't seen this bug with any 8.3 based patch (yet), so perhaps a little additional code that Virtuozzo provides was enough to hit the bug.

Jiri has a fix in the works for pahole (follow along on the bpf list: https://lore.kernel.org/bpf/), but we probably want to do something in kpatch-build to handle potential cases:

@sm00th and @julien-thierry floated the idea of stopping the kpatch kernel builds just short of vmlinux (and the BTF step). I tried a make vmlinux.o, but kbuild doesn't accept that as a valid target. Other ideas welcome.
Turning off CONFIG_DEBUG_INFO_BTF for the kpatch kernel builds could side step the issue, but then it turns off some code in the bpf subsystem, not ideal.
Kpatch builds for smaller kernels should succeed, so perhaps we just detect vulnerable config + pahole versions and add a kpatch-build warning referring to this issue in case of build problems.

Thoughts?

julien-thierry commented 3 years ago

Jiri has a fix in the works for pahole (follow along on the bpf list: https://lore.kernel.org/bpf/), but we probably want to do something in kpatch-build to handle potential cases:
* @sm00th and @julien-thierry floated the idea of stopping the kpatch kernel builds just short of `vmlinux` (and the BTF step).  I tried a `make vmlinux.o`, but kbuild doesn't accept that as a valid target.  Other ideas welcome.

Before starting the original build, we could patch the root makefile of the kernel tree adding a specific target to build only what we need. Something like:

kpatch-target: $(vmlinux-deps)

And then have kpatch-build build the kpatch-target.

julien-thierry commented 3 years ago

kpatch-target: $(vmlinux-deps)

Or, less prone to patching errors, we could add a Makefile.kpatch file which includes the main Makefile and adds the kpatch-target. Then do kpatch-build builds using the Makefile.kpatch instead of Makefile.

jpoimboe commented 3 years ago

But then I assume this would break 'kpatch-build -t vmlinux' for example? Also link-vmlinux.sh generates a lot of files, are any of them needed by later build steps like building modules? Or by kpatch-build itself?

How about sed -i s/CONFIG_DEBUG_INFO_BTF/NOPE/ scripts/link-vmlinux.s. That should disable BTF only at vmlinux link time, and should be harmless? Then we're not mucking with actual makefiles and targets.

It could be done unconditionally, or if we're fancy, only under the detected error conditions: large # of symbols in original vmlinux, CONFIG_DEBUG_INFO_BTF, and old version of pahole.

sm00th commented 3 years ago

How about sed -i s/CONFIG_DEBUG_INFO_BTF/NOPE/ scripts/link-vmlinux.s. That should disable BTF only at vmlinux link time, and should be harmless? Then we're not mucking with actual makefiles and targets.

Great idea. If we are messing with kernel sources (and I currently don't see a way to not do this) this will be much more robust and future-proof.

joe-lawrence commented 3 years ago

How about sed -i s/CONFIG_DEBUG_INFO_BTF/NOPE/ scripts/link-vmlinux.s. That should disable BTF only at vmlinux link time, and should be harmless? Then we're not mucking with actual makefiles and targets.

Great idea. If we are messing with kernel sources (and I currently don't see a way to not do this) this will be much more robust and future-proof.

Agreed, I'll rework my original avoidance to backup link-vmlinux.sh and make @jpoimboe 's suggested tweak.

dynup / kpatch

CONFIG_DEBUG_INFO_BTF and targets with many sections #1153