Closed vincentbernat closed 5 years ago
Hi @vincentbernat,
The kpatch-build script only exports KCFLAGS="-I$DATADIR/patch -ffunction-sections -fdata-sections"
, the rest of the kernel flags and options should be coming from the unpacked kernel source. With that in mind, the script does a little bit of CONFIG_...
checking to ensure a couple features like CONFIG_DEBUG_KERNEL
are set. Could you attach the kernel .config
for which you are trying to build and if possible, the .patch file as well?
As far as symbol correlation is concerned, I don't believe that ordering is an issue. kpatch-build/lookup.c
has a bunch of logic, including string comparison, to try and associate original to patched symbols. However, the reported error message find_local_syms for file.c: found_none
implies that it can't find any local symbols in those object files.
Can you repeat the build with kpatch-build --skip-cleanup ...
and run readelf
on the object files in /root/.kpatch/tmp/patched
?
Thanks!
Hey!
Thanks for the fast answer. I am using this patch. And the .config
: config-3.13.0-117-generic.txt.
The readelf -s
output for the non-patched flow.o
:
43: 0000000000000000 0 FILE LOCAL DEFAULT ABS flow.c
44: 0000000000000000 113 FUNC LOCAL DEFAULT 2 flow_cache_new_hashrnd
45: 0000000000000000 48 FUNC LOCAL DEFAULT 4 flow_cache_flush_per_cpu
46: 0000000000000000 181 FUNC LOCAL DEFAULT 6 flow_cache_cpu_prepare.is
47: 0000000000000000 314 FUNC LOCAL DEFAULT 10 flow_cache_flush_tasklet
48: 0000000000000000 263 FUNC LOCAL DEFAULT 8 flow_cache_queue_garbage.
49: 0000000000000000 4 OBJECT LOCAL DEFAULT 51 flow_cache_gc_lock
50: 0000000000000000 16 OBJECT LOCAL DEFAULT 44 flow_cache_gc_list
51: 0000000000000000 8 OBJECT LOCAL DEFAULT 43 flow_cachep
52: 0000000000000000 329 FUNC LOCAL DEFAULT 12 __flow_cache_shrink.isra.
53: 0000000000000000 118 FUNC LOCAL DEFAULT 14 flow_cache_cpu
54: 0000000000000000 128 OBJECT LOCAL DEFAULT 50 flow_cache_global
55: 0000000000000000 88 OBJECT LOCAL DEFAULT 38 CSWTCH.63
56: 0000000000000000 410 FUNC LOCAL DEFAULT 18 flow_cache_init_global
57: 0000000000000000 4 OBJECT LOCAL DEFAULT 52 flow_flush_lock.27780
58: 0000000000000000 16 FUNC LOCAL DEFAULT 22 flow_cache_flush_task
59: 0000000000000000 32 OBJECT LOCAL DEFAULT 46 flow_cache_flush_work
60: 0000000000000000 8 OBJECT LOCAL DEFAULT 41 __initcall_flow_cache_ini
61: 0000000000000000 18 OBJECT LOCAL DEFAULT 39 __kstrtab_flow_cache_look
62: 0000000000000000 8 OBJECT LOCAL DEFAULT 32 __kcrctab_flow_cache_look
63: 0000000000000012 17 OBJECT LOCAL DEFAULT 39 __kstrtab_flow_cache_geni
64: 0000000000000000 8 OBJECT LOCAL DEFAULT 36 __kcrctab_flow_cache_geni
And the patched flow.o
:
43: 0000000000000000 0 FILE LOCAL DEFAULT ABS flow.c
44: 0000000000000000 113 FUNC LOCAL DEFAULT 2 flow_cache_new_hashrnd
45: 0000000000000000 48 FUNC LOCAL DEFAULT 4 flow_cache_flush_per_cpu
46: 0000000000000000 181 FUNC LOCAL DEFAULT 6 flow_cache_cpu_prepare.is
47: 0000000000000000 314 FUNC LOCAL DEFAULT 10 flow_cache_flush_tasklet
48: 0000000000000000 263 FUNC LOCAL DEFAULT 8 flow_cache_queue_garbage.
49: 0000000000000000 4 OBJECT LOCAL DEFAULT 51 flow_cache_gc_lock
50: 0000000000000000 16 OBJECT LOCAL DEFAULT 44 flow_cache_gc_list
51: 0000000000000000 8 OBJECT LOCAL DEFAULT 43 flow_cachep
52: 0000000000000000 329 FUNC LOCAL DEFAULT 12 __flow_cache_shrink.isra.
53: 0000000000000000 118 FUNC LOCAL DEFAULT 14 flow_cache_cpu
54: 0000000000000000 128 OBJECT LOCAL DEFAULT 50 flow_cache_global
55: 0000000000000000 88 OBJECT LOCAL DEFAULT 38 CSWTCH.63
56: 0000000000000000 410 FUNC LOCAL DEFAULT 18 flow_cache_init_global
57: 0000000000000000 4 OBJECT LOCAL DEFAULT 52 flow_flush_lock.27780
58: 0000000000000000 16 FUNC LOCAL DEFAULT 22 flow_cache_flush_task
59: 0000000000000000 32 OBJECT LOCAL DEFAULT 46 flow_cache_flush_work
60: 0000000000000000 8 OBJECT LOCAL DEFAULT 41 __initcall_flow_cache_ini
61: 0000000000000000 18 OBJECT LOCAL DEFAULT 39 __kstrtab_flow_cache_look
62: 0000000000000000 8 OBJECT LOCAL DEFAULT 32 __kcrctab_flow_cache_look
63: 0000000000000012 17 OBJECT LOCAL DEFAULT 39 __kstrtab_flow_cache_geni
64: 0000000000000000 8 OBJECT LOCAL DEFAULT 36 __kcrctab_flow_cache_geni
No diff. Even without grep
.
Maybe my libelf is too old (0.152-1ubuntu3.1). I can try to walk find_local_syms()
to spot the problem.
OK, I have walked a bit with gdb and here is what happens. find_local_syms
is called with the symbol table from /usr/lib/debug/boot/vmlinux-3.13.0-117-generic
(so, it doesn't match what's compiled). It finds the STT_FILE flow.c
. Then, the first LOCAL, FUNC symbol it finds is flow_cache_gc_task
. It compares it to the first symbol in child_locals
which is flow_cache_new_hashrnd
. It doesn't match, it considers that it is not in flow.c
(in_file=0
) anymore and continue without any more success (as there is no more STT_FILE flow.c
), so no symbol found.
So:
I am using the same compiler than Ubuntu used to compile the package. I suppose there is some difference in the options. Or using -ffunction-sections
or -fdata-sections
make a difference in the generated code with my version of gcc (which is quite old).
Breakpoint 2, find_local_syms (table=0x6b2a80, hint=0x7ffff7fef061 "flow.c", child_locals=0x6b2910)
at lookup.c:100
100 file_sym = sym;
2: *sym = {value = 0, size = 0, name = 0x8c68e0 "flow.c", type = 4, bind = 0, skip = 0}
1: sym = (struct object_symbol *) 0x7fffed2ab208
123 !strcmp(child_sym->name, sym->name))
2: *child_sym = {name = 0x7ffff7fef068 "flow_cache_new_hashrnd", type = 2}
1: *sym = {value = 18446744071585605040, size = 191, name = 0x8c6900 "flow_cache_gc_task", type = 2,
bind = 0, skip = 0}
126 in_file = 0;
2: *child_sym = {name = 0x7ffff7fef068 "flow_cache_new_hashrnd", type = 2}
1: *sym = {value = 18446744071585605040, size = 191, name = 0x8c6900 "flow_cache_gc_task", type = 2,
bind = 0, skip = 0}
Trying to repro here on RHEL7 and it looks like I hit a different build program with that patch:
/usr/local/libexec/kpatch/create-diff-object: ERROR: mpls_gso.o: find_local_syms: 136: find_local_syms for mpls_gso.c: found_none
I hope to take a better look after lunch :)
If you only have this one, it's unlikely that's the same problem. I get 170+ errors like this on my build.
@vincentbernat -- I think I've chased down my build bug to an unrelated matter. As far as yours is concerned, after running through find_local_syms
in gdb myself, it looks like its trying to compare the reference kernel-object with a rebuilt, unpatched object file. And I think you're correct in that it wants to see similar symbol ordering of both tables.
On my RHEL7 setup, I compared this file's entries from my vmlinux file:
readelf --wide -s $VMLINUX | awk '/FILE/{out=0} /FILE.*flow\.c/{out=1} !/GLOBAL/{ if(out) print $0 }'
to the ones generated in ~/.kpatch/tmp/orig
and they were indeed the same. How different do the entries for flow.c look in your reference vmlinux file?
I wouldn't think that a gcc patch to implement -ffunction-sections
would change these values, but without knowing much about gcc, it's possible. What happens if you rebuild the kernel source rpm with the patched gcc and attempt to kpatch against that one?
The GCC patch is this one, so it's not really a patch to implement anything. The output of readelf -s
for the original vmlinux
is in the original comment.
I'll rebuild the source package and I'll check if there is some special flags provided during the build.
The output of readelf -s for the original vmlinux is in the original comment.
By "reference vmlinux", I meant /usr/lib/debug/boot/vmlinux-3.13.0-117-generic
. That's the one that find_local_syms
is comparing against, right? Just curious to see how different it is.
Can you edit/annotate the readelf
examples in the previous comments to indicate which vmlinux and which flow.o is being listed? (Full command-line or paths would be sufficient.)
If I understand the bug report correctly, the ordering of the symbols for flow.c is different in /usr/lib/debug/boot/vmlinux-3.13.0-117-generic
than it is when kpatch-build
recreates its original, unpatched build (only adding the $KCFLAGS
listed above.) The symbol ordering is however, consistent between kpatch-build
original and patched versions of flow.o. Does that sound correct?
Recompiled and patched flow.o
have the same symbols in the same order. Only /usr/lib/debug/boot/vmlinux
doesn't match. I'll update the other comments like you suggested.
@vincentbernat - in the first comment, is the output from readelf -s /usr/lib/debug/boot/vmlinux-3.13.0-117-generic
correct? (I don't see the flow_cache_gc_task
symbol that you reported as missing. If I understand the problem, isn't the reference flow.c FILE vmlinux symbol table unique to its counterparts found in the kpatch-built flow.o and patched flow.o?)
As for next steps, I might have a look at why the compiler is generating different symbol tables. Is it possible that vmlinux-3.13.0-117-generic
was built by a distro-specific gcc? (ie, includes other changes that your patched gcc doesn't.) @jpoimboe @flaming-toast any other ideas?
@joe-lawrence oh, it seems I have messed the copy paste. I have corrected it with the actual output.
As for the gcc used, the version is present in dmesg and matches mine. I have taken the same sources (with apt-get source
) and added the patch I mentioned and recompiled. I didn't get time yet to just try to recompile the kernel package without any change and check what the debug vmlinux looks like. Feel free to not investigate too much in the mean time.
The only thing I can think of is that the package build might be adding compiler flags on the command line beyond what the makefiles do.
Here is the command-line used by kpatch:
gcc -Wp,-MD,net/core/.flow.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/4.6/include -I/home/ubuntu/.kpatch/src/arch/x86/include -Iarch/x86/include/generated -Iinclude -I/home/ubuntu/.kpatch/src/arch/x86/include/uapi -Iarch/x86/include/generated/uapi -I/home/ubuntu/.kpatch/src/include/uapi -Iinclude/generated/uapi -include /home/ubuntu/.kpatch/src/include/linux/kconfig.h -Iubuntu/include -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -std=gnu89 -O2 -m64 -mno-mmx -mno-sse -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -Wframe-larger-than=1024 -fstack-protector -Wno-unused-but-set-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-var-tracking-assignments -g -pg -mfentry -DCC_USING_FENTRY -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -DCC_HAVE_ASM_GOTO -I/home/ubuntu/kpatch/kmod/patch -ffunction-sections -fdata-sections '-DKBUILD_STR(s)=#s' '-DKBUILD_BASENAME=KBUILD_STR(flow)' '-DKBUILD_MODNAME=KBUILD_STR(flow)' -c -o net/core/.tmp_flow.o net/core/flow.c
And the command-line used during package build:
gcc -Wp,-MD,net/core/.flow.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/4.6/include -I/usr/src/linux-headers-lbm- -I/home/ubuntu/linux2/arch/x86/include -Iarch/x86/include/generated -I/home/ubuntu/linux2/include -Iinclude -I/home/ubuntu/linux2/arch/x86/include/uapi -Iarch/x86/include/generated/uapi -I/home/ubuntu/linux2/include/uapi -Iinclude/generated/uapi -include /home/ubuntu/linux2/include/linux/kconfig.h -Iubuntu/include -I/home/ubuntu/linux2/ubuntu/include -I/home/ubuntu/linux2/net/core -Inet/core -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -std=gnu89 -O2 -m64 -mno-mmx -mno-sse -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_AVX=1-DCONFIG_AS_AVX2=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -Wframe-larger-than=1024 -fstack-protector -Wno-unused-but-set-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-var-tracking-assignments -g -pg -mfentry -DCC_USING_FENTRY -fno-inline-functions-called-once -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -DCC_HAVE_ASM_GOTO -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(flow)" -D"KBUILD_MODNAME=KBUILD_STR(flow)" -c -o net/core/.tmp_flow.o /home/ubuntu/linux2/net/core/flow.c
The difference is -fno-inline-functions-called-once
added when doing the package build (and -ffunction-sections
and -fdata-sections
added by kpatch). This should be added only if CONFIG_DEBUG_SECTION_MISMATCH
is set and while it is unset in .config
, it is forcibly set by the make command:
make ARCH=x86_64 CROSS_COMPILE= KERNELVERSION=3.13.0-117-generic CONFIG_DEBUG_SECTION_MISMATCH=y KBUILD_BUILD_VERSION="164~precise1" LOCALVERSION= localver-extra= CFLAGS_MODULE="-DPKG_ABI=117" O=/home/ubuntu/linux2/debian/build/build-generic -j8 bzImage modules
What's the best way to ensure kpatch also compiles the kernel with the same command line?
My advice would be to patch the makefile to add -fno-inline-functions-called-once
instead of adding the option manually on the command line.
So, I have made some progress but I still have some modules failing.
From readelf -s /usr/lib/debug/.../vmlinux
:
59720: 0000000000000000 0 FILE LOCAL DEFAULT ABS tcp_cubic.c
59721: ffffffff816e5af0 106 FUNC LOCAL DEFAULT 1 bictcp_recalc_ssthresh
59722: ffffffff81d1ba10 4 OBJECT LOCAL DEFAULT 16 fast_convergence
59723: ffffffff81d1ba14 4 OBJECT LOCAL DEFAULT 16 beta
59724: ffffffff816e5b60 30 FUNC LOCAL DEFAULT 1 bictcp_undo_cwnd
59725: ffffffff81d914e7 106 FUNC LOCAL DEFAULT 19 cubictcp_register
59726: ffffffff81d1ba34 4 OBJECT LOCAL DEFAULT 16 bic_scale
59727: ffffffff81d1ba28 4 OBJECT LOCAL DEFAULT 16 cube_rtt_scale
59728: ffffffff81d1ba30 4 OBJECT LOCAL DEFAULT 16 beta_scale
59729: ffffffff81d1ba00 4 OBJECT LOCAL DEFAULT 16 hystart
59730: ffffffff81d1ba20 8 OBJECT LOCAL DEFAULT 16 cube_factor
59731: ffffffff81d1b980 128 OBJECT LOCAL DEFAULT 16 cubictcp
59732: ffffffff81e72d7d 18 FUNC LOCAL DEFAULT 27 cubictcp_unregister
59733: ffffffff816e5b80 705 FUNC LOCAL DEFAULT 1 bictcp_cong_avoid.part.3
59734: ffffffff81d1ba2c 4 OBJECT LOCAL DEFAULT 16 tcp_friendliness
59735: ffffffff818a3160 64 OBJECT LOCAL DEFAULT 4 v.49260
59736: ffffffff816e5e50 259 FUNC LOCAL DEFAULT 1 bictcp_cong_avoid
59737: ffffffff816e5f60 336 FUNC LOCAL DEFAULT 1 hystart_update
59738: ffffffff81d1ba08 4 OBJECT LOCAL DEFAULT 16 hystart_detect
59739: ffffffff81d1ba0c 4 OBJECT LOCAL DEFAULT 16 hystart_ack_delta
59740: ffffffff816e60b0 217 FUNC LOCAL DEFAULT 1 bictcp_acked
59741: ffffffff81d1ba04 4 OBJECT LOCAL DEFAULT 16 hystart_low_window
59742: ffffffff816e6190 287 FUNC LOCAL DEFAULT 1 bictcp_init
59743: ffffffff81d1ba18 4 OBJECT LOCAL DEFAULT 16 initial_ssthresh
59744: ffffffff816e62b0 249 FUNC LOCAL DEFAULT 1 bictcp_state
59745: ffffffff81b6e778 8 OBJECT LOCAL DEFAULT 15 __modver_attr
59746: ffffffff81ce1b40 72 OBJECT LOCAL DEFAULT 16 ___modver_attr
59747: ffffffff81e597c8 8 OBJECT LOCAL DEFAULT 20 __initcall_cubictcp_regis
59748: ffffffff81b6e528 32 OBJECT LOCAL DEFAULT 14 __param_hystart_ack_delta
59749: ffffffff818a31a0 28 OBJECT LOCAL DEFAULT 4 __param_str_hystart_ack_d
59750: ffffffff81b6e548 32 OBJECT LOCAL DEFAULT 14 __param_hystart_low_windo
59751: ffffffff818a31c0 29 OBJECT LOCAL DEFAULT 4 __param_str_hystart_low_w
59752: ffffffff81b6e568 32 OBJECT LOCAL DEFAULT 14 __param_hystart_detect
59753: ffffffff818a31e0 25 OBJECT LOCAL DEFAULT 4 __param_str_hystart_detec
59754: ffffffff81b6e588 32 OBJECT LOCAL DEFAULT 14 __param_hystart
59755: ffffffff818a3200 18 OBJECT LOCAL DEFAULT 4 __param_str_hystart
59756: ffffffff81b6e5a8 32 OBJECT LOCAL DEFAULT 14 __param_tcp_friendliness
59757: ffffffff818a3220 27 OBJECT LOCAL DEFAULT 4 __param_str_tcp_friendlin
59758: ffffffff81b6e5c8 32 OBJECT LOCAL DEFAULT 14 __param_bic_scale
59759: ffffffff818a3240 20 OBJECT LOCAL DEFAULT 4 __param_str_bic_scale
59760: ffffffff81b6e5e8 32 OBJECT LOCAL DEFAULT 14 __param_initial_ssthresh
59761: ffffffff818a3260 27 OBJECT LOCAL DEFAULT 4 __param_str_initial_ssthr
59762: ffffffff81b6e608 32 OBJECT LOCAL DEFAULT 14 __param_beta
59763: ffffffff818a327b 15 OBJECT LOCAL DEFAULT 4 __param_str_beta
59764: ffffffff81b6e628 32 OBJECT LOCAL DEFAULT 14 __param_fast_convergence
59765: ffffffff818a3290 27 OBJECT LOCAL DEFAULT 4 __param_str_fast_converge
And the non-patched version:
1: 0000000000000000 0 FILE LOCAL DEFAULT ABS tcp_cubic.c
6: 0000000000000000 106 FUNC LOCAL DEFAULT 4 bictcp_recalc_ssthresh
7: 0000000000000090 4 OBJECT LOCAL DEFAULT 32 fast_convergence
8: 0000000000000094 4 OBJECT LOCAL DEFAULT 32 beta
10: 0000000000000000 30 FUNC LOCAL DEFAULT 6 bictcp_undo_cwnd
12: 0000000000000000 106 FUNC LOCAL DEFAULT 8 cubictcp_register
13: 00000000000000b4 4 OBJECT LOCAL DEFAULT 32 bic_scale
14: 00000000000000a8 4 OBJECT LOCAL DEFAULT 32 cube_rtt_scale
15: 00000000000000b0 4 OBJECT LOCAL DEFAULT 32 beta_scale
16: 0000000000000080 4 OBJECT LOCAL DEFAULT 32 hystart
17: 00000000000000a0 8 OBJECT LOCAL DEFAULT 32 cube_factor
18: 0000000000000000 128 OBJECT LOCAL DEFAULT 32 cubictcp
20: 0000000000000000 18 FUNC LOCAL DEFAULT 10 cubictcp_unregister
22: 0000000000000000 705 FUNC LOCAL DEFAULT 12 bictcp_cong_avoid.part.3
23: 00000000000000ac 4 OBJECT LOCAL DEFAULT 32 tcp_friendliness
24: 0000000000000000 64 OBJECT LOCAL DEFAULT 34 v.49260
26: 0000000000000000 259 FUNC LOCAL DEFAULT 14 bictcp_cong_avoid
28: 0000000000000000 336 FUNC LOCAL DEFAULT 16 hystart_update
29: 0000000000000088 4 OBJECT LOCAL DEFAULT 32 hystart_detect
30: 000000000000008c 4 OBJECT LOCAL DEFAULT 32 hystart_ack_delta
32: 0000000000000000 217 FUNC LOCAL DEFAULT 18 bictcp_acked
33: 0000000000000084 4 OBJECT LOCAL DEFAULT 32 hystart_low_window
35: 0000000000000000 287 FUNC LOCAL DEFAULT 20 bictcp_init
36: 0000000000000098 4 OBJECT LOCAL DEFAULT 32 initial_ssthresh
38: 0000000000000000 249 FUNC LOCAL DEFAULT 22 bictcp_state
40: 0000000000000000 8 OBJECT LOCAL DEFAULT 24 __modver_attr
41: 0000000000000000 72 OBJECT LOCAL DEFAULT 36 ___modver_attr
43: 0000000000000000 8 OBJECT LOCAL DEFAULT 26 __exitcall_cubictcp_unreg
45: 0000000000000000 8 OBJECT LOCAL DEFAULT 28 __initcall_cubictcp_regis
47: 0000000000000000 32 OBJECT LOCAL DEFAULT 30 __param_hystart_ack_delta
48: 0000000000000000 28 OBJECT LOCAL DEFAULT 38 __param_str_hystart_ack_d
49: 0000000000000020 32 OBJECT LOCAL DEFAULT 30 __param_hystart_low_windo
50: 0000000000000000 29 OBJECT LOCAL DEFAULT 39 __param_str_hystart_low_w
51: 0000000000000040 32 OBJECT LOCAL DEFAULT 30 __param_hystart_detect
52: 0000000000000000 25 OBJECT LOCAL DEFAULT 40 __param_str_hystart_detec
53: 0000000000000060 32 OBJECT LOCAL DEFAULT 30 __param_hystart
54: 0000000000000000 18 OBJECT LOCAL DEFAULT 41 __param_str_hystart
55: 0000000000000080 32 OBJECT LOCAL DEFAULT 30 __param_tcp_friendliness
56: 0000000000000000 27 OBJECT LOCAL DEFAULT 42 __param_str_tcp_friendlin
57: 00000000000000a0 32 OBJECT LOCAL DEFAULT 30 __param_bic_scale
58: 0000000000000000 20 OBJECT LOCAL DEFAULT 43 __param_str_bic_scale
59: 00000000000000c0 32 OBJECT LOCAL DEFAULT 30 __param_initial_ssthresh
60: 0000000000000000 27 OBJECT LOCAL DEFAULT 44 __param_str_initial_ssthr
61: 00000000000000e0 32 OBJECT LOCAL DEFAULT 30 __param_beta
62: 0000000000000000 15 OBJECT LOCAL DEFAULT 45 __param_str_beta
63: 0000000000000100 32 OBJECT LOCAL DEFAULT 30 __param_fast_convergence
64: 0000000000000000 27 OBJECT LOCAL DEFAULT 46 __param_str_fast_converge
The only difference is that I am missing this line in /usr/lib/debug/.../vmlinux
:
43: 0000000000000000 8 OBJECT LOCAL DEFAULT 26 __exitcall_cubictcp_unreg
If I look at the final vmlinux
(compiled with the patch), the line is also missing. I suppose that the function is stripped because the .o
becomes "built-in". And I see this case is handled in lookup.c
. In my case, table->vmlinux
is 0, so the symbol is not discarded. The problem is that my debug vmlinux
is vmlinux-3.13.0-117-generic
, not vmlinux
. I'll cook a patch for this.
After applying #707, I still have 4 errors (one of them is mpls_gso.o
). I'll investigate them later.
So, the remaining problems for me are:
mpls_gso
, but also lloop.c
)vsprintf.o
, the one from /usr/lib/debug/boot/vmlinux
contains this additional symbol: 30046: ffffffff8138e2e0 9 FUNC LOCAL DEFAULT 1 copy_page_rep
Any idea for the last one? Maybe copy_page_64.S
got combined with vsprintf.o
at link time?
Hi @vincentbernat , I don't know if you're still plugging away at the building your patch, but I've collected a few Ubuntu fixes in my own tree for testing (in the aptly named "ubuntu_fixes" branch). See the commit log for an example usage of KPATCH_GCC_OBJ_IGNORE
to help avoid #701 and #708.
With the test branch and running Ubuntu 3.13.0-119-generic, I can build and load the meminfo_string test integration example.
However, the patch you attached in the third comment still has issues:
KPATCH_GCC_OBJ_IGNORE
pattern list to ignore anything in staging if you're not using any of those drivers.-fno-inline-functions-called-once
- do you happen to know if this is unique to Ubuntu 3.13 kernels? Not sure yet how to best auto-apply this build flag, if possible. It would be nice if kpatch-build
could do this automatically.__func__.53250
symbols ... I think this is issue #700. The patch creates a lot of changed functions, so the size increases the likelihood of hitting this bug.Anyway, just thought I'd point you to that (slow) work-in-progress. You might have better luck if you can modify the patch to pare down the number affected functions. Thanks for all the bug reports and testing!
Thanks for the heads up. Unfortunately, I don't currently have the time to debug more (and I have finally opted to reboot the affected kernels, so I don't have an immediate use of the patch). My next step was to modify the patch to not modify any .h, to put the code in a dedicated .h (with a different function name) and only include it in the .c files that need it.
As for -fno-inline-functions-called-once
, this is still used in Ubuntu 4.4 (in Xenial).
@vincentbernat @joe-lawrence Sorry to revive an old thread, but i've hit a similar problem (I think) and I wonder if I could benefit from your insights, having faced what seems to be a similar issue.
/usr/libexec/kpatch/create-diff-object: ERROR: meminfo.o: find_local_syms: 174: find_local_syms for meminfo.c: couldn't find in vmlinux symbol table
Even applying a trivial patch is causing me the same issue (such as the one in the readme):
---
fs/proc/meminfo.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
index 6bb20f864259..97a62c33c130 100644
--- a/fs/proc/meminfo.c
+++ b/fs/proc/meminfo.c
@@ -133,7 +133,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
seq_printf(m, "VmallocTotal: %8lu kB\n",
(unsigned long)VMALLOC_TOTAL >> 10);
show_val_kb(m, "VmallocUsed: ", 0ul);
- show_val_kb(m, "VmallocChunk: ", 0ul);
+ show_val_kb(m, "VMallocChunk: ", 0ul);
#ifdef CONFIG_MEMORY_FAILURE
seq_printf(m, "HardwareCorrupted: %5lu kB\n",
--
2.16.4
The patch I am actually tring to apply is this one against kernel sources that I have built myself fore ContainerOS (based on chromium, which is based on gentoo).
The specific kernel revision I am trying to patch is: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/e7d0ede66eb4efd21bf8935391cc2797f480c15f, and I'm trying to apply one patch that reverts https://chromium.googlesource.com/chromiumos/third_party/kernel/+/27f29dbceb3c979d00833a90aa27ff0756ecc1e0 and a second one that applies https://github.com/torvalds/linux/commit/f0a2aa5a2a406d0a57aa9b320ffaa5538672b6c5#diff-bea6197e6c8c9c3d726c6514ca7c5a03
I am running an instance in GKE where I want to apply the above patch to get more debugging visibility via uprobes, and literally just need to change the address of one function call to another per the patch.
However, I don't think it is the patch that is the problem - any patch I try to apply gives me similar errors.
I've been using 0.6.0 and 0.6.1 and get the same problem (only 0.6.1 gives me a better error message, indicating vmlinux specifically).
The only file that changes is trace_uprobe.o, but kpatch_build
fails as such.
Here is an excerpt from the vmlinux symbol table: https://gist.github.com/dalehamel/599d5e4bd3d34511fb6126382ca5ed7d
And here are the before and after symbol tables from the patch
orig: https://gist.github.com/dalehamel/65d5b0503a343e3e47f0c3435ccbb204 patched: https://gist.github.com/dalehamel/de6021f5bbafd5bce7e09f70d4912f54
I'm pretty stumped... does the error message indicate that trace_uprobe.c
is the symbol it is looking for in vmlinux, and can't find it? Or is it looking for all of the symbols from trace_uprobe.c? Does it only look for ones that changed?
It is also odd to me that all those values are 0'd out in the uprobe object file, but it's been a while since I've worked with the ELF format.
Any insights you can share to get me unstuck would be very appreciated
I've been invoking kpatch-build as such and getting the following output:
kpatch-build patch1.patch patch2.patch -c .config -v /vmlinux
-s /usr/src/linux -t vmlinux
Using source directory at /usr/src/kernel
Testing patch file(s)
Reading special section data
Building original kernel
Building patched kernel
Extracting new and modified ELF sections
/usr/libexec/kpatch/create-diff-object: ERROR: trace_uprobe.o: find_local_syms: 174: find_local_syms for trace_uprobe.c: couldn't find in vmlinux symbol table
ERROR: 1 error(s) encountered. Check /root/.kpatch/build.log for more details.
/vmlinux is where the kernel I want to patch is, but I get the same error trying to use the 'orig' kernel produced by just running make vmlinux
in the kernel source directory.
I have verified the GCC versions match exactly. The only weird thing I'm doing is running this on an unsupported distro ,but the only distro-specific logic seems to be downloading kernel sources and debug symbols.... but I'm generating everything from source.
It's trying to find the part of the symbol table in vmlinux which corresponds to the orig .o symbol table. It expects them to match up exactly. I think the problem is that vmlinux has the following symbols:
12481: ffffffff821051c8 1 OBJECT LOCAL DEFAULT 21 __warned.42167
12486: ffffffff82520388 0 OBJECT LOCAL DEFAULT 55 __key.41841
12495: ffffffff82520388 0 OBJECT LOCAL DEFAULT 55 __key.41683
whereas the orig.o has:
84: 0000000000000000 1 OBJECT LOCAL DEFAULT 135 __warned.42176
96: 0000000000000000 0 OBJECT LOCAL DEFAULT 133 __key.41850
121: 0000000000000000 0 OBJECT LOCAL DEFAULT 134 __key.41692
Notice the numbered suffixes differ. This indicates that the .o file built by kpatch-build was somehow compiled differently than the original vmlinux: maybe some different compiler flags where used? Or some subtle difference in the sources or compilers used.
You could hack around it by changing the strcmp logic in locals_match() to allow differences after the '.' character, but that would still be covering up a root issue of some compiled difference between the files. They should be identical.
Thanks for the response, that gives me a next step for where to look - go find out if the compiler flags are different, which I should be able to do by just turning up the verbosity of the build system.
I think the hackier solution may also work, as I don’t mind if more object code is replaced here so long as it is only for that file’s symbols.
I guess to check success, when compiling the reference kernel and my orig patches I should verify these numbers match exactly, that should mean I’ve hit the exact flags and compiler conditions I suppose.
If I only have kernel rpm file, how can I get the compiler flags for this kernel rpm file ?
@NKTelnet : I'm not sure if there is anyway to dig that out of a plain RPM file (maybe there are ELF sections saved into vmlinux, but I doubt it). The easiest way would be to find and build a source rpm of the same version, then you can inspect the .filename.o.cmd
files that are left behind.
As far as I can tell issue seems to be obsolete, or at least not going anywhere. Please open a new issue if you still have a problem. Thanks.
@NKTelnet : I'm not sure if there is anyway to dig that out of a plain RPM file (maybe there are ELF sections saved into vmlinux, but I doubt it). The easiest way would be to find and build a source rpm of the same version, then you can inspect the
.filename.o.cmd
files that are left behind.
Thanks a lot, Joe. I can not find source rpm, because it is a private built rpm.
Hey!
I am trying to patch a 3.13 kernel and I am running into some difficulties. I get a lot of errors during symbol comparison:
If I use
readelf -s /usr/lib/debug/boot/vmlinux-3.13.0-117-generic
and look forflow.c
, I see (| grep -E 'FILE|FUNC|OBJECT' | grep LOCAL
):If I do that on patched
flow.o
, I get:So, I have a few symbols missing (
flow_cache_gc_task
) and a slightly different ordering.Is the ordering important (I would say it is)? How does kpatch-build ensure the appropriate CFLAGS are used?
flow_cache_gc_task
is static, small with only one reference and may have been inlined in my build. Kernel is from Ubuntu.I am using gcc 4.6 with the appropriate patch to make
-ffunction-sections
work.