cross compiling issue, run time error

[maindoor]

I am trying to compile ply for arm64: I have my the cross compiled libraries like libc.so.c at /home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/lib/ and my cross compiled tool chain like aarch64-openwrt-linux-gnu-gcc at /home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/bin so I exported the PATH variable to the "bin" directory and configured it like this: ./configure --host=aarch64-openwrt-linux-gnu --with-kerneldir=/home/demera/proj/linux/ make gives a warning like this at the end, but builds the binary:

aarch64-openwrt-linux-gnu-gcc: warning: environment variable 'STAGING_DIR' not defined
  CCLD     ply
aarch64-openwrt-linux-gnu-gcc: warning: environment variable 'STAGING_DIR' not defined
aarch64-openwrt-linux-gnu-gcc: warning: environment variable 'STAGING_DIR' not defined
/home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/lib/gcc/aarch64-openwrt-linux-gnu/5.3.0/../../../../aarch64-openwrt-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
/home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/lib/gcc/aarch64-openwrt-linux-gnu/5.3.0/../../../../aarch64-openwrt-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc.so.6 when searching for libc.so.6
/home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/lib/gcc/aarch64-openwrt-linux-gnu/5.3.0/../../../../aarch64-openwrt-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc_nonshared.a when searching for libc_nonshared.a

But when I run the program on the target system, I get: EB:/home/demera# ./ply -c 'kprobe:netif_rx { printf("%v\n", stack()) }' ERR pvdr_resolve_call : in 'kprobe:netif_rx', unknown function 'stack'

I suspect the linker issue above, so my question is: How can I ask the build system to look for libraries under the cross compiled libraries ?

Thanks.

[maindoor]

I Solved the Linker problem like so: ./configure LDFLAGS="-L/home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/lib/" --host=aarch64-openwrt-linux-gnu --with-kerneldir=/home/demera/proj/linux/ Build was clean without any linker complaints.

But the run time problem remains: EB:/home/demera# ./ply -c 'kprobe:netif_rx { printf("%v\n", stack()) }' ERR pvdr_resolve_call : in 'kprobe:netif_rx', unknown function 'stack'

any suggestions ?

[wkz]

The output from your linker is probably something you want to fix. But I would try to reach out to someone in the OpenWRT community to get help with that.

That said, the output is from ply so you seem to have produced a working binary.

That output is from the old 1.x branch. Please build from the latest release: https://github.com/wkz/ply/releases

The iovisor repo was not in sync with my private repo. I have updated that now, sorry about that.

I've removed the --with-kerneldir option so you'll have to supply the directory where you've installed your kernel headers in CPPFLAGS when running configure if your toolchain does not know where to find them.

[maindoor]

Sorry about that my script by default was taking from iovisor repo, I've fixed that now to take from wiz/ply. After using the latest version I get the below error. Going through the kernel sources this BPF_JLT etc. was defined in newer kernels. Mine embedded system runs on 4.4.60. Is there a tag or an older version that you'll recommend ? Meanwhile I'll see if I can port the BPF_JLT specific code from newer kernel to my older kernel. Do you think this is a good idea ? I'm new to this so I am doing all I can to get this working. Thanks for your support.

aarch64-openwrt-linux-gnu-gcc: warning: environment variable 'STAGING_DIR' not defined
  CC       built-in/libply_la-math.lo
aarch64-openwrt-linux-gnu-gcc: warning: environment variable 'STAGING_DIR' not defined
In file included from /home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/include/linux/bpf.h:11:0,
                 from ../../include/ply/ir.h:14,
                 from ../../include/ply/sym.h:13,
                 from ../../include/ply/ply.h:12,
                 from built-in/math.c:11:
built-in/math.c: In function 'relop_ir_post':
built-in/math.c:253:32: error: 'BPF_JLT' undeclared (first use in this function)
   jmp.code |= BPF_OP(unsignd ? BPF_JLT : BPF_JSLT);
                                ^
built-in/math.c:253:32: note: each undeclared identifier is reported only once for each function it appears in
built-in/math.c:253:42: error: 'BPF_JSLT' undeclared (first use in this function)
   jmp.code |= BPF_OP(unsignd ? BPF_JLT : BPF_JSLT);
                                          ^
built-in/math.c:257:42: error: 'BPF_JSLE' undeclared (first use in this function)
   jmp.code |= BPF_OP(unsignd ? BPF_JLE : BPF_JSLE);

[wkz]

I think that would be very difficult. Try the following patch:

diff --git a/include/ply/ir.h b/include/ply/ir.h
index 72d208d..8c6e025 100644
--- a/include/ply/ir.h
+++ b/include/ply/ir.h
@@ -19,6 +19,9 @@
 #endif

 /* TODO: TEMP workaround for old headers */
+#ifndef BPF_JLT
+#define BPF_JLT                0xa0    /* LT is unsigned, '<' */
+#endif
 #ifndef BPF_JLE
 #define BPF_JLE                0xb0    /* LE is unsigned, '<=' */
 #endif

It should allow you to build and, as long as you avoid these operators, everything should work.

[maindoor]

Hi wiz, Once I fix that I get the following error:

aarch64-openwrt-linux-gnu-gcc: warning: environment variable 'STAGING_DIR' not defined
In file included from ../../include/ply/sym.h:13:0,
                 from ../../include/ply/ply.h:12,
                 from built-in/memory.c:10:
built-in/memory.c: In function 'str_ir_post':
built-in/memory.c:143:24: error: 'BPF_FUNC_probe_read_str' undeclared (first use in this function)
  ir_emit_insn(ir, CALL(BPF_FUNC_probe_read_str), 0, 0);
                        ^

Do you have suggestions on how to overcome this error ?

[wkz]

I just looked through the kernel history and I think you're going to keep bumping in to issues with missing functionality. I think anything before 4.9 will be tough on 2.x.

Alternatively you could go with 1.x, but features will be missing (like stack traces, which integrated into Linux in 4.6).

[maindoor]

Hi Wiz, I managed to back port some kernel features and compile ply successfully. But the instructions to install on target does not work for me. I can't do a "make install' on the host. I installed on a dummy folder and saw it has shared libraries and executable and header files. Can you give me instructions on how to install this on my target ?

Thanks.