ARM cross compiler/linker cannot locate certain libraries

[geneva1]

ARM cross compiler/linker cannot locate certain libraries for https://github.com/irontec/netfilter-nfqueue-samples, but normal system-wide gcc compiler/linker can.

Why ?

dell@dell-hp:~/Downloads/netfilter-nfqueue-samples$ gcc -o sample-helloworld sample-helloworld.c -lnfnetlink -lnetfilter_queue -lpthread -lm -ldl -I/usr/include/ -I/usr/include/x86_64-linux-gnu/
sample-helloworld.c: In function ‘print_pkt’:
sample-helloworld.c:55:28: warning: passing argument 2 of ‘nfq_get_payload’ from incompatible pointer type [-Wincompatible-pointer-types]
   55 |  ret = nfq_get_payload(tb, &data);
      |                            ^~~~~
      |                            |
      |                            char **
In file included from sample-helloworld.c:7:
/usr/include/libnetfilter_queue/libnetfilter_queue.h:119:67: note: expected ‘unsigned char **’ but argument is of type ‘char **’
  119 | extern int nfq_get_payload(struct nfq_data *nfad, unsigned char **data);
      |                                                   ~~~~~~~~~~~~~~~~^~~~
dell@dell-hp:~/Downloads/netfilter-nfqueue-samples$ ls
LICENSE  README.md  sample-helloworld  sample-helloworld.c  sample-memcache.c  sample-mysql.c
dell@dell-hp:~/Downloads/netfilter-nfqueue-samples$ rm sample-helloworld
dell@dell-hp:~/Downloads/netfilter-nfqueue-samples$ /home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc -o sample-helloworld sample-helloworld.c -lnfnetlink -lnetfilter_queue -lpthread -lm -ldl -I/usr/include/ -I/usr/include/x86_64-linux-gnu/
sample-helloworld.c: In function ‘print_pkt’:
sample-helloworld.c:55:28: warning: passing argument 2 of ‘nfq_get_payload’ from incompatible pointer type [-Wincompatible-pointer-types]
  ret = nfq_get_payload(tb, &data);
                            ^~~~~
In file included from sample-helloworld.c:7:
/usr/include/libnetfilter_queue/libnetfilter_queue.h:119:67: note: expected ‘unsigned char **’ but argument is of type ‘char **’
 extern int nfq_get_payload(struct nfq_data *nfad, unsigned char **data);
                                                   ~~~~~~~~~~~~~~~~^~~~
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnfnetlink
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnetfilter_queue
collect2: error: ld returned 1 exit status
dell@dell-hp:~/Downloads/netfilter-nfqueue-samples$

[geneva1]

@etienne-lms you had only pinpointed the warning, but what about the 3 linker errors below :

/home/rog/optee_os/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnetfilter_queue
/home/rog/optee_os/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnfnetlink
/home/rog/optee_os/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lmnl

rog@rog-GL502VMK:~/optee_os/build$ cat common.mk | grep NET
BR2_PACKAGE_LIBNFNETLINK ?= y
BR2_PACKAGE_LIBNETFILTER_QUEUE ?= y
rog@rog-GL502VMK:~/optee_os/build$ 

[etienne-lms]

From the line you show, you compile manually sample-helloworld.c:

rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ /home/rog/optee_os/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc -o sample-helloworld sample-helloworld.c -lpthread -lm -ldl -I/usr/include/ -I/usr/include/x86_64-linux-gnu/ -L/home/rog/optee_os/out-br/target/usr/lib -lnetfilter_queue -lnfnetlink -lmnl

So you don't rely here on common.mk and other makefiles from OP-TEE/build/. If you do so, you need to provide path the several libraries used (i.e. netfilter_queue, nfnetlink, ...: can do with -Lpath/to/libnetfilter_queue, ...

Another way is that you create OP-TEE/build/br-ext/package/netfilter-nfqueue-samples/ (check how other OP-TEE/build/br-ext/package/* are defined) and let OP-TEE generic build get it built from Buildroot.

[geneva1]

So you don't rely here on common.mk and other makefiles from OP-TEE/build/.

No, I have already done make all inside OP-TEE/build which includes common.mk

So, why /home/rog/optee_os/out-br/target/usr/lib still does not have the compiled libraries (netfilter_queue, nfnetlink) ?

rog@rog-GL502VMK:~/optee_os/build$ cat common.mk | grep NET
BR2_PACKAGE_LIBNFNETLINK ?= y
BR2_PACKAGE_LIBNETFILTER_QUEUE ?= y
rog@rog-GL502VMK:~/optee_os/build$ 

[etienne-lms]

I have added BR2_PACKAGE_LIBNFNETLINK ?= y and BR2_PACKAGE_LIBNETFILTER_QUEUE ?= y in my common.mk, ran make buildroot (or make all) and found the expected libraries::

$ ls -go ../out-br/target/usr/lib/
(...)
lrwxrwxrwx 1    15 Jun 19 12:30 ../out-br/target/usr/lib/libmnl.so -> libmnl.so.0.2.0
lrwxrwxrwx 1    15 Jun 19 12:30 ../out-br/target/usr/lib/libmnl.so.0 -> libmnl.so.0.2.0
-rwxr-xr-x 1 13736 Jun 19 12:31 ../out-br/target/usr/lib/libmnl.so.0.2.0
lrwxrwxrwx 1    27 Jun 19 12:30 ../out-br/target/usr/lib/libnetfilter_queue.so -> libnetfilter_queue.so.1.4.0
lrwxrwxrwx 1    27 Jun 19 12:30 ../out-br/target/usr/lib/libnetfilter_queue.so.1 -> libnetfilter_queue.so.1.4.0
-rwxr-xr-x 1 13384 Jun 19 12:31 ../out-br/target/usr/lib/libnetfilter_queue.so.1.4.0
lrwxrwxrwx 1    21 Jun 19 12:30 ../out-br/target/usr/lib/libnfnetlink.so -> libnfnetlink.so.0.2.0
lrwxrwxrwx 1    21 Jun 19 12:30 ../out-br/target/usr/lib/libnfnetlink.so.0 -> libnfnetlink.so.0.2.0
-rwxr-xr-x 1 17392 Jun 19 12:31 ../out-br/target/usr/lib/libnfnetlink.so.0.2.0
(...)
$ 

[geneva1]

@etienne-lms I followed your advice above (make buildroot instead of make all which just made the host OS hangs forever) and I had the compiled libraries, but it seems to me that the compiled library libnetfilter_queue.so is not compiled properly or the library system architecture is not recognized by the cross-compiler arm-linux-gnueabihf-gcc

Here is full log output of make buildroot

rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ /home/rog/optee_os/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc -o sample-helloworld sample-helloworld.c -lpthread -lm -ldl -I/usr/include/ -I/usr/include/x86_64-linux-gnu/ -L/home/rog/optee_os/out-br/target/usr/lib -lnetfilter_queue -lnfnetlink -lmnl
sample-helloworld.c: In function ‘print_pkt’:
sample-helloworld.c:55:28: warning: passing argument 2 of ‘nfq_get_payload’ from incompatible pointer type [-Wincompatible-pointer-types]
  ret = nfq_get_payload(tb, &data);
                            ^~~~~
In file included from sample-helloworld.c:7:
/usr/include/libnetfilter_queue/libnetfilter_queue.h:119:67: note: expected ‘unsigned char **’ but argument is of type ‘char **’
 extern int nfq_get_payload(struct nfq_data *nfad, unsigned char **data);
                                                   ~~~~~~~~~~~~~~~~^~~~
/home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ file /home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so
/home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so: symbolic link to libnetfilter_queue.so.1.4.0
rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ file /home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so.1.4.0 
/home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so.1.4.0: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$