ARM cross compiler/linker cannot locate certain libraries

geneva1 commented 4 years ago

ARM cross compiler/linker cannot locate certain libraries for https://github.com/irontec/netfilter-nfqueue-samples, but normal system-wide gcc compiler/linker can.

Why ?

dell@dell-hp:~/Downloads/netfilter-nfqueue-samples$ gcc -o sample-helloworld sample-helloworld.c -lnfnetlink -lnetfilter_queue -lpthread -lm -ldl -I/usr/include/ -I/usr/include/x86_64-linux-gnu/
sample-helloworld.c: In function ‘print_pkt’:
sample-helloworld.c:55:28: warning: passing argument 2 of ‘nfq_get_payload’ from incompatible pointer type [-Wincompatible-pointer-types]
   55 |  ret = nfq_get_payload(tb, &data);
      |                            ^~~~~
      |                            |
      |                            char **
In file included from sample-helloworld.c:7:
/usr/include/libnetfilter_queue/libnetfilter_queue.h:119:67: note: expected ‘unsigned char **’ but argument is of type ‘char **’
  119 | extern int nfq_get_payload(struct nfq_data *nfad, unsigned char **data);
      |                                                   ~~~~~~~~~~~~~~~~^~~~
dell@dell-hp:~/Downloads/netfilter-nfqueue-samples$ ls
LICENSE  README.md  sample-helloworld  sample-helloworld.c  sample-memcache.c  sample-mysql.c
dell@dell-hp:~/Downloads/netfilter-nfqueue-samples$ rm sample-helloworld
dell@dell-hp:~/Downloads/netfilter-nfqueue-samples$ /home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc -o sample-helloworld sample-helloworld.c -lnfnetlink -lnetfilter_queue -lpthread -lm -ldl -I/usr/include/ -I/usr/include/x86_64-linux-gnu/
sample-helloworld.c: In function ‘print_pkt’:
sample-helloworld.c:55:28: warning: passing argument 2 of ‘nfq_get_payload’ from incompatible pointer type [-Wincompatible-pointer-types]
  ret = nfq_get_payload(tb, &data);
                            ^~~~~
In file included from sample-helloworld.c:7:
/usr/include/libnetfilter_queue/libnetfilter_queue.h:119:67: note: expected ‘unsigned char **’ but argument is of type ‘char **’
 extern int nfq_get_payload(struct nfq_data *nfad, unsigned char **data);
                                                   ~~~~~~~~~~~~~~~~^~~~
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnfnetlink
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnetfilter_queue
collect2: error: ld returned 1 exit status
dell@dell-hp:~/Downloads/netfilter-nfqueue-samples$

jforissier commented 4 years ago

What's surprising about that?

gcc is the host compiler, it uses host libraries (x86 or x86_64 architecture).
arm-linux-gnueabihf-gcc is a cross-compiler, it needs target libraries (arm architecture).

geneva1 commented 4 years ago

However, I could not install ARM version of libnetfilter-queue-dev on my AMD hardware.

dell@dell-hp:/tmp/mozilla_rog0$ sudo dpkg -i libnetfilter-queue-dev_1.0.2-2_arm64.deb 
dpkg: error processing archive libnetfilter-queue-dev_1.0.2-2_arm64.deb (--install):
 package architecture (arm64) does not match system (amd64)
Errors were encountered while processing:
 libnetfilter-queue-dev_1.0.2-2_arm64.deb
dell@dell-hp:/tmp/mozilla_rog0$

jforissier commented 4 years ago

Yes, it is a package for an Arm64 host, not for a cross-compilation environment.

Since it seems you are using the OP-TEE QEMU environment, which uses Buildroot, I think you should update the Buildroot configuration to install the libraries you need. Adding BR2_PACKAGE_LIBNETFILTER_QUEUE ?= y to common.mk should be enough.

geneva1 commented 4 years ago

@jforissier do you have any idea about https://github.com/OP-TEE/optee_os/issues/3901#issuecomment-635837697 ?

etienne-lms commented 4 years ago

@jforissier do you have any idea about #3901 (comment) ?

Hi @geneva1. Here buildroot should help you the same way as Jérôme showed for libnetfilterqueue support. To add package foo in the target, you should start by checking if package foo is supported by Buildroot and if so you can enable BR2_PACKAGE_FOO=y in common.mk.

For example, to add busybox tool watchdog support in the target, add BR2_PACKAGE_BUSYBOX_WATCHDOG=y. Runnning Qemu, you'll see binary watchdog is installed.

Another example: to embed ALSA library in the tested rootfs, add BR2_PACKAGE_ALSA_LIB=y. To add ALSA tools from alsa-utils package, enable BR2_PACKAGE_ALSA_UTILS=y.

Doing so, you will get the expected files already installed in the root filesystem used by the Qemu setup. No need to manually copy them into the embedded filesystem.

Check the packages supported by Builrdoot (.../buildroot/package/), there are quite a lot available. But maybe the software you want is not already supported by Buildroot...

geneva1 commented 4 years ago

@etienne-lms @jforissier you were discussing about adding package inside QEMU buildroot.

However, adding package inside buildroot does not help me to compile code (that needs to be running on ARM platform) on AMD platform using ARM cross-compiler.

Note that there is no compiler inside QEMU.

/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnfnetlink
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnetfilter_queue

etienne-lms commented 4 years ago

Buildroot will take care of cross-compiling the packages you enable through BR2_PACKAGE_xxx=y and copying the embedded files in the rootfs.

See for example GBD. The package is enable in br-ext/configs/gdbserver.conf (BR2_PACKAGE_GDB=y) and hence get cross -compiled ~~build~~ and embedded in the rootfs.

(edited)

geneva1 commented 4 years ago

cat common.mk | grep "BR2_PACKAGE_"
BR2_PACKAGE_LIBNFNETLINK ?= y
BR2_PACKAGE_LIBNETFILTER_QUEUE ?= y

@etienne-lms Adding package to buildroot does not just help to cross-compile. I think you are missing some more steps during your explanation above.

What did you exactly mean by copying the embedded files in the rootfs ?

etienne-lms commented 4 years ago

Here is the picture when you fetch the OP-TEE/Qemu repo manifest and use make all command:

Qemu emulator is build outside Buildroot, and install in ../qemu/arm-softmmu/qemu-system-arm. See target qemu in build/qemu.mk.
tf-a, optee_os, u-boot and linux are cross compiled outside Buildroot and generated binaries installed (using symlinks) in ../out/bin. See targets arm-tf, optee-os, u-boot and linux in build/qemu.mk.
(almost) all files stored in the rootfs use by the Qemu emulation are cross-compiled and installed in rootfs filetree using Buildroot. Buildroot also generates the rootfs image (cpio) for the Qemu emulation, installed (symlink) in ../out/bin/rootfs.cpio.gz. See target builroot in build/common.mk.

When you run make run-only, Qemu is launched and uses boot images from ../out/bin/: -bios bl1.bin argument provides 1st boot stage image reference. The configuration used to build tf-a/u-boot makes then also using boot images from ../out/bin/, including the rootfs mounted by Linux from ../out-br/images/rootfs.cpio.gz

Therefore, if you enable a package in Buildroot, it is cross-compiled and embedded in rootfs.cpio.gz.

This is the basic setup. @jbech-linaro details in https://github.com/OP-TEE/optee_os/issues/3901 thread how share of filetree between your host and the runtime Linux OS running in Qemu, using 9p virtio support. This is an extra (and useful feature) but not mandated to add a package in your tested target.

I hope the above is clear but don't hesitate to ask for details if something is confusing.

geneva1 commented 4 years ago

I suppose rootfs.cpio.gz contains the additional ARM packages.

@etienne-lms So, how do I tell ARM cross-compiler to use rootfs.cpio.gz ?

etienne-lms commented 4 years ago

how do I tell ARM cross-compiler to use rootfs.cpio.gz ?

Buildroot takes care of that.

For me to understand your concerns: do you have a specific, maybe private, package you would like to cross-compile and embed in the rootfs? or is it generic packages already supported by Buildroot that you want to add in the tested rootfs?

geneva1 commented 4 years ago

@etienne-lms See https://github.com/OP-TEE/optee_os/issues/3902#issuecomment-636796389

Even after rootfs contains the additional packages, ARM cross compiler still throws me error.

I think you missed out on steps to do cross-compilation for ARM after adding additional packages in buildroot

/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnfnetlink
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnetfilter_queue

etienne-lms commented 4 years ago

Ok , lets see what happens when we ask Buildroot to enable libnfnetlink and libnetfilter_queue. In the description below, lines starting with ubuntu> show command used on my host (Ubuntu-18.04) and lines starting with # show command I run in the Linux console running in Qemu.

As a start point I fetched a vanilla OP-TEE/qemu repo and build it:

ubuntu> repo init -u https://github.com/OP-TEE/manifest.git -m default.xml
ubuntu> repo sync
ubuntu> make toolchains
ubuntu> make all
ubuntu> make run-only

From here, from the Linux console, i've listed all files found by embedded tested Linux OS that are named lib*net*:

buildroot login: root
# find / -name lib\*net\*
#

Nothing found. That is what I expected.

Now let's add the above lines in common.mk:

BR2_PACKAGE_LIBNFNETLINK ?= y
BR2_PACKAGE_LIBNETFILTER_QUEUE ?= y

Let's rebuild (from the host, not from Qemu!). From the build traces, I can see some new packages being built (by Buildroot):

ubuntu> make all
(...)
>>> libmnl 1.0.4 Downloading
>>> libmnl 1.0.4 Extracting
>>> libmnl 1.0.4 Patching
(...)
>>> libmnl 1.0.4 Building
>>> libmnl 1.0.4 Installing to staging directory
>>> libmnl 1.0.4 Fixing libtool files
(...)
>>> libnfnetlink 1.0.1 Downloading
(...)
>>> libnfnetlink 1.0.1 Building
(...)
>>> libnfnetlink 1.0.1 Installing to target
(...)
>>> libnetfilter_queue 1.0.3 Downloading
(...)
>>> libnetfilter_queue 1.0.3 Building
(...)
>>> libnetfilter_queue 1.0.3 Installing to target
(...)

No build error as the one you report in https://github.com/OP-TEE/optee_os/issues/3902#issuecomment-636819575.

Now let's run Qemu again:

ubuntu> make run-only

Again, from the test Linux console, i've listed again all file found by embedded tested Linux OS that are named lib*net*:

buildroot login: root
# find / -name lib\*net\*
/usr/lib/libnfnetlink.so.0.2.0
/usr/lib/libnetfilter_queue.so.1
/usr/lib/libnfnetlink.so
/usr/lib/libnetfilter_queue.so
/usr/lib/libnfnetlink.so.0
/usr/lib/libnetfilter_queue.so.1.4.0
#

That's fine: the libraries we expected have been copied to the tested rootfs. Did you expect more?

geneva1 commented 4 years ago

# ls -al /usr/lib/libn*
lrwxrwxrwx    1 root     root            27 May 29 09:52 /usr/lib/libnetfilter_queue.so -> libnetfilter_queue.so.1.4.0
lrwxrwxrwx    1 root     root            27 May 29 09:52 /usr/lib/libnetfilter_queue.so.1 -> libnetfilter_queue.so.1.4.0
-rwxr-xr-x    1 root     root         13384 Jun  1 08:08 /usr/lib/libnetfilter_queue.so.1.4.0
lrwxrwxrwx    1 root     root            21 May 29 09:51 /usr/lib/libnfnetlink.so -> libnfnetlink.so.0.2.0
lrwxrwxrwx    1 root     root            21 May 29 09:51 /usr/lib/libnfnetlink.so.0 -> libnfnetlink.so.0.2.0
-rwxr-xr-x    1 root     root         17392 Jun  1 08:08 /usr/lib/libnfnetlink.so.0.2.0
#

@etienne-lms are you implying that I could use these libraries found inside QEMU (ARM emulation platform) to do cross compilation on actual AMD platform ?

etienne-lms commented 4 years ago

hmm, I would say yes, but i'm not sure I properly understand your question. The AMD platform you're referring to is the host PC you use to build Qemu, OP-TEE, etc...? Or it is a target platform you want OP-TEE to run in?

geneva1 commented 4 years ago

@etienne-lms The AMD platform is the ubuntu host OS that I used to build QEMU, OP-TEE, etc.....

So, I will just copy those libraries from inside QEMU to the host OS for external linking during cross-compilation

etienne-lms commented 4 years ago

Then, ok. You will find the generated cross-compiled files (libnetfilter_queue.so, libnfnetlink.so, etc...) on your AMD host machine at path ../out-br/target/usr/lib/. ../out-br/target/ is the filetree generated by buildroot with all files embedded in the rootfs used by Linux running in Qemu.

geneva1 commented 4 years ago

dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$ ls /home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib/
libatomic.so.1       libmnl.so                    libnfnetlink.so.0.2.0       libteec.so.1.0.0
libatomic.so.1.2.0   libmnl.so.0                  libssl.so                   libz.so
libckteec.so         libmnl.so.0.2.0              libssl.so.1.1               libz.so.1
libckteec.so.0.1.0   libnetfilter_queue.so        libstdc++.so.6              libz.so.1.2.11
libckteec.so.ckteec  libnetfilter_queue.so.1      libstdc++.so.6.0.25         os-release
libcrypto.so         libnetfilter_queue.so.1.4.0  libstdc++.so.6.0.25-gdb.py
libcrypto.so.1.1     libnfnetlink.so              libteec.so
libgcc_s.so.1        libnfnetlink.so.0            libteec.so.1
dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$ /home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc -o sample-helloworld sample-helloworld.c -lpthread -lm -ldl -I/usr/include/ -I/usr/include/x86_64-linux-gnu/ -L/home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib/ -lnetfilter_queue -lnfnetlink
sample-helloworld.c: In function ‘print_pkt’:
sample-helloworld.c:55:28: warning: passing argument 2 of ‘nfq_get_payload’ from incompatible pointer type [-Wincompatible-pointer-types]
  ret = nfq_get_payload(tb, &data);
                            ^~~~~
In file included from sample-helloworld.c:7:
/usr/include/libnetfilter_queue/libnetfilter_queue.h:119:67: note: expected ‘unsigned char **’ but argument is of type ‘char **’
 extern int nfq_get_payload(struct nfq_data *nfad, unsigned char **data);
                                                   ~~~~~~~~~~~~~~~~^~~~
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: warning: libmnl.so.0, needed by /home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib//libnetfilter_queue.so, not found (try using -rpath or -rpath-link)
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: /home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib//libnetfilter_queue.so: undefined reference to `mnl_attr_type_valid@LIBMNL_1.0'
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: /home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib//libnetfilter_queue.so: undefined reference to `mnl_attr_parse@LIBMNL_1.0'
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: /home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib//libnetfilter_queue.so: undefined reference to `mnl_attr_put_u32@LIBMNL_1.0'
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: /home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib//libnetfilter_queue.so: undefined reference to `mnl_attr_validate2@LIBMNL_1.0'
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: /home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib//libnetfilter_queue.so: undefined reference to `mnl_attr_get_type@LIBMNL_1.0'
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: /home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib//libnetfilter_queue.so: undefined reference to `mnl_attr_put@LIBMNL_1.0'
/home/dell/Downloads/optee_os/optee-qemu/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: /home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib//libnetfilter_queue.so: undefined reference to `mnl_attr_validate@LIBMNL_1.0'
collect2: error: ld returned 1 exit status
dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$

@etienne-lms why all the undefined reference errors ?

etienne-lms commented 4 years ago

I can't tell. libmnl is installed in the target rootfs:

ubuntu> find ../out-br/target -name libmnl\*
../out-br/target/usr/lib/libmnl.so.0.2.0
../out-br/target/usr/lib/libmnl.so.0
../out-br/target/usr/lib/libmnl.so
ubuntu>

Maybe add argument -lmnl when building sample-helloworld.c.

geneva1 commented 4 years ago

Why the difference in output result for the same code in https://github.com/irontec/netfilter-nfqueue-samples ?

on QEMU ARM emulation platform :

# mv /mnt/host/sample-helloworld /home/qemu/
# cd home/qemu/
# ./sample-helloworld 
opening library handle
error during nfq_open()
#

on actual AMD Ubuntu PC platform :

dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$ ./sample-helloworld 
opening library handle
unbinding existing nf_queue handler for AF_INET (if any)
error during nfq_unbind_pf()
dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$

etienne-lms commented 4 years ago

I don't think I can help you here. Maybe Linux kernel lacks required configuration switches? I'll need to investigate and ask help from someone that knows about netfilter things.

geneva1 commented 4 years ago

on actual AMD Ubuntu PC platform :

dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$ lsof -i -P -n | grep LISTEN
PM2\x20v4  2751  rog   16u  IPv4  43760      0t0  TCP 127.0.0.1:8081 (LISTEN)
soc_term  19017  rog    3u  IPv4 228170      0t0  TCP *:54321 (LISTEN)
soc_term  19018  rog    3u  IPv4 232923      0t0  TCP *:54320 (LISTEN)
qemu-syst 19021  rog   14u  IPv4 232938      0t0  TCP *:1234 (LISTEN)
qemu-syst 19021  rog   15u  IPv6 232939      0t0  TCP *:1234 (LISTEN)
dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$

no output on QEMU ARM emulation platform

geneva1 commented 4 years ago

@etienne-lms

Someone told me that it does not seem like https://optee.readthedocs.io/en/latest/building/devices/qemu.html runs a kernel for me.

Please correct me if wrong.

etienne-lms commented 4 years ago

I assume here you fetched OP-TEE repo default.xml and built it as detailed in https://github.com/OP-TEE/optee_os/issues/3902#issuecomment-636844310.

On your AMD Ubuntu PC platform, the shell command make run-only from OP-TEE build/ directory launches a Qemu emulation. The emulated system uses:

TF-A as bootloader (TF-A BL1 as initial bootstage which boots TF-A BL2)
OP-TEE core in the secure world (loaded & booted by TF-A BL2)
U-Boot as first bootloader in the non-secure world (loaded by TF-A BL2, boot right after OP-TEE core initialization is completed)
Linux kernel in the non-secure world (loaded/booted by U-Boot)
Buildroot rootfs as Linux userland (mounted by Linux).

All in one: with OP-TEE Qemu/armv7 device [1], you DO run a cross-compiled Linux kernel in Qemu.

[1] https://optee.readthedocs.io/en/latest/building/devices/qemu.html#qemu-v7

geneva1 commented 4 years ago

So, what causes https://github.com/OP-TEE/optee_os/issues/3902#issuecomment-637241853 that it seems no socket is listening within QEMU ?

etienne-lms commented 4 years ago

Sorry, I lack of knowledge to answer your question.

jenswi-linaro commented 4 years ago

So, what causes #3902 (comment) that it seems no socket is listening within QEMU ?

Why should that be reported there?

geneva1 commented 4 years ago

@jenswi-linaro because it works well in AMD Ubuntu host PC platform, but not inside QEMU ARM emulation platform

dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$ sudo ./sample-helloworld 
opening library handle
unbinding existing nf_queue handler for AF_INET (if any)
binding nfnetlink_queue as nf_queue handler for AF_INET
binding this socket to queue '0'
setting copy_packet mode
^C
dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$

jenswi-linaro commented 4 years ago

Try running the command inside the emulator then.

geneva1 commented 4 years ago

@jenswi-linaro

Welcome to Buildroot, type root or test to login
buildroot login: root
# cd /home/qemu/
# ./sample-helloworld 
opening library handle
error during nfq_open()
#

jenswi-linaro commented 4 years ago

I don't see how this is OP-TEE related.

geneva1 commented 4 years ago

the QEMU buildroot is from OP-TEE repo.

I suspect something might have went wrong during QEMU buildroot compilation.

etienne-lms commented 4 years ago

@geneva1, this looks like dependencies needed by netfilter-nfqueue-samples package or so. Buildroot is expected to take care of that. Maybe you should ask help on a netfilter forum or on the Buildroot mailing list.

geneva1 commented 4 years ago

@etienne-lms

Sure, but why is it related to this specific package netfilter-nfqueue-samples ?

jenswi-linaro commented 4 years ago

Because it needs special kernel options?

etienne-lms commented 4 years ago

It is not specific to this package. Many packages have dependencies. For example libnetfilter_queue has dependencies. There are listed in the native Buildroot [1]. Maybe the package(s) you need to enable in Buildroot are missing some dependencies declaration. If so, this should be addresses inside Buildroot component, not in OP-TEE. Unless of course if it is a dependency on a Linux kernel configuration, in which case you will need to update the Linux config for the OP-TEE/Qemu build in [2].

[1] https://git.buildroot.net/buildroot/tree/package/libnetfilter_queue/libnetfilter_queue.mk?h=2020.02.x [2] https://github.com/OP-TEE/build/blob/3.9.0/kconfigs/qemu.conf

geneva1 commented 4 years ago

@etienne-lms @jenswi-linaro

LIBNETFILTER_QUEUE_DEPENDENCIES = host-pkgconf libnfnetlink libmnl

the buildroot rootfs already had libnfnetlink and libmnl

Could you advise about host-pkgconf ?

How do I include host-pkgconf into QEMU buildroot for OP-TEE ?

Note: I had already had BR2_PACKAGE_HOST_PKGCONF ?= y inside common.mk

dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$ ls /home/dell/Downloads/optee_os/optee-qemu/out-br/target/usr/lib/
libatomic.so.1       libnetfilter_queue.so        libstdc++.so.6
libatomic.so.1.2.0   libnetfilter_queue.so.1      libstdc++.so.6.0.25
libckteec.so         libnetfilter_queue.so.1.4.0  libstdc++.so.6.0.25-gdb.py
libckteec.so.0.1.0   libnfnetlink.so              libteec.so
libckteec.so.ckteec  libnfnetlink.so.0            libteec.so.1
libcrypto.so         libnfnetlink.so.0.2.0        libteec.so.1.0.0
libcrypto.so.1.1     libpkgconf.so                libz.so
libgcc_s.so.1        libpkgconf.so.3              libz.so.1
libmnl.so            libpkgconf.so.3.0.0          libz.so.1.2.11
libmnl.so.0          libssl.so                    os-release
libmnl.so.0.2.0      libssl.so.1.1
dell@dell-hpc:~/Downloads/netfilter-nfqueue-samples$

geneva1 commented 4 years ago

Inside QEMU, it seems like libpkgconf.so is already installed.

So, does this imply that nfq_open() error is not due to libnetfilter_queue and its dependencies ?

Welcome to Buildroot, type root or test to login
buildroot login: root
# ls ../
bin      etc      lib32    mnt      root     sys      var
data     init     linuxrc  opt      run      tmp
dev      lib      media    proc     sbin     usr
# mkdir -p /home/qemu
# mkdir -p /mnt/host
# mount -t 9p -o trans=virtio host /mnt/host/
# cp /mnt/host/sample-helloworld /home/qemu/
# cd /home/qemu/
# ./sample-helloworld 
opening library handle
error during nfq_open()
# ls /usr/lib
libatomic.so.1               libnfnetlink.so.0.2.0
libatomic.so.1.2.0           libpkgconf.so
libckteec.so                 libpkgconf.so.3
libckteec.so.0.1.0           libpkgconf.so.3.0.0
libckteec.so.ckteec          libssl.so
libcrypto.so                 libssl.so.1.1
libcrypto.so.1.1             libstdc++.so.6
libgcc_s.so.1                libstdc++.so.6.0.25
libmnl.so                    libstdc++.so.6.0.25-gdb.py
libmnl.so.0                  libteec.so
libmnl.so.0.2.0              libteec.so.1
libnetfilter_queue.so        libteec.so.1.0.0
libnetfilter_queue.so.1      libz.so
libnetfilter_queue.so.1.4.0  libz.so.1
libnfnetlink.so              libz.so.1.2.11
libnfnetlink.so.0            os-release
# ./sample-helloworld 
opening library handle
error during nfq_open()
# lsof -i -P -n | grep LISTEN
#

geneva1 commented 4 years ago

The more significant difference between running this netfilter code on actual AMD x64 platform and ARM emulation qemu platform is char buf[4096] __attribute__ ((aligned)); which is really system or architecture-dependent.

Please correct me if wrong.

jforissier commented 4 years ago

Have you checked errno? Any message in the kernel ring buffer (dmesg)? How about using strace to see what's happening at the user space/kernel interface?

geneva1 commented 4 years ago

@jforissier Which errno were you referring to ?

I have socket(AF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = -1 EPROTONOSUPPORT (Protocol not supported) from strace inside QEMU ARM emulation platform

and I have socket(AF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = 3 using strace inside actual AMD x64 ubuntu host PC.

Welcome to Buildroot, type root or test to login
buildroot login: root
# cd ../
# ls
bin      etc      lib32    mnt      root     sys      var
data     init     linuxrc  opt      run      tmp
dev      lib      media    proc     sbin     usr
# mkdir -p /home/qemu
# mkdir -p /mnt/host
# mount -t 9p -o trans=virtio host /mnt/host/
# cp /mnt/host/sample-helloworld  /home/qemu/
# cd home/qemu/
# ./sample-helloworld 
opening library handle
error during nfq_open()
# dmesg
Booting Linux on physical CPU 0x0
Linux version 5.5.0 (rog@rog-GL502VMK) (gcc version 8.3.0 (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36))) #1 SMP PREEMPT Thu May 28 10:04:24 +08 2020
CPU: ARMv7 Processor [412fc0f1] revision 1 (ARMv7), cr=10c5387d
CPU: div instructions available: patching division code
CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
OF: fdt: Machine model: linux,dummy-virt
OF: fdt: Ignoring memory block 0xe000000 - 0xf000000
Memory policy: Data cache writealloc
cma: Reserved 16 MiB at 0x81000000
On node 0 totalpages: 270080
  Normal zone: 2114 pages used for memmap
  Normal zone: 0 pages reserved
  Normal zone: 270080 pages, LIFO batch:63
psci: probing for conduit method from DT.
psci: PSCIv0.2 detected in firmware.
psci: Using standard PSCI v0.2 function IDs
psci: MIGRATE_INFO_TYPE not supported.
percpu: Embedded 19 pages/cpu s45900 r8192 d23732 u77824
pcpu-alloc: s45900 r8192 d23732 u77824 alloc=19*4096
pcpu-alloc: [0] 0 [0] 1 
Built 1 zonelists, mobility grouping on.  Total pages: 267966
Kernel command line: console=ttyAMA0,115200 earlyprintk=serial,ttyAMA0,115200
Dentry cache hash table entries: 262144 (order: 8, 1048576 bytes, linear)
Inode-cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 1037844K/1080320K available (7168K kernel code, 426K rwdata, 1676K rodata, 1024K init, 155K bss, 26092K reserved, 16384K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
rcu: Preemptible hierarchical RCU implementation.
rcu:    RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=2.
    Tasks RCU enabled.
rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
random: get_random_bytes called from start_kernel+0x318/0x4b4 with crng_init=0
arch_timer: cp15 timer(s) running at 62.50MHz (virt).
clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns
sched_clock: 56 bits at 62MHz, resolution 16ns, wraps every 4398046511096ns
Switching to timer-based delay loop, resolution 16ns
Console: colour dummy device 80x30
Calibrating delay loop (skipped), value calculated using timer frequency.. 125.00 BogoMIPS (lpj=625000)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 4096 (order: 2, 16384 bytes, linear)
Mountpoint-cache hash table entries: 4096 (order: 2, 16384 bytes, linear)
CPU: Testing write buffer coherency: ok
CPU0: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
/cpus/cpu@0 missing clock-frequency property
/cpus/cpu@1 missing clock-frequency property
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x40100000 - 0x40100060
rcu: Hierarchical SRCU implementation.
smp: Bringing up secondary CPUs ...
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
CPU1: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
smp: Brought up 1 node, 2 CPUs
SMP: Total of 2 processors activated (250.00 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 4 part 30 variant f rev 0
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 512 (order: 3, 32768 bytes, linear)
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
cpuidle: using governor ladder
hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 8 bytes.
Serial: AMBA PL011 UART driver
9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 54, base_baud = 0) is a PL011 rev1
printk: console [ttyAMA0] enabled
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Advanced Linux Sound Architecture Driver Initialized.
clocksource: Switched to clocksource arch_sys_counter
NET: Registered protocol family 2
tcp_listen_portaddr_hash hash table entries: 1024 (order: 1, 12288 bytes, linear)
TCP established hash table entries: 16384 (order: 4, 65536 bytes, linear)
TCP bind hash table entries: 16384 (order: 5, 131072 bytes, linear)
TCP: Hash tables configured (established 16384 bind 16384)
UDP hash table entries: 1024 (order: 3, 32768 bytes, linear)
UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes, linear)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Unpacking initramfs...
Initramfs unpacking failed: invalid magic at start of compressed archive
Freeing initrd memory: 4992K
workingset: timestamp_bits=30 max_order=19 bucket_order=0
squashfs: version 4.0 (2009/01/31) Phillip Lougher
9p: Installing v9fs 9p2000 file system support
io scheduler mq-deadline registered
io scheduler kyber registered
physmap-flash 4000000.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]
4000000.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
erase region 0: offset=0x0,size=0x40000,blocks=256
libphy: Fixed MDIO Bus: probed
usbcore: registered new interface driver usb-storage
rtc-pl031 9010000.pl031: registered as rtc0
ledtrig-cpu: registered to indicate activity on CPUs
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
optee: probing for conduit method from DT.
optee: revision 3.9 (0949602c)
optee: dynamic shared memory is enabled
optee: initialized driver
oprofile: no performance counters
oprofile: using timer interrupt.
NET: Registered protocol family 17
9pnet: Installing 9P2000 support
Registering SWP/SWPB emulation handler
rtc-pl031 9010000.pl031: setting system clock to 2020-06-03T07:33:07 UTC (1591169587)
ALSA device list:
  No soundcards found.
Freeing unused kernel memory: 1024K
Run /init as init process
random: dd: uninitialized urandom read (512 bytes read)

# strace ./sample-helloworld 
execve("./sample-helloworld", ["./sample-helloworld"], 0x7ef30e40 /* 12 vars */) = 0
brk(NULL)                               = 0x23000
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76fe8000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/tls/v7l/neon/vfp/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/v7l/neon/vfp", 0x7ea6c278) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/tls/v7l/neon/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/v7l/neon", 0x7ea6c278) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/tls/v7l/vfp/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/v7l/vfp", 0x7ea6c278)  = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/tls/v7l/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/v7l", 0x7ea6c278)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/tls/neon/vfp/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/neon/vfp", 0x7ea6c278) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/tls/neon/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/neon", 0x7ea6c278)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/tls/vfp/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/vfp", 0x7ea6c278)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/tls/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls", 0x7ea6c278)          = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/v7l/neon/vfp/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/v7l/neon/vfp", 0x7ea6c278) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/v7l/neon/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/v7l/neon", 0x7ea6c278)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/v7l/vfp/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/v7l/vfp", 0x7ea6c278)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/v7l/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/v7l", 0x7ea6c278)          = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/neon/vfp/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/neon/vfp", 0x7ea6c278)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/neon/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/neon", 0x7ea6c278)         = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/vfp/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/vfp", 0x7ea6c278)          = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\270N\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=125608, ...}) = 0
mmap2(NULL, 164460, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76f91000
mprotect(0x76fa7000, 61440, PROT_NONE)  = 0
mmap2(0x76fb6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x76fb6000
mmap2(0x76fb8000, 4716, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76fb8000
close(3)                                = 0
openat(AT_FDCWD, "/lib/libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\350q\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=451932, ...}) = 0
mmap2(NULL, 516208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76f12000
mprotect(0x76f80000, 61440, PROT_NONE)  = 0
mmap2(0x76f8f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6d000) = 0x76f8f000
close(3)                                = 0
openat(AT_FDCWD, "/lib/libdl.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\210\t\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=9600, ...}) = 0
mmap2(NULL, 73916, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76eff000
mprotect(0x76f01000, 61440, PROT_NONE)  = 0
mmap2(0x76f10000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x76f10000
close(3)                                = 0
openat(AT_FDCWD, "/lib/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/tls/v7l/neon/vfp/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/v7l/neon/vfp", 0x7ea6c248) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/tls/v7l/neon/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/v7l/neon", 0x7ea6c248) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/tls/v7l/vfp/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/v7l/vfp", 0x7ea6c248) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/tls/v7l/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/v7l", 0x7ea6c248)  = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/tls/neon/vfp/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/neon/vfp", 0x7ea6c248) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/tls/neon/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/neon", 0x7ea6c248) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/tls/vfp/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/vfp", 0x7ea6c248)  = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/tls/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls", 0x7ea6c248)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/v7l/neon/vfp/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/v7l/neon/vfp", 0x7ea6c248) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/v7l/neon/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/v7l/neon", 0x7ea6c248) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/v7l/vfp/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/v7l/vfp", 0x7ea6c248)  = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/v7l/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/v7l", 0x7ea6c248)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/neon/vfp/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/neon/vfp", 0x7ea6c248) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/neon/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/neon", 0x7ea6c248)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/vfp/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/vfp", 0x7ea6c248)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/libnetfilter_queue.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0$\33\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=13384, ...}) = 0
mmap2(NULL, 78052, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76eeb000
mprotect(0x76eee000, 61440, PROT_NONE)  = 0
mmap2(0x76efd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x76efd000
close(3)                                = 0
openat(AT_FDCWD, "/lib/libnfnetlink.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/libnfnetlink.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\360\22\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=17392, ...}) = 0
mmap2(NULL, 82104, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76ed6000
mprotect(0x76eda000, 61440, PROT_NONE)  = 0
mmap2(0x76ee9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x76ee9000
close(3)                                = 0
openat(AT_FDCWD, "/lib/libmnl.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/libmnl.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\300\26\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=13736, ...}) = 0
mmap2(NULL, 78004, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76ec2000
mprotect(0x76ec5000, 61440, PROT_NONE)  = 0
mmap2(0x76ed4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x76ed4000
close(3)                                = 0
openat(AT_FDCWD, "/lib/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\234x\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1246708, ...}) = 0
mmap2(NULL, 1315620, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d80000
mprotect(0x76ead000, 61440, PROT_NONE)  = 0
mmap2(0x76ebc000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12c000) = 0x76ebc000
mmap2(0x76ebf000, 8996, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76ebf000
close(3)                                = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76fe6000
set_tls(0x76fe64d0)                     = 0
mprotect(0x76ebc000, 8192, PROT_READ)   = 0
mprotect(0x76ed4000, 4096, PROT_READ)   = 0
mprotect(0x76ee9000, 4096, PROT_READ)   = 0
mprotect(0x76efd000, 4096, PROT_READ)   = 0
mprotect(0x76f10000, 4096, PROT_READ)   = 0
mprotect(0x76f8f000, 4096, PROT_READ)   = 0
mprotect(0x76fb6000, 4096, PROT_READ)   = 0
mprotect(0x21000, 4096, PROT_READ)      = 0
mprotect(0x76fea000, 4096, PROT_READ)   = 0
set_tid_address(0x76fe6078)             = 135
set_robust_list(0x76fe6080, 12)         = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x76f95854, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x76dac2c0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x76f95910, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x76dac2c0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
fstat64(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0xcc, 0x40), ...}) = 0
ioctl(1, TCGETS, {B115200 opost isig icanon echo ...}) = 0
brk(NULL)                               = 0x23000
brk(0x44000)                            = 0x44000
write(1, "opening library handle\n", 23opening library handle
) = 23
socket(AF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = -1 EPROTONOSUPPORT (Protocol not supported)
write(2, "error during nfq_open()\n", 24error during nfq_open()
) = 24
exit_group(1)                           = ?
+++ exited with 1 +++
#

jforissier commented 4 years ago

@jforissier Which errno were you referring to ?

:man_facepalming: nevermind


Welcome to Buildroot, type root or test to login
buildroot login: root
# strace ./sample-helloworld 
execve("./sample-helloworld", ["./sample-helloworld"], 0x7ef30e40 /* 12 vars */) = 0
...
write(1, "opening library handle\n", 23opening library handle
) = 23
socket(AF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = -1 EPROTONOSUPPORT (Protocol not supported)

So, as suggested by @jenswi-linaro and @etienne-lms you are missing a kernel option. You need to add CONFIG_NETFILTER=y to your kernel configuration (kconfigs/qemu.conf).

write(2, "error during nfq_open()\n", 24error during nfq_open() ) = 24 exit_group(1) = ? +++ exited with 1 +++

geneva1 commented 4 years ago

@jforissier Someone told me the following :

the kernel inside QEMU doesn't have netlink socket support, or sock_raw or the other parm aren't valid for netlink sockets.

geneva1 commented 4 years ago

@jforissier I have already added CONFIG_NETFILTER=y to my kernel configuration kconfigs/qemu.conf , but still the same error during nfq_open()

geneva1 commented 4 years ago

@jforissier someone told me that I need some other kernel parameters (such as CONFIG_NETFILTER_NETLINK=y) related to netfilter as described in https://github.com/intel/qemu-lite/wiki/Linux-config-4.5.0

jforissier commented 4 years ago

@jforissier someone told me that I need some other kernel parameters [...]

Yes, probably. We have been telling you this for a couple of days. Now would you please work with this knowledgeable person and stop posting here? I don't want to be rude, but your questions are now unrelated to OP-TEE and we can't spend time debugging everyone's application. Thank you.

geneva1 commented 4 years ago

@jforissier sorry. I just reused the entire kernel config given from intel qemu repo which contains all the related netfilter kernel configs, but I still have the same strace error.

geneva1 commented 4 years ago

Why the following compilation error ?

rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ /home/rog/optee_os/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc -o sample-helloworld sample-helloworld.c -lpthread -lm -ldl -I/usr/include/ -I/usr/include/x86_64-linux-gnu/ -L/home/rog/optee_os/out-br/target/usr/lib -lnetfilter_queue -lnfnetlink -lmnl
sample-helloworld.c: In function ‘print_pkt’:
sample-helloworld.c:55:28: warning: passing argument 2 of ‘nfq_get_payload’ from incompatible pointer type [-Wincompatible-pointer-types]
  ret = nfq_get_payload(tb, &data);
                            ^~~~~
In file included from sample-helloworld.c:7:
/usr/include/libnetfilter_queue/libnetfilter_queue.h:119:67: note: expected ‘unsigned char **’ but argument is of type ‘char **’
 extern int nfq_get_payload(struct nfq_data *nfad, unsigned char **data);
                                                   ~~~~~~~~~~~~~~~~^~~~
/home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status

rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ file /home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so
/home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so: symbolic link to libnetfilter_queue.so.1.4.0

rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ ls -al /home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so
lrwxrwxrwx 1 rog rog 27 Jun 18 20:20 /home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so -> libnetfilter_queue.so.1.4.0

rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ ls -al /home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so.1.4.0 
-rwxr-xr-x 1 rog rog 22256 Jun 18 20:21 /home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so.1.4.0

rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ file /home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so.1.4.0 
/home/rog/optee_os/out-br/target/usr/lib/libnetfilter_queue.so.1.4.0: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped

rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$

geneva1 commented 4 years ago

After removing and recompiling the cross-compiled libraries, I am able to eliminate the "file format not recognized" errors.

However, now I have the following errors. Why ?

rog@rog-GL502VMK:~/optee_os/build$ cat common.mk | grep NET
BR2_PACKAGE_LIBNFNETLINK ?= y
BR2_PACKAGE_LIBNETFILTER_QUEUE ?= y
rog@rog-GL502VMK:~/optee_os/build$

rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ rm /home/rog/optee_os/out-br/target/usr/lib/lib*
rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ /home/rog/optee_os/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc -o sample-helloworld sample-helloworld.c -lpthread -lm -ldl -I/usr/include/ -I/usr/include/x86_64-linux-gnu/ -L/home/rog/optee_os/out-br/target/usr/lib -lnetfilter_queue -lnfnetlink -lmnl
sample-helloworld.c: In function ‘print_pkt’:
sample-helloworld.c:55:28: warning: passing argument 2 of ‘nfq_get_payload’ from incompatible pointer type [-Wincompatible-pointer-types]
  ret = nfq_get_payload(tb, &data);
                            ^~~~~
In file included from sample-helloworld.c:7:
/usr/include/libnetfilter_queue/libnetfilter_queue.h:119:67: note: expected ‘unsigned char **’ but argument is of type ‘char **’
 extern int nfq_get_payload(struct nfq_data *nfad, unsigned char **data);
                                                   ~~~~~~~~~~~~~~~~^~~~
/home/rog/optee_os/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnetfilter_queue
/home/rog/optee_os/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lnfnetlink
/home/rog/optee_os/build/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lmnl
collect2: error: ld returned 1 exit status
rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$ ls /home/rog/optee_os/out-br/target/usr/lib/
libckteec.so        libckteec.so.ckteec  libteec.so.1      os-release
libckteec.so.0.1.0  libteec.so           libteec.so.1.0.0
rog@rog-GL502VMK:~/Downloads/netfilter-nfqueue-samples$

etienne-lms commented 4 years ago

The issue is detailed in the build trace:

sample-helloworld.c:55:28: warning: passing argument 2 of ‘nfq_get_payload’ from incompatible pointer type [-Wincompatible-pointer-types] ret = nfq_get_payload(tb, &data);

In file included from sample-helloworld.c:7: /usr/include/libnetfilter_queue/libnetfilter_queue.h:119:67: note: expected ‘unsigned char ’ but > argument is of type ‘char ’ extern int nfq_get_payload(struct nfq_data *nfad, unsigned char **data);

At line 55 of sample-helloworld.c, you use &data which is a char * whereas function nfq_get_payload() expects a unsigned char *. Either make data as unsigned char array reference; Or use a cast at line 55: ret = nfq_get_payload(tb, (unsigned char *)&data);

OP-TEE / optee_os

ARM cross compiler/linker cannot locate certain libraries #3902