aldebaran / libqi-python

qiSDK python bindings
BSD 3-Clause "New" or "Revised" License
15 stars 11 forks source link

Compiled libqi-python for Jetson TX2 #9

Closed robertanto closed 3 years ago

robertanto commented 4 years ago

Dear maintainers,

I read in a ArXiv paper from SoftBank Robotics Europe that the library has been already compiled for Jetson TX2.

May you share a link to download the compiled library?

Thanks in advance.

SOLUTION:

I have created a repository containing a faster way to use the library: https://github.com/robertanto/libqi-python-nvidia-jetson

It also works with the latest JetPack (4.4.1).

I hope that this repository can help researcher in their activities.

mbusy commented 4 years ago

Hi, if you want to reproduce the setup described in the paper, you will need to compile both libqi and libqi-python for the TX2. To do so, please follow this tutorial:

libqi & libqi-python installation instructions

Clone the libqi repository and the libqi-python repositories on the Jetson (in a qi_ws folder for instance, or wherever you want basically).

# From the Jetson:

# OPTIONAL: If you want to use qibuild
sudo pip install qibuild

# We will use a qi_ws folder in this tutorial, but you can clone the repositories wherever you want
mkdir qi_ws && cd qi_ws

# Clone the repos
git clone git@gitlab.aldebaran.lan:qi/libqi.git
git clone git@gitlab.aldebaran.lan:qi/libqi-python.git

# checkout the release-2.5 branch
cd libqi && git checkout release-2.5
cd ../libqi-python && git checkout release-2.5

cd ..

Then, download and copy the patches libqi-python-2.5_jetson.patch and libqi-2.5_jetson.patch on the Jetson. Respectively apply the patches to the repositories.

# From your computer:
scp libqi-2.5_jetson.patch nvidia@tegra-ubuntu.local:
scp libqi-python-2.5_jetson.patch nvidia@tegra-ubuntu.local:

# From the Jetson:
cd qi_ws/libqi && git apply path/to/patch/libqi-2.5_jetson.patch
cd ../libqi-python && git apply path/to/patch/libqi-python-2.5_jetson.patch

Once the patches are applied, build and install the projects.

# From the Jetson:
cd qi_ws/libqi
./compile.sh
sudo ./install.sh

cd ../libqi-python
./compile.sh
sudo ./install.sh
robertanto commented 3 years ago

Sorry for the late response. First of all, thank you @mbusy.

I cannot install it on my Jetson TX2 . I have installed the JetPack 4.4 (Ubuntu 18.04.5 LTS), is this setup compatible?

mbusy commented 3 years ago

@robertanto you're welcome. The setup isn't compatible, we used JetPack 3.3 (Ubuntu 16.04). The Boost versions (Boost used by Ubuntu 16.04 vs 18.04) being different, the patch that I provided won't work for your setup

robertanto commented 3 years ago

@mbusy thank you again!

I will install the same jetpack version and I will let you know.

robertanto commented 3 years ago

I compiled libqi.

Trying to compiling libqi-python, cmake asks me for QIPYTHON_DIR variable.

mbusy commented 3 years ago

Strange... Could you paste the full error log ?

robertanto commented 3 years ago

I solved adding to the cmake command -DWITH_BOOST_PYTHON. Now I have compiled both libqi and libqi-python for arm but when I try to import qi this is the message:

File` "/usr/lib/python2.7/dist-packages/qi/init.py", line 88, in from _qi import Application as _Application ImportError: dynamic module does not define init function (init_qi)

robertanto commented 3 years ago

I see that this is an open issue. Refer to #2 .

nyibbang commented 3 years ago

Following the discussion on #2, it seems you're using the qi module installed from pip, although you said you couldn't install it for your target platform (arm). Are you using the Python interpreter from a x86 host ?

robertanto commented 3 years ago

The qi package has been moved in the dist-packages folder due to the install.sh script contained in the patch.

robertanto commented 3 years ago

Are you using the Python interpreter from a x86 host ?

I am running the docker container directly on the Jetson.

nyibbang commented 3 years ago

Okay, so your _qi.so file should be located at /usr/lib/python2.7/dist-packages/_qi.so right ? is it the one you ran readelf on ? if not, can you check that one ?

Can you also check that libqi.so was installed in /usr/lib/, and check that all the dependencies of _qi.so are found by running ldd _qi.so ?

robertanto commented 3 years ago

It seems all ok.

root@c0598afa3794:/usr/lib/python2.7/dist-packages# readelf -s _qi.so | grep init
    60: 0000000000001a98     0 FUNC    GLOBAL DEFAULT    9 _init
    53: 0000000000014a48     0 OBJECT  LOCAL  DEFAULT   17 __frame_dummy_init_array_
    81: 0000000000003138     1 OBJECT  LOCAL  DEFAULT   13 _ZN5boostL17initialized_v
   111: 0000000000015158     1 OBJECT  LOCAL  DEFAULT   25 _ZStL8__ioinit
   130: 00000000000032d8    16 OBJECT  LOCAL  DEFAULT   13 _ZZ15init_module__qivE12_
   166: 0000000000002038   416 FUNC    LOCAL  DEFAULT   11 _Z41__static_initializati
   207: 0000000000001e48   496 FUNC    LOCAL  DEFAULT   11 _Z15init_module__qiv
   275: 0000000000001e20    40 FUNC    LOCAL  DEFAULT   11 init_qi
   329: 0000000000001a98     0 FUNC    GLOBAL DEFAULT    9 _init
root@c0598afa3794:/usr/lib/python2.7/dist-packages# ldd _qi.so 
 linux-vdso.so.1 =>  (0x0000007faf84c000)
 libpython2.7.so.1.0 => /usr/lib/aarch64-linux-gnu/libpython2.7.so.1.0 (0x0000007faf4b5000)
 libqipython.so => /usr/lib/libqipython.so (0x0000007faf242000)
 libboost_python.so.1.64.0 => /usr/local/lib/libboost_python.so.1.64.0 (0x0000007faf1de000)
 libqi.so => /usr/lib/libqi.so (0x0000007faded4000)
 libboost_system.so.1.64.0 => /usr/local/lib/libboost_system.so.1.64.0 (0x0000007fadec0000)
 libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007fadd31000)
 libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007fadd10000)
 libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007fadbca000)
 /lib/ld-linux-aarch64.so.1 (0x0000007faf821000)
 libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007fadb9e000)
 libz.so.1 => /lib/aarch64-linux-gnu/libz.so.1 (0x0000007fadb77000)
 libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007fadb64000)
 libutil.so.1 => /lib/aarch64-linux-gnu/libutil.so.1 (0x0000007fadb51000)
 libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007fadaa4000)
 libboost_thread.so.1.64.0 => /usr/local/lib/libboost_thread.so.1.64.0 (0x0000007fada6c000)
 libboost_filesystem.so.1.64.0 => /usr/local/lib/libboost_filesystem.so.1.64.0 (0x0000007fada42000)
 libboost_locale.so.1.64.0 => /usr/local/lib/libboost_locale.so.1.64.0 (0x0000007fad990000)
 libboost_chrono.so.1.64.0 => /usr/local/lib/libboost_chrono.so.1.64.0 (0x0000007fad978000)
 libboost_regex.so.1.64.0 => /usr/local/lib/libboost_regex.so.1.64.0 (0x0000007fad87a000)
 libboost_program_options.so.1.64.0 => /usr/local/lib/libboost_program_options.so.1.64.0 (0x0000007fad7f0000)
 libssl.so.1.0.0 => /lib/aarch64-linux-gnu/libssl.so.1.0.0 (0x0000007fad787000)
 libcrypto.so.1.0.0 => /lib/aarch64-linux-gnu/libcrypto.so.1.0.0 (0x0000007fad5e3000)
nyibbang commented 3 years ago

Can you try to run strace -e trace=file python -c "import qi" 2>strace.log and give us the content of grep _qi.so strace.log ?

robertanto commented 3 years ago
root@c0598afa3794:/# strace -e trace=file python -c "import qi" 2>strace.log
root@c0598afa3794:/# grep _qi.so strace.log 
openat(AT_FDCWD, "_qi.so", O_RDONLY)    = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/plat-aarch64-linux-gnu/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/lib-tk/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/lib-old/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/lib-dynload/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/lib/python2.7/dist-packages/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/dist-packages/_qi.so", O_RDONLY) = 4
openat(AT_FDCWD, "/usr/lib/python2.7/dist-packages/qi/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "_qi.so", O_RDONLY)    = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/plat-aarch64-linux-gnu/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/lib-tk/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/lib-dynload/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/lib/python2.7/dist-packages/_qi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python2.7/dist-packages/_qi.so", O_RDONLY) = 4
openat(AT_FDCWD, "/usr/lib/python2.7/dist-packages/_qi.so", O_RDONLY|O_CLOEXEC) = 5
nyibbang commented 3 years ago

Everything seems good, I'm sorry, I'm out of ideas at the moment. I'll come back to you with more things to check to investigate this issue.

robertanto commented 3 years ago

@nyibbang thank you for your time! I will try by using directly the Jetpack 3.3 as done by @mbusy.

A solution may be to provide your official container image for the Jetson Platform. In this way, developers can directly pull that image from the Docker Hub even on recent JetPacks avoiding problems!

robertanto commented 3 years ago

@mbusy have you compiled boost by yourself?

mbusy commented 3 years ago

@robertanto

I will try by using directly the Jetpack 3.3 as done by @mbusy.

Good idea, I'm not sure to understand how using a docker container would cause that problem, but that's the only difference between your setup and ours

A solution may be to provide your official container image for the Jetson Platform. In this way, developers can directly pull that image from the Docker Hub even on recent JetPacks avoiding problems!

Indeed, the idea is interesting and we might upload that on docker hub

@mbusy have you compiled boost by yourself?

Not at all, the instructions and patches that I gave should be sufficient for you to mirror our setup (provided that you use JetPack 3.3 (Ubuntu 16.04))

robertanto commented 3 years ago

Thank you again to both!

I will let you know!

robertanto commented 3 years ago

@mbusy @nyibbang

It worked with the Jet pack 3.3.3.

I installed boost 1.58.0 with:

sudo apt-get install libboost-all-dev

Then I followed the guide.

I will try to do the same in a container.

Thank you again!

robertanto commented 3 years ago

Now (using boost from apt) it works also on Docker.

Probably the problem was in the way in which I compiled boost or in the version itself.

robertanto commented 3 years ago

I have created a repository containing a faster way to use the library: https://github.com/robertanto/libqi-python-nvidia-jetson

It also works with the latest JetPack (4.4.1).

I hope that this repository can help researchers in their activities.