search

termux / termux-packages

A package build system for Termux.
https://termux.dev
Other
13.01k stars 2.99k forks source link

[Bug]: Grpcio fails to builds #17583

Closed connekitius closed 9 months ago

connekitius commented 1 year ago

Problem description

gRPCio fails to build when installed via pip install grpcio or installed by any package who depends on grpcio. The last command before it fails is:

aarch64-linux-android-clang -dndebug -g -fwrapv -o3 -wall -fstack-protector-strong -o3 -fstack-protector-strong -o3 -fpic -i/data/data/com.termux/files/usr/include/python3.11 -c /data/data/com.termux/files/usr/tmp/tmpu48_biu1/a.c -o none/data/data/com.termux/files/usr/tmp/tmpu48_biu1/a.o

What steps will reproduce the bug?

python3 -m pip install grpcio

What is the expected behavior?

A successful installation

System information

termux-info:

Termux Variables:
TERMUX_API_VERSION=0.50.1
TERMUX_APK_RELEASE=F_DROID
TERMUX_APP_PACKAGE_MANAGER=apt
TERMUX_APP_PID=4860
TERMUX_IS_DEBUGGABLE_BUILD=0
TERMUX_MAIN_PACKAGE_FORMAT=debian
TERMUX_VERSION=0.118.0
Packages CPU architecture:
aarch64
Subscribed repositories:
# sources.list
deb https://mirror.accum.se/mirror/termux.dev/termux-main stable main
# x11-repo (sources.list.d/x11.list)
deb https://mirror.accum.se/mirror/termux.dev/termux-x11 x11 main
Updatable packages:
All packages up to date
termux-tools version:
1.38.5
Android version:
9
Kernel build information:
Linux localhost 4.4.147+ #1 SMP PREEMPT Tue Nov 3 16:16:50 CST 2020 aarch64 Android
Device manufacturer:
LENOVO
Device model:
Lenovo L19111
LD Variables:
LD_LIBRARY_PATH=
LD_PRELOAD=/data/data/com.termux/files/usr/lib/libtermux-exec.so
Installed termux plugins:
com.termux.api versionCode:51

Clang version:

clang version 16.0.6
Target: aarch64-unknown-linux-android24
Thread model: posix
InstalledDir: /data/data/com.termux/files/usr/bin
connekitius commented 1 year ago

Python error log: grpcio.error.txt

romanovj commented 1 year ago 
error: "__ANDROID_API__ must be defined for Android builds."
error: "Requires Android API v21 and above"

try this

CFLAGS+=" -U__ANDROID_API__ -D__ANDROID_API__=28" pip install grpcio

jtplatt99 commented 1 year ago

I was experiencing the same issues installing grpcio on Android 8.0. The advice from @romanovj got me a bit further, but I still had problems with compilation steps further down the line, mostly with building third party libraries. The install command that finally ended up working for me was:

GRPC_PYTHON_DISABLE_LIBC_COMPATIBILITY=1 GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 GRPC_PYTHON_BUILD_SYSTEM_CARES=1 CFLAGS+=" -U__ANDROID_API__ -D__ANDROID_API__=26 -include unistd.h" LDFLAGS+=" -llog" pip install grpcio

Just make sure you have the headers for openssl, zlib, and c-ares in /data/data/com.termux/files/usr/include.

bongkiel commented 10 months ago

I was experiencing the same issues installing grpcio on Android 8.0. The advice from @romanovj got me a bit further, but I still had problems with compilation steps further down the line, mostly with building third party libraries. The install command that finally ended up working for me was:

GRPC_PYTHON_DISABLE_LIBC_COMPATIBILITY=1 GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 GRPC_PYTHON_BUILD_SYSTEM_CARES=1 CFLAGS+=" -U__ANDROID_API__ -D__ANDROID_API__=26 -include unistd.h" LDFLAGS+=" -llog" pip install grpcio

Just make sure you have the headers for openssl, zlib, and c-ares in /data/data/com.termux/files/usr/include.

thanks for you termux lightnovel-crawler working

noobyzy commented 10 months ago

I was experiencing the same issues installing grpcio on Android 8.0. The advice from @romanovj got me a bit further, but I still had problems with compilation steps further down the line, mostly with building third party libraries. The install command that finally ended up working for me was:

GRPC_PYTHON_DISABLE_LIBC_COMPATIBILITY=1 GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 GRPC_PYTHON_BUILD_SYSTEM_CARES=1 CFLAGS+=" -U__ANDROID_API__ -D__ANDROID_API__=26 -include unistd.h" LDFLAGS+=" -llog" pip install grpcio

Just make sure you have the headers for openssl, zlib, and c-ares in /data/data/com.termux/files/usr/include.

Hi guys, I encountered similar problems, your solution does not apply to me. I post the description in #18444 , the following is a short snip of my output

  aarch64-linux-android-clang -DNDEBUG -g -fwrapv -O3 -Wall -fstack-protector-strong -O3 -fstack-protector-strong -O3 -U__ANDROID_API__ -D__ANDROID_API__=28 -fPIC -I/data/data/com.termux/files/usr/include/python3.11 -c /data/data/com.termux/files/usr/tmp/tmplxfh016q/a.c -o None/data/data/com.termux/files/usr/tmp/tmplxfh016q/a.o
      Traceback (most recent call last):
        File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/setuptools/_distutils/unixccompiler.py", line 185, in _compile
          self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs)
        File "/data/data/com.termux/files/usr/tmp/pip-install-t_dlesqn/grpcio_23481140d6e54aad8a262db7e50486d6/src/python/grpcio/_spawn_patch.py", line 58, in _commandfile_spawn
          _classic_spawn(self, command, **kwargs)
        File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/setuptools/_distutils/ccompiler.py", line 1041, in spawn
          spawn(cmd, dry_run=self.dry_run, **kwargs)
        File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/setuptools/_distutils/spawn.py", line 70, in spawn
          raise DistutilsExecError(
      distutils.errors.DistutilsExecError: command '/data/data/com.termux/files/usr/bin/aarch64-linux-android-clang' failed with exit code 1

Can you help a little bit with that? Thanks a lot

jtplatt99 commented 10 months ago

@noobyzy Based on your logs it looks like the installer is still trying to build c-ares and failing. You mentioned that you tried several installation methods. What is the exact command you used that resulted in the log messages you posted here? Could you also post the result of running ls /data/data/com.termux/files/usr/include?

noobyzy commented 10 months ago

@noobyzy Based on your logs it looks like the installer is still trying to build c-ares and failing. You mentioned that you tried several installation methods. What is the exact command you used that resulted in the log messages you posted here? Could you also post the result of running ls /data/data/com.termux/files/usr/include?

@jtplatt99 Thanks.

The command that gives this output is

CFLAGS+=" -U__ANDROID_API__ -D__ANDROID_API__=28"  pip install grpcio

Listing all the files and directories in the path, I have

FLAC                   byteswap.h  dlfcn.h           ffitarget.h   gpg-error.h         krb5.h           libssh2_publickey.h  lzma.h         netinet             pcrecpp.h       riscv64-linux-android  srt            threads.h        uninorm.h                x264.h
FLAC++                 bzlib.h     drm               fmt123.h      gpgrt.h             lame             libssh2_sftp.h       lzo            netpacket           pcrecpparg.h    rmd160.h               stdatomic.h    tiff.h           unistd.h                 x264_config.h
OMXAL                  c++         editline          fnmatch.h     graphite2           langinfo.h       libswresample        malloc.h       nettle              pcreposix.h     sched.h                stdint.h       tiffconf.h       unistdio.h               x265.h
SLES                   cairo       elf.h             fontconfig    grp.h               lastlog.h        libswscale           math.h         nghttp2             pixman-1        scsi                   stdio.h        tiffio.h         unistr.h                 x265_config.h
X11                    camera      endian.h          form.h        gssapi              lcms2.h          libv4l-plugin.h      md2.h          nl_types.h          png.h           search.h               stdio_ext.h    tiffio.hxx       unistring                x86_64-linux-android
aarch64-linux-android  cap-ng.h    err.h             freetype2     gssapi.h            lcms2_plugin.h   libv4l1-videodev.h   md4.h          npth.h              pngconf.h       semaphore.h            stdlib.h       tiffvers.h       unitypes.h               xcb
aaudio                 clang       errno.h           fribidi       gssrpc              ldns             libv4l1.h            md5.h          ocl_icd.h           pnglibconf.h    setjmp.h               string.h       time.h           uniwbrk.h                xen
absl                   clang-c     error.h           fts.h         harfbuzz            libavcodec       libv4l2.h            media          ogg                 poll.h          sha.h                  strings.h      time64.h         uniwidth.h               xlocale.h
alloca.h               clang-tidy  eti.h             ftw.h         histedit.h          libavdevice      libv4l2rds.h         memory.h       openblas            proc            sha1.h                 svt-av1        tirpc            utime.h                  xvid.h
amidi                  com_err.h   evdns.h           gawkapi.h     i686-linux-android  libavfilter      libv4lconvert.h      menu.h         opencore-amrnb      profile.h       sha2.h                 syn123.h       turbojpeg.h      utmp.h                   xxh3.h
android                complex.h   event.h           gcrypt.h      iconv.h             libavformat      libxml2              misc           opencore-amrwb      pthread.h       sha256.h               sys            uchar.h          utmpx.h                  xxhash.h
aom                    cpio.h      event2            gdbm.h        idn2.h              libavutil        limits.h             mntent.h       opencv4             pty.h           sha512.h               syscall.h      uconfig_local.h  verto-module.h           zconf.h
apt-pkg                crypt.h     evhttp.h          getopt.h      ifaddrs.h           libbluray        link.h               mpf2mpfr.h     openjpeg-2.5        pulse           signal.h               sysexits.h     ucontext.h       verto.h                  zdict.h
ar.h                   ctype.h     evrpc.h           gif_lib.h     inttypes.h          libcharset.h     linux                mpfr.h         openssl             pwd.h           sndfile.h              syslog.h       udfread          vid.stab                 zimg++.hpp
arm-linux-androideabi  curl        evutil.h          gio-unix-2.0  jconfig.h           libgen.h         lld                  mpg123.h       opus                python3.11      sndfile.hh             tar.h          unbound-event.h  video                    zimg.h
arpa                   curses.h    execinfo.h        glib-2.0      jerror.h            libimagequant.h  llvm                 mtd            out123.h            raqm-version.h  sodium                 term.h         unbound.h        vo-amrwbenc              zlib.h
asm-generic            dav1d       expat.h           glob.h        jmorecfg.h          libintl.h        llvm-c               ncurses        panel.h             raqm.h          sodium.h               term_entry.h   unctrl.h         vorbis                   zmq.h
ass                    db.h        expat_config.h    gme           jni.h               libopenmpt       localcharset.h       ncurses.h      paths.h             rav1e           sound                  termcap.h      unicase.h        vpx                      zmq_utils.h
assert.h               db_185.h    expat_external.h  gmp.h         jpeglib.h           libpng16         locale.h             ncurses_dll.h  pcre.h              rdma            soxr-lsr.h             termio.h       uniconv.h        wait.h                   zstd.h
assuan.h               db_cxx.h    fcntl.h           gmpxx.h       kadm5               libpostproc      lz4.h                ncursesw       pcre2.h             readline        soxr.h                 termios.h      unictype.h       wchar.h                  zstd_errors.h
bash                   dbm.h       features.h        gnumake.h     kdb.h               libsmartcols     lz4frame.h           ndbm.h         pcre2posix.h        regex.h         speex                  termux-api.h   unigbrk.h        wctype.h
bits                   dbus-1.0    fenv.h            gnutls        krad.h              libssh           lz4hc.h              net            pcre_scanner.h      resolv.h        sqlite3.h              termux-auth.h  unilbrk.h        webp
brotli                 dirent.h    ffi.h             google        krb5                libssh2.h        lzma                 netdb.h        pcre_stringpiece.h  ripemd.h        sqlite3ext.h           theora         uniname.h        webrtc_audio_processing

I don't see c-ares here. Is the problem with that? Do you know how to fix this?

jtplatt99 commented 10 months ago

Yes, if c-ares was installed you would see the file ares.h in that directory. You should be able to install that using:

pkg install c-ares

You'll also need to instruct pip to build using the system c-ares library instead of trying to built it on its own. To do that, try running the command I originally posted again and hopefully that should get you a bit further through the compilation.

GRPC_PYTHON_DISABLE_LIBC_COMPATIBILITY=1 GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 GRPC_PYTHON_BUILD_SYSTEM_CARES=1 CFLAGS+=" -U__ANDROID_API__ -D__ANDROID_API__=26 -include unistd.h" LDFLAGS+=" -llog" pip install grpcio

Since you're on Android 11 instead of 8 you might also want to change the -D__ANDROID_API__=26 to -D__ANDROID_API__=30, but I'm not sure if that matters or not.

noobyzy commented 10 months ago

Thanks, your solution did work. I use

GRPC_PYTHON_DISABLE_LIBC_COMPATIBILITY=1 GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 GRPC_PYTHON_BUILD_SYSTEM_CARES=1 CFLAGS+=" -U__ANDROID_API__ -D__ANDROID_API__=30 -include unistd.h" LDFLAGS+=" -llog" pip install grpcio
ayoubelmhamdi commented 3 days ago

My solution for installing PyPI packages using pip in a virtual environment in Termux:

For example, to install the foo package that requires grpcio:

pkg install python-grpcio
pip show grpcio  # Check the path of "grpcio" in the system for later use

python3 -m venv venv
source venv/bin/activate
export PYTHONPATH="$PYTHONPATH:/data/data/com.termux/files/usr/lib/python3.11/site-packages"

pip install foo

but, we can use it with direnv and make it easy to load the $PYTHONPATH