search

yifeikong / curl-impersonate

An active fork of curl-impersonate with more versions and build targets.
MIT License
48 stars 10 forks source link

Build for armv7(?) Android Termux #51

Open h-a-s-k opened 4 months ago

h-a-s-k commented 4 months ago

Tried adding this to build-and-test-make.yml but it errors out when building zlib

          - arch: arm
            target: arm-linux-gnueabihf.2.17
            host: arm-linux-gnueabihf
            capture_interface: eth0
            cflags: "-mfpu=vfp -mfloat-abi=soft -mcpu=cortex-a53"

arm-build

bjia56 commented 4 months ago

The existing arm builds are compatible with both armv6 and armv7 (Raspberry Pi models), with NEON selectively used in BoringSSL's crypto routines based on CPU feature detection. I am not sure if we need a separate build for armv7, however the issue is you should use -mcpu=cortex_a53 (note the underscore) with the Zig compiler.

h-a-s-k commented 4 months ago

The issue is that whenever I try to run your arm compiled binaries on a arm test phone I got through termux, it does not work. So I've been trying different things.

Some info for reference:

~ $ ./curl_chrome120 https://www.google.com ./curl_chrome120: line 10: ./curl-impersonate-chrome: cannot execute: required file not found ~ $ ./curl-impersonate-chrome https://www.google.com bash: ./curl-impersonate-chrome: cannot execute: required file not found

Your compiled binary ~ $ readelf -h ./curl-impersonate-chrome

ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x9cf80
  Start of program headers:          52 (bytes into file)
  Start of section headers:          2276616 (bytes into file)
  Flags:                             0x5000400, Version5 EABI, hard-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         12
  Size of section headers:           40 (bytes)
  Number of section headers:         29
  Section header string table index: 28

Termux binary: ~ $ readelf -h /data/data/com.termux/files/usr/bin/curl

ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x1ea90
  Start of program headers:          52 (bytes into file)
  Start of section headers:          201788 (bytes into file)
  Flags:                             0x5000200, Version5 EABI, soft-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         11
  Size of section headers:           40 (bytes)
  Number of section headers:         26
  Section header string table index: 25

$ cat /proc/cpuinfo

Processor       : AArch64 Processor rev 4 (aarch64)
processor       : 0
model name      : ARMv8 Processor rev 4 (v8l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 1
model name      : ARMv8 Processor rev 4 (v8l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 2
model name      : ARMv8 Processor rev 4 (v8l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 3
model name      : ARMv8 Processor rev 4 (v8l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 4
model name      : ARMv8 Processor rev 4 (v8l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 5
model name      : ARMv8 Processor rev 4 (v8l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 6
model name      : ARMv8 Processor rev 4 (v8l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 7
model name      : ARMv8 Processor rev 4 (v8l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

Hardware        : Qualcomm Technologies, Inc SDM439

I appreciate the yml correction, it now errors at brotli:

Run make chrome-build
curl -L "https://github.com/google/brotli/archive/refs/tags/v1.0.9.tar.gz" \
    -o "brotli-1.0.9.tar.gz"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  475k  100  475k    0     0  1909k      0 --:--:-- --:--:-- --:--:-- 1909k
tar xf brotli-1.0.9.tar.gz
cd brotli-1.0.9
mkdir -p out
cd out
# Convert autoconf style os name to CMake style os name.
case linux-gnueabihf in           \
  linux*)                    \
    system_name=Linux        \
    ;;                       \
  darwin*)                   \
    system_name=Darwin       \
    ;;                       \
  *)                         \
    system_name=linux-gnueabihf   \
    ;;                       \
esac
cmake -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_INSTALL_PREFIX=./installed \
        -DCMAKE_INSTALL_LIBDIR=lib \
        -DCMAKE_CXX_COMPILER=/home/runner/work/curl-impersonate/curl-impersonate/zigshim/cxx \
        -DCMAKE_C_COMPILER=/home/runner/work/curl-impersonate/curl-impersonate/zigshim/cc \
        -DCMAKE_C_FLAGS="-mfpu=vfp -mfloat-abi=soft -mcpu=cortex_a53" \
        -DCMAKE_SYSTEM_NAME=$system_name \
        -DCMAKE_SYSTEM_PROCESSOR=arm \
        ..
cmake --build . --config Release --target install --parallel 4
CMake Deprecation Warning at CMakeLists.txt:5 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The C compiler identification is Clang 16.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/runner/work/curl-impersonate/curl-impersonate/zigshim/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Build type is 'Release'
-- Performing Test BROTLI_EMSCRIPTEN
-- Performing Test BROTLI_EMSCRIPTEN - Failed
-- Compiler is not EMSCRIPTEN
-- Looking for log2
-- Looking for log2 - found
CMake Warning at CMakeLists.txt:319 (message):
  Test file tests/testdata/alice29.txt does not exist.

CMake Warning at CMakeLists.txt:319 (message):
  Test file tests/testdata/asyoulik.txt does not exist.

CMake Warning at CMakeLists.txt:319 (message):
  Test file tests/testdata/lcet10.txt does not exist.

CMake Warning at CMakeLists.txt:319 (message):
  Test file tests/testdata/plrabn12.txt does not exist.

-- Configuring done (1.4s)
-- Generating done (0.0s)
-- Build files have been written to: /home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out
gmake[1]: Entering directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[2]: Entering directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Entering directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Entering directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Leaving directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Entering directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Leaving directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Entering directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
[  1%] Building C object CMakeFiles/brotlicommon.dir/c/common/constants.c.o
[  3%] Building C object CMakeFiles/brotlicommon.dir/c/common/context.c.o
[  4%] Building C object CMakeFiles/brotlicommon.dir/c/common/dictionary.c.o
[  6%] Building C object CMakeFiles/brotlicommon-static.dir/c/common/constants.c.o
[  7%] Building C object CMakeFiles/brotlicommon.dir/c/common/platform.c.o
[  9%] Building C object CMakeFiles/brotlicommon.dir/c/common/transform.c.o
[ 10%] Building C object CMakeFiles/brotlicommon-static.dir/c/common/context.c.o
[ 12%] Building C object CMakeFiles/brotlicommon-static.dir/c/common/dictionary.c.o
[ 13%] Building C object CMakeFiles/brotlicommon-static.dir/c/common/platform.c.o
[ 15%] Building C object CMakeFiles/brotlicommon-static.dir/c/common/transform.c.o
[ 16%] Linking C shared library libbrotlicommon.so
[ 18%] Linking C static library libbrotlicommon-static.a
gmake[3]: Leaving directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
[ 18%] Built target brotlicommon-static
gmake[3]: Entering directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Entering directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Leaving directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Leaving directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Entering directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[3]: Entering directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
[ 19%] Building C object CMakeFiles/brotlienc-static.dir/c/enc/backward_references.c.o
[ 21%] Building C object CMakeFiles/brotlienc-static.dir/c/enc/backward_references_hq.c.o
[ 22%] Building C object CMakeFiles/brotlidec-static.dir/c/dec/bit_reader.c.o
[ 24%] Building C object CMakeFiles/brotlidec-static.dir/c/dec/decode.c.o
zig: error: clang frontend command failed with exit code 133 (use -v to see invocation)
clang version 16.0.6 (https://github.com/ziglang/zig-bootstrap 1dda86241204c4649f668d46b6a37feed707c7b4)
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
zig: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
zig: note: diagnostic msg: /tmp/backward_references_hq-ec2dac.c
zig: note: diagnostic msg: /tmp/backward_references_hq-ec2dac.sh
zig: note: diagnostic msg: 

********************
gmake[3]: *** [CMakeFiles/brotlienc-static.dir/build.make:90: CMakeFiles/brotlienc-static.dir/c/enc/backward_references_hq.c.o] Error 1
gmake[3]: *** Waiting for unfinished jobs....
[ 25%] Building C object CMakeFiles/brotlidec-static.dir/c/dec/huffman.c.o
[ 27%] Building C object CMakeFiles/brotlidec-static.dir/c/dec/state.c.o
zig: error: clang frontend command failed with exit code 133 (use -v to see invocation)
clang version 16.0.6 (https://github.com/ziglang/zig-bootstrap 1dda86241204c4649f668d46b6a37feed707c7b4)
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
zig: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
zig: note: diagnostic msg: /tmp/state-eb3d12.c
zig: note: diagnostic msg: /tmp/state-eb3d12.sh
zig: note: diagnostic msg: 

********************
gmake[3]: *** [CMakeFiles/brotlidec-static.dir/build.make:118: CMakeFiles/brotlidec-static.dir/c/dec/state.c.o] Error 1
gmake[3]: *** Waiting for unfinished jobs....
zig: error: clang frontend command failed with exit code 133 (use -v to see invocation)
clang version 16.0.6 (https://github.com/ziglang/zig-bootstrap 1dda86241204c4649f668d46b6a37feed707c7b4)
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
zig: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
zig: note: diagnostic msg: /tmp/decode-670617.c
zig: note: diagnostic msg: /tmp/decode-670617.sh
zig: note: diagnostic msg: 

********************
gmake[3]: *** [CMakeFiles/brotlidec-static.dir/build.make:90: CMakeFiles/brotlidec-static.dir/c/dec/decode.c.o] Error 1
gmake[3]: Leaving directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[2]: *** [CMakeFiles/Makefile2:227: CMakeFiles/brotlidec-static.dir/all] Error 2
gmake[2]: *** Waiting for unfinished jobs....
gmake[3]: Leaving directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[2]: *** [CMakeFiles/Makefile2:253: CMakeFiles/brotlienc-static.dir/all] Error 2
gmake[3]: Leaving directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
[ 27%] Built target brotlicommon
gmake[2]: Leaving directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
gmake[1]: *** [Makefile:146: all] Error 2
gmake[1]: Leaving directory '/home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out'
make: *** [Makefile:178: /home/runner/work/curl-impersonate/curl-impersonate/brotli-1.0.9/out/installed/lib/libbrotlicommon-static.a] Error 2
Error: Process completed with exit code 2.
bjia56 commented 4 months ago

Ah, I see you're building for Termux. Android's arm environment is not very compatible with the Linux builds, notably the glibc is either missing or located at a different path. There was some related investigation over on https://github.com/yifeikong/curl_cffi/issues/74

bjia56 commented 4 months ago

Some resources that look useful:

I wonder if using patchelf to edit the dynamic interpreter and libc paths for the aarch64 Linux builds would be sufficient. If not, perhaps will need to build inside a Termux environment.

h-a-s-k commented 4 months ago

Oh boy that's a bit above my knowledge of compiling things, appreciate it regardless

yifeikong commented 4 months ago

AFAIK, this should be resovled by building with Android's NDK, potentially using zig cc.

The official doc: https://curl.se/docs/install.html

bjia56 commented 4 months ago

Looks like that Zig example got it working with just musl, so maybe the NDK is not required