search

termux / termux-packages

A package build system for Termux.
https://termux.dev
Other
12.94k stars 2.98k forks source link

[Bug]: build-package.sh: cannot compile subpackages #10462

Open Maxython opened 2 years ago

Maxython commented 2 years ago

Problem description

Now I'm setting up build-bootstraps.sh to support compiling bootstrap which will be based on pacman. During testing, a bug was noticed, it appears when the compiler reaches the bzip2 package.

Error message (testing happens in docker):

[*] Building 'bzip2'...
ERROR: No package bzip2 found in any of the enabled repositories. Are you trying to set up a custom repository?
[*] Building 'bzip2' exited with exit code 1
Failed to build package 'bzip2' for arch 'aarch64' in format 'pacman'

In fact, this can be fixed in build-bootstraps.sh, but I don't know if this will be correct.

What steps will reproduce the bug?

Command:

./scripts/build-bootstraps.sh --pm pacman --architectures aarch64

What is the expected behavior?

No response

System information

On all devices

Maxython commented 2 years ago

I will try to fix this bug in build-bootstraps.sh.

agnostic-apollo commented 2 years ago

The issue exists because build-bootstraps.sh currently doesn't support building subpackages and the three subpackages bzip2, curl and xz-utils builds were ignored but it wasn't an issue at the time of the prototype since their parent packages were dependencies of apt and so would automatically get included.

There was a note for a related issue as well at https://github.com/termux/termux-packages/blob/44572292816c5eca4eca32ec66068af5456255a3/scripts/build-bootstraps.sh#L80

Previously, the builds would complete because No build.sh script at package dir errors were ignored but recently the monorepo commit also added a precheck for the TERMUX_PKG_BUILDER_DIR itself, which changed the error thrown so it wasn't ignored and building failed.

https://github.com/termux/termux-packages/commit/f0ea9b922bf2b4a642ba78136cd01d46b5e0f624#diff-731444686e10066f1aeb6193b5c45aa13844d82e0f54de5aec3c325402972f2bR472

However, at this stage, this will need to fixed, since parent packages may not always be dependencies of other packages.

So firstly the build-bootstrap.sh will need to check if build.sh is not found, then try checking if its a subpackage, and then build its parent, basically, automatically build libbz2 instead of bzip2. But that creates an issue, we actually don't want the debs of parent packages to be added to bootstrap and only the subpackage and their dependencies. So we need to know exactly what package dependencies will be built for the parent package and the subpackage and only include the debs of subpackage and its dependencies in the bootstrap. The parent package and their dependencies could also be included if parent package is itself a dependency of another package, like libbz2 is of apt. I need to refactor the build-boostraps.sh and generate-bootstraps.sh scripts, will look into fixing this too. It may be possible to use buildorder.py somehow.

Maxython commented 2 years ago

Ok, I see

stale[bot] commented 2 years ago

This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

truboxl commented 2 years ago

This also affects normal apt based packaging fyi

I think temporarily patching build-bootstraps.sh for now is better than having it broken for a long time...

agnostic-apollo commented 2 years ago

This also affects normal apt based packaging fyi

The bootstraps generated for apps are done with generate-bootstraps.sh instead.

https://github.com/termux/termux-packages/wiki/For-maintainers#bootstraps

Also check https://github.com/termux/termux-app/issues/2892#issuecomment-1186482737

I plan on working more on it soon before next termux-app release. Both scripts require lot of changes.

truboxl commented 2 years ago

Actually I have been experimenting build-bootstraps.sh for arm vfpv3-d16 for some time so I can't use generate-bootsraps.sh

I kept retrying (by updating packages that have been outdated) until finally reach this error only to see its been reported a long time ago (but its definitely not related to using pacman)

agnostic-apollo commented 2 years ago

You can use changes in #10540 for now.

stale[bot] commented 1 year ago

This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

diyism commented 1 year ago

You can use changes in #10540 for now.

I have tested #10540, it shows:

$ ./build-bootstraps.sh --architectures aarch64

[*] Building 'apt'...
apt@2.5.2-3 built - skipping (rm /data/data/.built-packages/apt to force rebuild)
[*] Building 'apt' exited with exit code 0

[*] Building 'bash'...
bash@5.1.16-1 built - skipping (rm /data/data/.built-packages/bash to force rebuild)
[*] Building 'bash' exited with exit code 0

[*] Building 'libbz2'...
libbz2@1.0.8-6 built - skipping (rm /data/data/.built-packages/libbz2 to force rebuild)
[*] Building 'libbz2' exited with exit code 0

[*] Building 'command-not-found'...
termux - building command-not-found for arch aarch64...
Downloading https://github.com/termux/command-not-found/archive/2.1.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100  8184    0  8184    0     0   1950      0 --:--:--  0:00:04 --:--:--  4019
Downloading https://nodejs.org/dist/v16.14.0/node-v16.14.0-linux-x64.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20.9M  100 20.9M    0     0  26109      0  0:13:59  0:13:59 --:--:-- 10593
-- Android: Targeting API '24' with architecture 'arm64', ABI 'arm64-v8a', and processor 'aarch64'
-- The C compiler identification is Clang 14.0.6
-- The CXX compiler identification is Clang 14.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/builder/.termux-build/_cache/android-r25b-api-24-v0/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/builder/.termux-build/_cache/android-r25b-api-24-v0/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    BUILD_TESTING
    CMAKE_FIND_ROOT_PATH
    CMAKE_FIND_ROOT_PATH_MODE_LIBRARY
    CMAKE_INSTALL_LIBDIR
    CMAKE_USE_SYSTEM_LIBRARIES
    DOXYGEN_EXECUTABLE

-- Build files have been written to: /home/builder/.termux-build/command-not-found/build
[1/3] cd /home/builder/.termux-build/command-not-found/src && ./generate-db.js
[2/3] Building CXX object CMakeFiles/command-not-found.dir/command-not-found.cpp.o
FAILED: CMakeFiles/command-not-found.dir/command-not-found.cpp.o 
/home/builder/.termux-build/_cache/android-r25b-api-24-v0/bin/clang++ --target=aarch64-none-linux-android --gcc-toolchain=/home/builder/.termux-build/_cache/android-r25b-api-24-v0 --sysroot=/home/builder/.termux-build/_cache/android-r25b-api-24-v0/sysroot   -fstack-protector-strong -Oz --target=aarch64-linux-android24  -I/data/data/com.termux/files/usr/include -O3 -DNDEBUG -fPIE -Wall -Wextra -Werror -pedantic -MD -MT CMakeFiles/command-not-found.dir/command-not-found.cpp.o -MF CMakeFiles/command-not-found.dir/command-not-found.cpp.o.d -o CMakeFiles/command-not-found.dir/command-not-found.cpp.o -c /home/builder/.termux-build/command-not-found/src/command-not-found.cpp
/home/builder/.termux-build/command-not-found/src/command-not-found.cpp:30:11: fatal error: 'commands-aarch64-termux-main.h' file not found
# include "commands-aarch64-termux-main.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
ninja: build stopped: subcommand failed.
[*] Building 'command-not-found' exited with exit code 1
Failed to build package 'command-not-found' for arch 'aarch64' in format 'debian'
builder@86fd6add4010:~/termux-packages/scripts$ ls /home/builder/.termux-build/command-not-found/src/command-not-found.cpp
/home/builder/.termux-build/command-not-found/src/command-not-found.cpp

Any idea?

agnostic-apollo commented 1 year ago

https://github.com/termux/command-not-found/blob/e187c1099fd6ef22524e2ca2ce7314f12065a593/command-not-found.cpp#L30

cc: @thunder-coding

thunder-coding commented 1 year ago

I can't reproduce the behaviour, atleast when building just command-not-found. Can you try building just command-not-found? Also, make sure you're able to ping packages-cf.termux.org, and make HTTPS connections with the site.

diyism commented 1 year ago

I can't reproduce the behaviour, atleast when building just command-not-found. Can you try building just command-not-found? Also, make sure you're able to ping packages-cf.termux.org, and make HTTPS connections with the site.

You're right, it's my network problem, I tried again, it can continue now.

diyism commented 1 year ago

Sorry for bothering you again, with: TERMUX_PKG_ESSENTIAL=true ./termux-packages/scripts/run-docker.sh ./scripts/build-bootstraps.10540.sh --architectures aarch64 I have successfully built the "./termux-packages/bootstrap-aarch64.zip", but its size is about 242MB, while the bootstrap-aarch64.zip in releases page(https://github.com/termux/termux-packages/releases/tag/bootstrap-2022.09.18-r1%2Bapt-android-7) is only 25MB,

Is there a special building parameter for the slim recipe?

Grimler91 commented 1 year ago

@diyism well what does ./scripts/build-bootstraps.10540.sh contain? Which packages have you added? Possibly added packages (and their dependencies) bring bootstraps to 242 mb

truboxl commented 1 year ago

We use generate-bootstraps.sh which only packs the essential packages

https://github.com/termux/termux-packages/blob/9381ffe8b5c0d6c66386504c6d52c3751ec7e3e0/.github/workflows/bootstrap_archives.yml#L30

https://github.com/termux/termux-packages/blob/9381ffe8b5c0d6c66386504c6d52c3751ec7e3e0/scripts/generate-bootstraps.sh#L429-L472

But generate-bootstraps.sh pulls packages from server instead of locally built packages, that's where #11491 comes in However generate-bootstraps.sh does not keep Unix permission so some packages do not work, https://github.com/termux/termux-packages/issues/11844#issuecomment-1239713251

segfault-bilibili commented 1 year ago

@diyism I encountered both of your problems. Maybe I have to give up previous progress to fix the missing "commands-aarch64-termux-main.h" file? ~I still have no idea - it would be helpful if I know exactly what went wrong. Tried to fix this in https://github.com/termux/command-not-found/pull/17~ Not a real fix, still thinking about how to handle this.

I added nothing but nodejs - which then seems to bring many extra packages - however it seems quite weird that some seemingly unrelated packages like "tor" are also included. Have no idea what went wrong either.

segfault-bilibili commented 1 year ago

By the way I'm using the patch from #10540 as well.

stale[bot] commented 1 year ago

This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

hnmn commented 1 year ago

Is any plan to address these bugs both issues are still there with https://github.com/termux/termux-packages/pull/10540 also

/home/builder/.termux-build/command-not-found/src/command-not-found.cpp:30:11: fatal error: 'commands-aarch64-termux-main.h' file not found
# include "commands-aarch64-termux-main.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

and clang and tor like packages are still building when running build-bootstraps.sh

QGB commented 1 year ago 
docker run --name=unruffled_williamson --hostname=ded0bdd038e4 --user=builder:builder --mac-address=02:42:ac:11:00:06 --volume=/:/all --volume=/root/docker/termux-package-builder/home_builder:/home/builder --volume=/home/qgb/github/termux_python3.9.7:/home/builder/termux-packages --volume=/root/docker/termux-package-builder/usr:/usr --workdir=/home/builder/termux-packages -p 397:397 --restart=no --runtime=runc -t termux/package-builder /bin/bash
QGB commented 1 year ago

./build-package.sh -a arm ncurses

thunder-coding commented 1 year ago

@QGB please don't spam random shell commands here.

Agono0 commented 1 year ago

@agnostic-apollo the error still showing (lastest repo clone) , and i still cannot build using build-bootstraps.sh as it show:

[*] Building 'bzip2'...
ERROR: No package bzip2 found in any of the enabled repositories. Are you trying to set up a custom repository?
[*] Building 'bzip2' exited with exit code 1
Failed to build package 'bzip2' for arch 'aarch64'

and yeah i am trying to build package locally. if i use generate-bootstraps.sh , it generate a small bootstrap.zip file (150KB) it contain var/lib/dpkg/info/grep.list and more like that.... is this because this warning

curl: (22) The requested URL returned error: 404
[!] Skipping architecture-independent package list as not available...
[*] Downloading package list for architecture 'arm'...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--
...... some downloads .......
[*] Extracting 'unzip'...
[*] Creating 'bootstrap-arm.zip'...
  adding: etc/ (stored 0%)
  adding: etc/apt/ (stored 0%)
  adding: etc/apt/preferences.d/ (stored 0%)
  adding: etc/apt/apt.conf.d/ (stored 0%)
  adding: tmp/ (stored 0%)
  adding: var/ (stored 0%)
  adding: var/log/ (stored 0%)
  adding: var/log/apt/ (stored 0%)
  adding: var/lib/ (stored 0%)
  adding: var/lib/dpkg/ (stored 0%)
  adding: var/lib/dpkg/status (deflated 77%)
  adding: var/lib/dpkg/updates/ (stored 0%)
  adding: var/lib/dpkg/triggers/ (stored 0%)
  adding: var/lib/dpkg/info/ (stored 0%)
  adding: var/lib/dpkg/info/util-linux.list (deflated 91%)
  adding: var/lib/dpkg/info/findutils.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/bzip2.list (deflated 86%)
  adding: var/lib/dpkg/info/command-not-found.md5sums (deflated 12%)
  adding: var/lib/dpkg/info/termux-licenses.list (deflated 87%)
  adding: var/lib/dpkg/info/readline.conffiles (deflated 5%)
  adding: var/lib/dpkg/info/libgmp.list (deflated 84%)
  adding: var/lib/dpkg/info/dos2unix.md5sums (deflated 66%)
  adding: var/lib/dpkg/info/libunistring.md5sums (deflated 72%)
  adding: var/lib/dpkg/info/inetutils.md5sums (deflated 66%)
  adding: var/lib/dpkg/info/termux-tools.list (deflated 88%)
  adding: var/lib/dpkg/info/libssh2.list (deflated 91%)
  adding: var/lib/dpkg/info/libsmartcols.list (deflated 79%)
  adding: var/lib/dpkg/info/libtirpc.list (deflated 90%)
  adding: var/lib/dpkg/info/net-tools.md5sums (deflated 67%)
  adding: var/lib/dpkg/info/nano.conffiles (deflated 5%)
  adding: var/lib/dpkg/info/libnghttp2.md5sums (deflated 54%)
  adding: var/lib/dpkg/info/zstd.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/lsof.md5sums (deflated 42%)
  adding: var/lib/dpkg/info/libtirpc.md5sums (deflated 70%)
  adding: var/lib/dpkg/info/tar.list (deflated 84%)
  adding: var/lib/dpkg/info/libandroid-support.md5sums (deflated 48%)
  adding: var/lib/dpkg/info/resolv-conf.list (deflated 78%)
  adding: var/lib/dpkg/info/libnettle.list (deflated 90%)
  adding: var/lib/dpkg/info/procps.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/gawk.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/nano.list (deflated 90%)
  adding: var/lib/dpkg/info/ncurses.list (deflated 92%)
  adding: var/lib/dpkg/info/patch.list (deflated 79%)
  adding: var/lib/dpkg/info/libgmp.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/dialog.list (deflated 81%)
  adding: var/lib/dpkg/info/termux-am.md5sums (deflated 30%)
  adding: var/lib/dpkg/info/procps.list (deflated 88%)
  adding: var/lib/dpkg/info/gawk.list (deflated 88%)
  adding: var/lib/dpkg/info/sed.md5sums (deflated 43%)
  adding: var/lib/dpkg/info/net-tools.list (deflated 87%)
  adding: var/lib/dpkg/info/tar.md5sums (deflated 58%)
  adding: var/lib/dpkg/info/sed.list (deflated 82%)
  adding: var/lib/dpkg/info/liblzma.list (deflated 89%)
  adding: var/lib/dpkg/info/termux-am-socket.list (deflated 81%)
  adding: var/lib/dpkg/info/grep.md5sums (deflated 53%)
  adding: var/lib/dpkg/info/termux-exec.list (deflated 79%)
  adding: var/lib/dpkg/info/libgpg-error.list (deflated 85%)
  adding: var/lib/dpkg/info/libcap-ng.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/coreutils.md5sums (deflated 71%)
  adding: var/lib/dpkg/info/liblz4.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/gzip.md5sums (deflated 67%)
  adding: var/lib/dpkg/info/libidn2.list (deflated 88%)
  adding: var/lib/dpkg/info/ca-certificates.list (deflated 78%)
  adding: var/lib/dpkg/info/libassuan.list (deflated 82%)
  adding: var/lib/dpkg/info/libnpth.list (deflated 80%)
  adding: var/lib/dpkg/info/nano.md5sums (deflated 69%)
  adding: var/lib/dpkg/info/bash.list (deflated 89%)
  adding: var/lib/dpkg/info/libidn2.md5sums (deflated 70%)
  adding: var/lib/dpkg/info/libandroid-glob.md5sums (deflated 40%)
  adding: var/lib/dpkg/info/diffutils.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/libc++.md5sums (deflated 8%)
  adding: var/lib/dpkg/info/openssl.md5sums (deflated 71%)
  adding: var/lib/dpkg/info/termux-licenses.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/dpkg.md5sums (deflated 66%)
  adding: var/lib/dpkg/info/libmd.list (deflated 92%)
  adding: var/lib/dpkg/info/bash.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/zlib.list (deflated 82%)
  adding: var/lib/dpkg/info/resolv-conf.md5sums (deflated 32%)
  adding: var/lib/dpkg/info/diffutils.list (deflated 85%)
  adding: var/lib/dpkg/info/liblz4.list (deflated 84%)
  adding: var/lib/dpkg/info/less.md5sums (deflated 58%)
  adding: var/lib/dpkg/info/patch.md5sums (deflated 31%)
  adding: var/lib/dpkg/info/command-not-found.list (deflated 79%)
  adding: var/lib/dpkg/info/openssl.list (deflated 91%)
  adding: var/lib/dpkg/info/gzip.list (deflated 88%)
  adding: var/lib/dpkg/info/libassuan.md5sums (deflated 54%)
  adding: var/lib/dpkg/info/libgpg-error.md5sums (deflated 64%)
  adding: var/lib/dpkg/info/psmisc.list (deflated 85%)
  adding: var/lib/dpkg/info/less.list (deflated 84%)
  adding: var/lib/dpkg/info/lsof.list (deflated 81%)
  adding: var/lib/dpkg/info/libmpfr.list (deflated 85%)
  adding: var/lib/dpkg/info/pcre2.md5sums (deflated 74%)
  adding: var/lib/dpkg/info/libunbound.md5sums (deflated 69%)
  adding: var/lib/dpkg/info/curl.md5sums (deflated 31%)
  adding: var/lib/dpkg/info/libcurl.list (deflated 90%)
  adding: var/lib/dpkg/info/dialog.md5sums (deflated 32%)
  adding: var/lib/dpkg/info/dpkg.list (deflated 87%)
  adding: var/lib/dpkg/info/libgcrypt.list (deflated 84%)
  adding: var/lib/dpkg/info/libmpfr.md5sums (deflated 64%)
  adding: var/lib/dpkg/info/inetutils.list (deflated 87%)
  adding: var/lib/dpkg/info/libunbound.list (deflated 88%)
  adding: var/lib/dpkg/info/bash.conffiles (deflated 42%)
  adding: var/lib/dpkg/info/readline.list (deflated 88%)
  adding: var/lib/dpkg/info/libgnutls.list (deflated 87%)
  adding: var/lib/dpkg/info/libmd.md5sums (deflated 66%)
  adding: var/lib/dpkg/info/dash.md5sums (deflated 39%)
  adding: var/lib/dpkg/info/xxhash.md5sums (deflated 53%)
  adding: var/lib/dpkg/info/unzip.list (deflated 85%)
  adding: var/lib/dpkg/info/termux-am.list (deflated 81%)
  adding: var/lib/dpkg/info/libunistring.list (deflated 91%)
  adding: var/lib/dpkg/info/liblzma.md5sums (deflated 70%)
  adding: var/lib/dpkg/info/libssh2.md5sums (deflated 74%)
  adding: var/lib/dpkg/info/pcre2.list (deflated 91%)
  adding: var/lib/dpkg/info/zlib.md5sums (deflated 52%)
  adding: var/lib/dpkg/info/apt.md5sums (deflated 69%)
  adding: var/lib/dpkg/info/curl.list (deflated 79%)
  adding: var/lib/dpkg/info/findutils.list (deflated 84%)
  adding: var/lib/dpkg/info/xxhash.list (deflated 86%)
  adding: var/lib/dpkg/info/libnpth.md5sums (deflated 46%)
  adding: var/lib/dpkg/info/gpgv.md5sums (deflated 30%)
  adding: var/lib/dpkg/info/libevent.md5sums (deflated 69%)
  adding: var/lib/dpkg/info/debianutils.list (deflated 83%)
  adding: var/lib/dpkg/info/apt.conffiles (deflated 4%)
  adding: var/lib/dpkg/info/libandroid-support.list (deflated 80%)
  adding: var/lib/dpkg/info/termux-keyring.md5sums (deflated 63%)
  adding: var/lib/dpkg/info/termux-keyring.list (deflated 88%)
  adding: var/lib/dpkg/info/gpgv.list (deflated 79%)
  adding: var/lib/dpkg/info/readline.md5sums (deflated 67%)
  adding: var/lib/dpkg/info/libgnutls.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/dash.list (deflated 83%)
  adding: var/lib/dpkg/info/libgcrypt.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/termux-exec.md5sums (deflated 9%)
  adding: var/lib/dpkg/info/zstd.list (deflated 86%)
  adding: var/lib/dpkg/info/libsmartcols.md5sums (deflated 46%)
  adding: var/lib/dpkg/info/libiconv.md5sums (deflated 67%)
  adding: var/lib/dpkg/info/libevent.list (deflated 89%)
  adding: var/lib/dpkg/info/termux-tools.conffiles (deflated 87%)
  adding: var/lib/dpkg/info/util-linux.md5sums (deflated 71%)
  adding: var/lib/dpkg/info/xz-utils.md5sums (deflated 65%)
  adding: var/lib/dpkg/info/termux-tools.md5sums (deflated 70%)
  adding: var/lib/dpkg/info/libbz2.list (deflated 81%)
  adding: var/lib/dpkg/info/ed.md5sums (deflated 48%)
  adding: var/lib/dpkg/info/xz-utils.list (deflated 91%)
  adding: var/lib/dpkg/info/libnettle.md5sums (deflated 69%)
  adding: var/lib/dpkg/info/dos2unix.list (deflated 87%)
  adding: var/lib/dpkg/info/libnghttp2.list (deflated 82%)
  adding: var/lib/dpkg/info/nano.postinst (deflated 54%)
  adding: var/lib/dpkg/info/libiconv.list (deflated 87%)
  adding: var/lib/dpkg/info/libc++.list (deflated 78%)
  adding: var/lib/dpkg/info/libandroid-glob.list (deflated 79%)
  adding: var/lib/dpkg/info/unzip.md5sums (deflated 62%)
  adding: var/lib/dpkg/info/coreutils.list (deflated 91%)
  adding: var/lib/dpkg/info/psmisc.md5sums (deflated 63%)
  adding: var/lib/dpkg/info/openssl.conffiles (deflated 4%)
  adding: var/lib/dpkg/info/debianutils.md5sums (deflated 59%)
  adding: var/lib/dpkg/info/libcap-ng.list (deflated 87%)
  adding: var/lib/dpkg/info/nano.prerm (deflated 46%)
  adding: var/lib/dpkg/info/apt.list (deflated 89%)
  adding: var/lib/dpkg/info/libbz2.md5sums (deflated 38%)
  adding: var/lib/dpkg/info/libcurl.md5sums (deflated 72%)
  adding: var/lib/dpkg/info/termux-am-socket.md5sums (deflated 35%)
  adding: var/lib/dpkg/info/resolv-conf.conffiles (deflated 38%)
  adding: var/lib/dpkg/info/ca-certificates.md5sums (deflated 7%)
  adding: var/lib/dpkg/info/ed.list (deflated 84%)
  adding: var/lib/dpkg/info/grep.list (deflated 83%)
  adding: var/lib/dpkg/info/bzip2.md5sums (deflated 64%)
  adding: var/lib/dpkg/info/ncurses.md5sums (deflated 71%)
  adding: var/lib/dpkg/available (stored 0%)
[*] Finished successfully (arm).

what is my problem?

sylirre commented 1 year ago

@Agono0 This is because build-bootstraps script is broken. There indeed no bzip2 package because it is a sub package of libbz2. Same for curl and few others. (btw this is what the current issue about)

On other hand generate-bootstraps script works because it operates on already built packages installable by apt, i.e. not their recipes in the termux-packages repository.

See this pull request https://github.com/termux/termux-packages/pull/17031, but anyway built bootstrap still doesn't match generated by generate-bootstraps script.

Agono0 commented 1 year ago

@sylirre Thanks for response. I try generate-bootstrap.sh to build bootstrap but like i said above, it create (useless) bootstrap.zip 150KB which contains only folders with packages name and here is the output(from generate-bootstrap.sh ):

curl: (22) The requested URL returned error: 404
[!] Skipping architecture-independent package list as not available...
[*] Downloading package list for architecture 'arm'...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--
...... some downloads .......
[*] Extracting 'unzip'...
[*] Creating 'bootstrap-arm.zip'...
  adding: etc/ (stored 0%)
  adding: etc/apt/ (stored 0%)
  adding: etc/apt/preferences.d/ (stored 0%)
  adding: etc/apt/apt.conf.d/ (stored 0%)
  adding: tmp/ (stored 0%)
  adding: var/ (stored 0%)
  adding: var/log/ (stored 0%)
  adding: var/log/apt/ (stored 0%)
  adding: var/lib/ (stored 0%)
  adding: var/lib/dpkg/ (stored 0%)
  adding: var/lib/dpkg/status (deflated 77%)
  adding: var/lib/dpkg/updates/ (stored 0%)
  adding: var/lib/dpkg/triggers/ (stored 0%)
  adding: var/lib/dpkg/info/ (stored 0%)
  adding: var/lib/dpkg/info/util-linux.list (deflated 91%)
  adding: var/lib/dpkg/info/findutils.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/bzip2.list (deflated 86%)
  adding: var/lib/dpkg/info/command-not-found.md5sums (deflated 12%)
  adding: var/lib/dpkg/info/termux-licenses.list (deflated 87%)
  adding: var/lib/dpkg/info/readline.conffiles (deflated 5%)
  adding: var/lib/dpkg/info/libgmp.list (deflated 84%)
  adding: var/lib/dpkg/info/dos2unix.md5sums (deflated 66%)
  adding: var/lib/dpkg/info/libunistring.md5sums (deflated 72%)
  adding: var/lib/dpkg/info/inetutils.md5sums (deflated 66%)
  adding: var/lib/dpkg/info/termux-tools.list (deflated 88%)
  adding: var/lib/dpkg/info/libssh2.list (deflated 91%)
  adding: var/lib/dpkg/info/libsmartcols.list (deflated 79%)
  adding: var/lib/dpkg/info/libtirpc.list (deflated 90%)
  adding: var/lib/dpkg/info/net-tools.md5sums (deflated 67%)
  adding: var/lib/dpkg/info/nano.conffiles (deflated 5%)
  adding: var/lib/dpkg/info/libnghttp2.md5sums (deflated 54%)
  adding: var/lib/dpkg/info/zstd.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/lsof.md5sums (deflated 42%)
  adding: var/lib/dpkg/info/libtirpc.md5sums (deflated 70%)
  adding: var/lib/dpkg/info/tar.list (deflated 84%)
  adding: var/lib/dpkg/info/libandroid-support.md5sums (deflated 48%)
  adding: var/lib/dpkg/info/resolv-conf.list (deflated 78%)
  adding: var/lib/dpkg/info/libnettle.list (deflated 90%)
  adding: var/lib/dpkg/info/procps.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/gawk.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/nano.list (deflated 90%)
  adding: var/lib/dpkg/info/ncurses.list (deflated 92%)
  adding: var/lib/dpkg/info/patch.list (deflated 79%)
  adding: var/lib/dpkg/info/libgmp.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/dialog.list (deflated 81%)
  adding: var/lib/dpkg/info/termux-am.md5sums (deflated 30%)
  adding: var/lib/dpkg/info/procps.list (deflated 88%)
  adding: var/lib/dpkg/info/gawk.list (deflated 88%)
  adding: var/lib/dpkg/info/sed.md5sums (deflated 43%)
  adding: var/lib/dpkg/info/net-tools.list (deflated 87%)
  adding: var/lib/dpkg/info/tar.md5sums (deflated 58%)
  adding: var/lib/dpkg/info/sed.list (deflated 82%)
  adding: var/lib/dpkg/info/liblzma.list (deflated 89%)
  adding: var/lib/dpkg/info/termux-am-socket.list (deflated 81%)
  adding: var/lib/dpkg/info/grep.md5sums (deflated 53%)
  adding: var/lib/dpkg/info/termux-exec.list (deflated 79%)
  adding: var/lib/dpkg/info/libgpg-error.list (deflated 85%)
  adding: var/lib/dpkg/info/libcap-ng.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/coreutils.md5sums (deflated 71%)
  adding: var/lib/dpkg/info/liblz4.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/gzip.md5sums (deflated 67%)
  adding: var/lib/dpkg/info/libidn2.list (deflated 88%)
  adding: var/lib/dpkg/info/ca-certificates.list (deflated 78%)
  adding: var/lib/dpkg/info/libassuan.list (deflated 82%)
  adding: var/lib/dpkg/info/libnpth.list (deflated 80%)
  adding: var/lib/dpkg/info/nano.md5sums (deflated 69%)
  adding: var/lib/dpkg/info/bash.list (deflated 89%)
  adding: var/lib/dpkg/info/libidn2.md5sums (deflated 70%)
  adding: var/lib/dpkg/info/libandroid-glob.md5sums (deflated 40%)
  adding: var/lib/dpkg/info/diffutils.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/libc++.md5sums (deflated 8%)
  adding: var/lib/dpkg/info/openssl.md5sums (deflated 71%)
  adding: var/lib/dpkg/info/termux-licenses.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/dpkg.md5sums (deflated 66%)
  adding: var/lib/dpkg/info/libmd.list (deflated 92%)
  adding: var/lib/dpkg/info/bash.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/zlib.list (deflated 82%)
  adding: var/lib/dpkg/info/resolv-conf.md5sums (deflated 32%)
  adding: var/lib/dpkg/info/diffutils.list (deflated 85%)
  adding: var/lib/dpkg/info/liblz4.list (deflated 84%)
  adding: var/lib/dpkg/info/less.md5sums (deflated 58%)
  adding: var/lib/dpkg/info/patch.md5sums (deflated 31%)
  adding: var/lib/dpkg/info/command-not-found.list (deflated 79%)
  adding: var/lib/dpkg/info/openssl.list (deflated 91%)
  adding: var/lib/dpkg/info/gzip.list (deflated 88%)
  adding: var/lib/dpkg/info/libassuan.md5sums (deflated 54%)
  adding: var/lib/dpkg/info/libgpg-error.md5sums (deflated 64%)
  adding: var/lib/dpkg/info/psmisc.list (deflated 85%)
  adding: var/lib/dpkg/info/less.list (deflated 84%)
  adding: var/lib/dpkg/info/lsof.list (deflated 81%)
  adding: var/lib/dpkg/info/libmpfr.list (deflated 85%)
  adding: var/lib/dpkg/info/pcre2.md5sums (deflated 74%)
  adding: var/lib/dpkg/info/libunbound.md5sums (deflated 69%)
  adding: var/lib/dpkg/info/curl.md5sums (deflated 31%)
  adding: var/lib/dpkg/info/libcurl.list (deflated 90%)
  adding: var/lib/dpkg/info/dialog.md5sums (deflated 32%)
  adding: var/lib/dpkg/info/dpkg.list (deflated 87%)
  adding: var/lib/dpkg/info/libgcrypt.list (deflated 84%)
  adding: var/lib/dpkg/info/libmpfr.md5sums (deflated 64%)
  adding: var/lib/dpkg/info/inetutils.list (deflated 87%)
  adding: var/lib/dpkg/info/libunbound.list (deflated 88%)
  adding: var/lib/dpkg/info/bash.conffiles (deflated 42%)
  adding: var/lib/dpkg/info/readline.list (deflated 88%)
  adding: var/lib/dpkg/info/libgnutls.list (deflated 87%)
  adding: var/lib/dpkg/info/libmd.md5sums (deflated 66%)
  adding: var/lib/dpkg/info/dash.md5sums (deflated 39%)
  adding: var/lib/dpkg/info/xxhash.md5sums (deflated 53%)
  adding: var/lib/dpkg/info/unzip.list (deflated 85%)
  adding: var/lib/dpkg/info/termux-am.list (deflated 81%)
  adding: var/lib/dpkg/info/libunistring.list (deflated 91%)
  adding: var/lib/dpkg/info/liblzma.md5sums (deflated 70%)
  adding: var/lib/dpkg/info/libssh2.md5sums (deflated 74%)
  adding: var/lib/dpkg/info/pcre2.list (deflated 91%)
  adding: var/lib/dpkg/info/zlib.md5sums (deflated 52%)
  adding: var/lib/dpkg/info/apt.md5sums (deflated 69%)
  adding: var/lib/dpkg/info/curl.list (deflated 79%)
  adding: var/lib/dpkg/info/findutils.list (deflated 84%)
  adding: var/lib/dpkg/info/xxhash.list (deflated 86%)
  adding: var/lib/dpkg/info/libnpth.md5sums (deflated 46%)
  adding: var/lib/dpkg/info/gpgv.md5sums (deflated 30%)
  adding: var/lib/dpkg/info/libevent.md5sums (deflated 69%)
  adding: var/lib/dpkg/info/debianutils.list (deflated 83%)
  adding: var/lib/dpkg/info/apt.conffiles (deflated 4%)
  adding: var/lib/dpkg/info/libandroid-support.list (deflated 80%)
  adding: var/lib/dpkg/info/termux-keyring.md5sums (deflated 63%)
  adding: var/lib/dpkg/info/termux-keyring.list (deflated 88%)
  adding: var/lib/dpkg/info/gpgv.list (deflated 79%)
  adding: var/lib/dpkg/info/readline.md5sums (deflated 67%)
  adding: var/lib/dpkg/info/libgnutls.md5sums (deflated 68%)
  adding: var/lib/dpkg/info/dash.list (deflated 83%)
  adding: var/lib/dpkg/info/libgcrypt.md5sums (deflated 61%)
  adding: var/lib/dpkg/info/termux-exec.md5sums (deflated 9%)
  adding: var/lib/dpkg/info/zstd.list (deflated 86%)
  adding: var/lib/dpkg/info/libsmartcols.md5sums (deflated 46%)
  adding: var/lib/dpkg/info/libiconv.md5sums (deflated 67%)
  adding: var/lib/dpkg/info/libevent.list (deflated 89%)
  adding: var/lib/dpkg/info/termux-tools.conffiles (deflated 87%)
  adding: var/lib/dpkg/info/util-linux.md5sums (deflated 71%)
  adding: var/lib/dpkg/info/xz-utils.md5sums (deflated 65%)
  adding: var/lib/dpkg/info/termux-tools.md5sums (deflated 70%)
  adding: var/lib/dpkg/info/libbz2.list (deflated 81%)
  adding: var/lib/dpkg/info/ed.md5sums (deflated 48%)
  adding: var/lib/dpkg/info/xz-utils.list (deflated 91%)
  adding: var/lib/dpkg/info/libnettle.md5sums (deflated 69%)
  adding: var/lib/dpkg/info/dos2unix.list (deflated 87%)
  adding: var/lib/dpkg/info/libnghttp2.list (deflated 82%)
  adding: var/lib/dpkg/info/nano.postinst (deflated 54%)
  adding: var/lib/dpkg/info/libiconv.list (deflated 87%)
  adding: var/lib/dpkg/info/libc++.list (deflated 78%)
  adding: var/lib/dpkg/info/libandroid-glob.list (deflated 79%)
  adding: var/lib/dpkg/info/unzip.md5sums (deflated 62%)
  adding: var/lib/dpkg/info/coreutils.list (deflated 91%)
  adding: var/lib/dpkg/info/psmisc.md5sums (deflated 63%)
  adding: var/lib/dpkg/info/openssl.conffiles (deflated 4%)
  adding: var/lib/dpkg/info/debianutils.md5sums (deflated 59%)
  adding: var/lib/dpkg/info/libcap-ng.list (deflated 87%)
  adding: var/lib/dpkg/info/nano.prerm (deflated 46%)
  adding: var/lib/dpkg/info/apt.list (deflated 89%)
  adding: var/lib/dpkg/info/libbz2.md5sums (deflated 38%)
  adding: var/lib/dpkg/info/libcurl.md5sums (deflated 72%)
  adding: var/lib/dpkg/info/termux-am-socket.md5sums (deflated 35%)
  adding: var/lib/dpkg/info/resolv-conf.conffiles (deflated 38%)
  adding: var/lib/dpkg/info/ca-certificates.md5sums (deflated 7%)
  adding: var/lib/dpkg/info/ed.list (deflated 84%)
  adding: var/lib/dpkg/info/grep.list (deflated 83%)
  adding: var/lib/dpkg/info/bzip2.md5sums (deflated 64%)
  adding: var/lib/dpkg/info/ncurses.md5sums (deflated 71%)
  adding: var/lib/dpkg/available (stored 0%)
[*] Finished successfully (arm).

Am i doing something wrong? Or i miss something to do with script?

Agono0 commented 1 year ago

@sylirre Can i just use your modified script you provide it on the pull request? Or still not working?

sylirre commented 1 year ago

generate-bootstraps is only for official packaging which means it basically useless for Termux derivatives. The Termux application package name should be com.termux, if you want to use it.

Or still not working?

It works, but the generated bootstrap archive is much bigger than it should be. Lots of "junk" included which mostly come from subpackage dependencies.

Agono0 commented 1 year ago

So , generate-bootstrap.sh will not working if i change the prefix (the package name) idk , i just guess. Anyway, thanks so much for help, I really appreciate that

Agono0 commented 1 year ago

still have problem with generate-bootstrap.sh i cannot make a vaild bootstrap.zip with SYMLINKS.txt and there is no lib folder i just use sudo ./scripts/run-docker.sh ./scripts/generate-bootstraps.sh what i am missing?? here is produced file : bootstrap-arm.zip

sylirre commented 1 year ago

@Agono0 generate-bootstrap.sh is suitable only for official Termux. If you change prefix it would not work because all our deb files are for com.termux (this also hardcoded in every binary file inside .deb).

Here is the log file for successfully built bootstraps: bootstraps_build_log.txt

Again, if you want to use generate-bootstrap.sh then one of these conditions must be met:

There no various hidden workarounds. If deb file is built with prefix /data/data/com.termux/files/usr, then it can be used only for this one. This path is hardcoded in deb file itself, each ELF binary file through the DT_RUNPATH section and hardcoded strings containing paths to various files and directories.

Agono0 commented 1 year ago

@sylirre And yeah this what i expected , now i need to build bootstrap for custom PREFIX there is no solution for that? thanks again, you saved me from alot of tries.

Agono0 commented 1 year ago

seems like i will open new issue for that, as i change prefix in all termux script, and after ignore subpackages the bootstrap.zip generated successfully, but still have problem after i build my apk , it show same problem like here https://github.com/termux/termux-app/issues/1383 after i open my app, it show : exec("/data/data/myprefix/files/usr/bin/login"): No such file or directory i check SYMLINKS.txt it contains myprefix normal. but when i go to bin/login file which in bootstrap.zip i found this:

#!/data/data/com.myprefix/files/usr/bin/sh

if tty >/dev/null 2>&1 && [ $# = 0 ] && [ ! -f ~/.hushlogin ] && [ -z "$TERMUX_HUSHLOGIN" ]; then
    # Use user defined dynamic motd file if it exists
    if [ -f ~/.termux/motd.sh ]; then
        [ ! -x ~/.termux/motd.sh ] && chmod u+x ~/.termux/motd.sh
        ~/.termux/motd.sh
    # Default to termux-tools package provided static motd file if it exists
    elif [ -f /data/data/com.termux/files/usr/etc/motd ]; then
        cat /data/data/com.termux/files/usr/etc/motd
    fi
else
    # This variable shouldn't be kept set.
    unset TERMUX_HUSHLOGIN
fi

# TERMUX_VERSION env variable has been exported since v0.107 and PATH was being set to following value in <0.104. Last playstore version was v0.101.
if tty >/dev/null 2>&1 && [ $# = 0 ] && [ -f /data/data/com.termux/files/usr/etc/motd-playstore ] && [ -z "$TERMUX_VERSION" ] && [ "$PATH" = "/data/data/com.termux/files/usr/bin:/data/data/com.termux/files/usr/bin/applets" ]; then
    printf '\033[0;31m'; cat /data/data/com.termux/files/usr/etc/motd-playstore; printf '\033[0m'
fi

if [ -G ~/.termux/shell ]; then
    export SHELL="`realpath ~/.termux/shell`"
else
    for file in /data/data/com.termux/files/usr/bin/bash /data/data/com.termux/files/usr/bin/sh /system/bin/sh; do
        if [ -x $file ]; then
            export SHELL=$file
            break
        fi
    done
fi

# TERMUX_APP_PACKAGE_MANAGER should be exported by termux-app v0.119.0+ itself
if [ -z "${TERMUX_APP_PACKAGE_MANAGER-}" ]; then
    if { [ -n "$(command -v dpkg)" ] && dpkg --compare-versions "$TERMUX_VERSION" lt 0.119.0; } ||   # apt
        { [ -n "$(command -v vercmp)" ] && [ "$(vercmp "$TERMUX_VERSION" 0.119.0)" = "-1" ]; }; then # pacman
        # For the correct operation of scripts that work with the package manager
        export TERMUX_MAIN_PACKAGE_FORMAT="debian"
    fi
fi

if [ -f /data/data/com.termux/files/usr/lib/libtermux-exec.so ]; then
    export LD_PRELOAD=/data/data/com.termux/files/usr/lib/libtermux-exec.so
    $SHELL -c "coreutils --coreutils-prog=true" > /dev/null 2>&1 || unset LD_PRELOAD
fi

if [ -f /data/data/com.termux/files/usr/etc/termux-login.sh ]; then
    . /data/data/com.termux/files/usr/etc/termux-login.sh
fi

if [ -n "$TERM" ]; then
    exec "$SHELL" -l "$@"
else
    exec "$SHELL" "$@"
fi

like you see above, first line of the file #!/data/data/com.myprefix/files/usr/bin/sh only contain my prefix but rest of the file still contains com.termux prefix. i try to change all perfix of bootstrap files (about 97 files found with com.termux perfix) and try to zip the folder and try again the apk, still show the same problem I check those things to make sure i didnot do anything wrong:

and i cannot until now to solve the problem. any suggestion?

sylirre commented 1 year ago

@Agono0 Please don't try to change prefix inside the original precompiled Termux packages. This is not supported! Please look my previous message.

Yes, scripts can be easily edited, but binaries are not. Prefix is hardcoded everywhere. Each binary contains at least one reference to the Termux prefix directory. You need to compile everything from scratch.

Screenshot_20230704-022644

Screenshot_20230704-022620

Reminding that generate-bootstraps.sh downloads official Termux binaries from our APT repository. In order to use own packages you need to setup a local package server (e.g. using Sonatype Nexus) and change URL inside script.

Agono0 commented 1 year ago

Thanks so much for info, I need last thing, if you can give me the road to build the same (required) package for termux (with custom prefix) I will happy with that.

Agono0 commented 1 year ago

@sylirre i think i am right bro (i think its a bug), as i check all libs in bin folder in bootstrap.zip using your showen command readelf -d bash i got my prefix as shown here :

Dynamic section at offset 0xc2098 contains 32 entries:
  Tag        Type                         Name/Value
 0x000000000000001d (RUNPATH)            Library runpath: [/data/data/com.agono.myprefix/files/usr/lib]
 0x0000000000000001 (NEEDED)             Shared library: [libandroid-support.so]
 0x0000000000000001 (NEEDED)             Shared library: [libreadline.so.8]
 0x0000000000000001 (NEEDED)             Shared library: [libiconv.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000007 (RELA)               0x17888
 0x0000000000000008 (RELASZ)             34896 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffff9 (RELACOUNT)          1366
 0x0000000000000017 (JMPREL)             0x200d8
 0x0000000000000002 (PLTRELSZ)           6624 (bytes)
 0x0000000000000003 (PLTGOT)             0xc4f80
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000006 (SYMTAB)             0x358
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0xfc50
 0x000000000000000a (STRSZ)              31797 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0xcc98
 0x0000000000000020 (PREINIT_ARRAY)      0xc1bb0
 0x0000000000000021 (PREINIT_ARRAYSZ)    16 (bytes)
 0x0000000000000019 (INIT_ARRAY)         0xc1bc0
 0x000000000000001b (INIT_ARRAYSZ)       16 (bytes)
 0x000000000000001a (FINI_ARRAY)         0xc1bd0
 0x000000000000001c (FINI_ARRAYSZ)       16 (bytes)
 0x000000006ffffff0 (VERSYM)             0xbce0
 0x000000006ffffffe (VERNEED)            0xcc58
 0x000000006fffffff (VERNEEDNUM)         2
 0x0000000000000000 (NULL)               0x0

this means that build-bootstraps.sh works fine. i check output folder , i try to decompile bash_5.2.15-1_arm.deb file and i see that folder of data give the same prefix of mine. and i am still getting this : exec("/data/data/com.agono.myprefix/files/usr/bin/login"): No such file or directory.(from my apk) again, i check bin/login file and i see it's just a script(idk) like i send it above. it's a bug or not, My conclusion is that the prefix has no problems Edit: i have check all shared library in lib using same command readelf -a libandroid-support.so and i find Library runpath same as mine.