chGoodchild / TollGateNostrToolKit

0 stars 0 forks source link

sysupgrade with minimal requirements from install script #18

Open chGoodchild opened 2 months ago

chGoodchild commented 2 months ago
username@ubuntu-4gb-nbg1-2:~/openwrt$ find . -regex ".*openwrt-.*-sysupgrade.bin"
./bin/targets/ath79/generic/openwrt-ath79-generic-glinet_gl-ar300m16-squashfs-sysupgrade.bin
./build_dir/target-mips_24kc_musl/linux-ath79_generic/tmp/openwrt-ath79-generic-glinet_gl-ar300m16-squashfs-sysupgrade.bin
The two files you've found are both sysupgrade firmware images for the GL.iNet GL-AR300M16 router, but they are located in different directories and serve different purposes:

1. `./bin/targets/ath79/generic/openwrt-ath79-generic-glinet_gl-ar300m16-squashfs-sysupgrade.bin`
   - This is the final, ready-to-use firmware image.
   - It's located in the `bin` directory, which typically contains the output files of the build process.
   - **Use this file** when you want to upgrade your router's firmware.

2. `./build_dir/target-mips_24kc_musl/linux-ath79_generic/tmp/openwrt-ath79-generic-glinet_gl-ar300m16-squashfs-sysupgrade.bin`
   - This is a temporary or intermediate file created during the build process.
   - It's located in the `build_dir`, which contains files used during compilation.
   - You should **not use this file** for upgrading your router.

Key differences:

1. **Location**: The first file is in the output directory, while the second is in a temporary build directory.
2. **Purpose**: The first file is the final product, while the second is an intermediate file.
3. **Reliability**: The file in the `bin` directory has gone through all necessary build steps and checks, making it more reliable for use.

In summary, always use the firmware image from the `bin/targets/...` directory when upgrading your router. The file in the `build_dir` is not intended for end-user use and may be incomplete or unstable.
chGoodchild commented 2 months ago

Yes, you can expect the sysupgrade.bin file generated from your make process to include all the packages from your custom feeds, assuming you've configured everything correctly. Here's a breakdown:

  1. make output:

    • When you compile OpenWrt using make, it builds the entire system, including the kernel, root filesystem, and all specified packages.
    • If you've properly added and configured your custom feeds, the resulting sysupgrade.bin file should include all the programs from these custom feeds.
    • This image is a complete system image with all your specified packages pre-installed.
  2. OpenWrt Image Builder output:

    • The Image Builder is designed to create custom images quickly without compiling the entire OpenWrt system.
    • It uses pre-compiled packages to create images, which is why it doesn't include your custom feed packages by default.
    • You would need to manually specify any additional packages when using the Image Builder.

Key points:

  1. The make process gives you more control and includes everything in your build configuration, including custom feeds.
  2. The Image Builder is faster but less flexible, especially for custom packages.

To ensure your custom feeds are included in the make output:

  1. Make sure your custom feeds are properly defined in feeds.conf or feeds.conf.default.
  2. Run ./scripts/feeds update -a and ./scripts/feeds install -a.
  3. In make menuconfig, select your custom packages for inclusion.
  4. Run make to build the image.

The resulting sysupgrade.bin from this process should contain all your custom feed packages, ready to use after flashing.

If you need the flexibility of the Image Builder but want to include your custom packages, you might consider:

  1. Building your custom packages separately.
  2. Adding these packages to a local package repository.
  3. Configuring the Image Builder to use this local repository.

This approach combines the speed of the Image Builder with the inclusion of your custom packages.

chGoodchild commented 2 months ago

Currently at:

GLTollGate, commit be5a1a365b2bf1a766a8c5bebc32a46fa0e7d8cc
Branch: install_script_plain_openwrt
chGoodchild commented 2 months ago

To include your custom script in the sysupgrade.bin file for OpenWrt, you'll need to follow these steps:

  1. Create your custom script
  2. Add it to the OpenWrt build system
  3. Configure the build system to include your script
  4. Build the custom firmware image

Here's a detailed walkthrough:

  1. Create your custom script:

Create a file named 99-first-login (or any other name you prefer) with the following content:

#!/bin/sh /etc/rc.common

START=99
STOP=15

start() {
    if [ ! -f /etc/first_login_done ]; then
        # Your custom commands here

        # Mark as done
        touch /etc/first_login_done
    fi
}

stop() {
    # Optional: Add any cleanup steps here
    :
}
  1. Add it to the OpenWrt build system:

Place your script in the OpenWrt source tree. A good location would be:

package/base-files/files/etc/init.d/99-first-login
  1. Configure the build system to include your script:

Edit the package/base-files/files/lib/preinit/80_mount_root file and add the following line near the end (before the last }:

[ -f /etc/init.d/99-first-login ] && /etc/init.d/99-first-login enable

This ensures that your script is enabled during the initial boot process.

  1. Build the custom firmware image:

Now, build your custom OpenWrt image as usual:

make menuconfig
# Select your target device and desired packages
make -j$(nproc)

The resulting sysupgrade.bin file will include your custom script.

Additional notes:

By following these steps, your sysupgrade.bin file will contain the custom script that runs on first login using /etc/rc.common. When you flash this firmware to your router, the script will be in place and ready to execute on the first boot.

chGoodchild commented 2 months ago

Missing some packages in sysupgrade.bin

root@OpenWrt:/tmp# ./package_checker.sh "base-files busybox ca-bundle dnsmasq dropbear firewall4 fstools kmod-ath9k kmod-gpio-button-hotplug kmod-nft-offload kmod-usb2 libc lib
gcc libustream-mbedtls logd luci mtd netifd nftables odhcp6c odhcpd-ipv6only opkg ppp ppp-mod-pppoe procd procd-seccomp procd-ujail swconfig uboot-envtools uci uclient-fetch ur
andom-seed urngd wpad-basic-mbedtls"
Installed packages:

base-files
busybox
ca-bundle
dnsmasq
dropbear
firewall4
fstools
kmod-ath9k
kmod-gpio-button-hotplug
kmod-nft-offload
libc
logd
mtd
netifd
odhcp6c
odhcpd-ipv6only
opkg
ppp
ppp-mod-pppoe
procd
procd-seccomp
procd-ujail
swconfig
uboot-envtools
uci
uclient-fetch
urandom-seed
urngd
wpad-basic-mbedtls

Not installed packages:

kmod-usb2
libgcc
libustream-mbedtls
luci
nftables

root@OpenWrt:/tmp# ./package_checker.sh "jq curl unzip coreutils-base64 libxtables12 kmod-nf-ipt kmod-ipt-core kmod-ipt-ipopt iptables-mod-ipopt libmicrohttpd-no-ssl kmod-nft-c
ompat libiptext0 libiptext6-0 libiptext-nft0 xtables-nft iptables-nft kmod-ipt-nat kmod-ipt-nat-extra iptables-mod-nat-extra kmod-nf-conncount kmod-ipt-conntrack kmod-ipt-connt
rack-extra iptables-mod-conntrack-extra"
Installed packages:

libxtables12
kmod-nf-ipt
kmod-ipt-core
kmod-ipt-ipopt
iptables-mod-ipopt
libmicrohttpd-no-ssl
kmod-nft-compat
libiptext0
libiptext6-0
libiptext-nft0
xtables-nft
iptables-nft
kmod-ipt-nat
kmod-ipt-nat-extra
iptables-mod-nat-extra
kmod-nf-conncount
kmod-ipt-conntrack
kmod-ipt-conntrack-extra
iptables-mod-conntrack-extra

Not installed packages:

jq
curl
unzip
coreutils-base64

Add:

chGoodchild commented 2 months ago

To find and add those libraries in the OpenWrt configuration menu, you'll need to navigate through different sections. Here's a guide on where to find each:

  1. kmod-usb2: Navigate to: Kernel modules -> USB Support Look for "kmod-usb2" and select it with

  2. libgcc: This is usually included by default, but you can check under: Libraries -> gcc libraries -> libgcc

  3. libustream-mbedtls: Navigate to: Libraries -> SSL Look for "libustream-mbedtls" and select it

  4. luci: Navigate to: LuCI -> Collections Select "luci" to include the web interface

  5. nftables: Navigate to: Network -> Firewall Look for "nftables" and select it

  6. jq: Navigate to: Utilities Look for "jq" and select it

  7. curl: Navigate to: Network -> File Transfer Look for "curl" and select it

  8. unzip: Navigate to: Utilities Look for "unzip" and select it

  9. coreutils-base64: Navigate to: Utilities -> coreutils Look for "coreutils-base64" and select it

To navigate the menu:

If you can't find a package, it might be necessary to update your feed sources. You can do this by exiting the menu, and running:

./scripts/feeds update -a
./scripts/feeds install -a

Then run make menuconfig again to re-enter the configuration menu. This should populate the menu with more packages.

Remember to save your configuration before exiting the menu.

chGoodchild commented 2 months ago

Based on your requirements, you're correct that you shouldn't rely on the Makefile of GLTollGate to introduce these changes if you want them to be included in the sysupgrade.bin file. Instead, you should make these changes before running make. Here's a suggested approach:

  1. Create a custom package in your OpenWrt feed that includes these customizations.
  2. Modify your build script to apply these changes before building the firmware.

Here's how you can modify your build script (build_custom_dependencies.sh) to achieve this:

  1. First, create a custom package in your OpenWrt feed (e.g., in ~/secp256k1_openwrt_feed/custom/my-customizations).

  2. In this package, include your custom files:

    • files/etc/init.d/99-first-login
    • files/lib/preinit/80_mount_root
    • files/etc/uci-defaults/99-custom-scripts (for your custom UCI scripts)
  3. Create a Makefile in this package directory that installs these files to the correct locations.

  4. Modify your build_custom_dependencies.sh script to include these steps before building:

#!/bin/bash

# ... (previous content)

# Install the dependencies from the custom feed
echo "Installing dependencies from custom feed..."
./scripts/feeds install -a

# Install your custom package
./scripts/feeds install my-customizations

# Copy configuration files
# ... (rest of the script)

# Before running make, ensure your custom package is selected
sed -i 's/# CONFIG_PACKAGE_my-customizations is not set/CONFIG_PACKAGE_my-customizations=y/' .config

# Now run make
echo "Build with dependencies before using them..."
make -j$(nproc) V=sc > make_logs.md 2>&1
if [ $? -ne 0 ]; then
   echo "Firmware build failed."
   exit 1
fi

# ... (rest of the script)

This approach has several advantages:

  1. Your customizations are properly packaged and can be version-controlled.
  2. The changes are applied before the build process starts, ensuring they're included in the sysupgrade.bin.
  3. It's more maintainable and easier to update in the future.
  4. You can easily enable or disable these customizations by selecting or deselecting the package in the OpenWrt configuration.

Remember to add your new package to your custom feed's Makefile or packages.mk file so that OpenWrt knows about it.

This method ensures that your customizations are included in the build process from the beginning and will be present in the final sysupgrade.bin file.

chGoodchild commented 2 months ago
username@ubuntu-32gb-fsn1-1:~/TollGateNostrToolKit$ grep -n -E "Error" -C 5 ~/openwrt/make_logs.md
35360-/bin/bash ../libtool  --tag=CC   --mode=compile mips-openwrt-linux-musl-gcc -DHAVE_CONFIG_H -I. -I..  -D__GMP_WITHIN_GMP -I.. -DOPERATION_`echo ior_n | sed 's/_$//'` -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/usr/include -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/include/fortify -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/include   -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -msoft-float -fmacro-prefix-map=/home/username/openwrt/build_dir/target-mips_24kc_musl/gmp-6.2.1=gmp-6.2.1 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -DPIC -fpic  -c -o ior_n.lo ior_n.c
35361-rm -rf /home/username/openwrt/build_dir/target-mips_24kc_musl/urandom-seed/.pkgdir/urandom-seed.installed /home/username/openwrt/build_dir/target-mips_24kc_musl/urandom-seed/.pkgdir/urandom-seed
35362-mkdir -p /home/username/openwrt/build_dir/target-mips_24kc_musl/urandom-seed/.pkgdir/urandom-seed
35363-cp -fpR ./files/* /home/username/openwrt/build_dir/target-mips_24kc_musl/urandom-seed/.pkgdir/urandom-seed/
35364-touch /home/username/openwrt/build_dir/target-mips_24kc_musl/urandom-seed/.pkgdir/urandom-seed.installed
35365:checking for GPG Error - version >= 1.27... mkdir -p /home/username/openwrt/staging_dir/target-mips_24kc_musl/root-ath79/stamp
35366-SHELL= flock /home/username/openwrt/tmp/.root-copy.flock -c 'cp -fpR /home/username/openwrt/build_dir/target-mips_24kc_musl/urandom-seed/.pkgdir/urandom-seed/. /home/username/openwrt/staging_dir/target-mips_24kc_musl/root-ath79/'
35367-touch /home/username/openwrt/staging_dir/target-mips_24kc_musl/root-ath79/stamp/.urandom-seed_installed
35368-OpenWrt-libtool: compile:  mips-openwrt-linux-musl-gcc -DHAVE_CONFIG_H -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_nand_n -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/usr/include -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/include/fortify -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/include -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -msoft-float -fmacro-prefix-map=/home/username/openwrt/build_dir/target-mips_24kc_musl/gmp-6.2.1=gmp-6.2.1 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -DPIC -fpic -c nand_n.c -o nand_n.o >/dev/null 2>&1
35369-mkdir -p /home/username/openwrt/bin/targets/ath79/nand/packages /home/username/openwrt/build_dir/target-mips_24kc_musl/urandom-seed/ipkg-mips_24kc/urandom-seed/CONTROL /home/username/openwrt/staging_dir/target-mips_24kc_musl/pkginfo
35370-cp -fpR ./files/* /home/username/openwrt/build_dir/target-mips_24kc_musl/urandom-seed/ipkg-mips_24kc/urandom-seed/
--
44718-conf.c: In function 'main':
44719-conf.c:617:41: warning: format not a string literal and no format arguments [-Wformat-security]
44720-  617 |                                         _("\n*** The configuration requires explicit update.\n\n"));
44721-      |                                         ^
44722-conf.c:669:25: warning: format not a string literal and no format arguments [-Wformat-security]
44723:  669 |                         fprintf(stderr, _("\n*** Error during writing of the configuration.\n\n"));
44724-      |                         ^~~~~~~
44725-conf.c:673:25: warning: format not a string literal and no format arguments [-Wformat-security]
44726:  673 |                         fprintf(stderr, _("\n*** Error during update of the configuration.\n\n"));
44727-      |                         ^~~~~~~
44728-conf.c:684:25: warning: format not a string literal and no format arguments [-Wformat-security]
44729:  684 |                         fprintf(stderr, _("\n*** Error during writing of the configuration.\n\n"));
44730-      |                         ^~~~~~~
44731-mips-openwrt-linux-gnu
44732-checking for grep that handles long lines and -e... (cached) /home/username/openwrt/staging_dir/host/bin/grep
44733-checking for egrep... /home/username/openwrt/staging_dir/host/bin/grep -E
44734-checking if OS is AIX (to define _ALL_SOURCE)... /usr/bin/env bash ../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -DNO_XMALLOC -I. -I..  -I. -I. -I.. -I.. -DIN_LIBUNISTRING -DDEPENDS_ON_LIBICONV=1 -I/home/username/openwrt/staging_dir/host/include -I/home/username/openwrt/staging_dir/hostpkg/include -I/home/username/openwrt/staging_dir/target-mips_24kc_musl/host/include  -Wno-cast-qual -Wno-conversion -Wno-float-equal -Wno-sign-compare -Wno-undef -Wno-unused-function -Wno-unused-parameter -Wno-float-conversion -Wimplicit-fallthrough -Wno-pedantic -Wno-sign-conversion -Wno-type-limits -Wno-unsuffixed-float-constants -O2 -I/home/username/openwrt/staging_dir/host/include -I/home/username/openwrt/staging_dir/hostpkg/include -I/home/username/openwrt/staging_dir/target-mips_24kc_musl/host/include -DPIC -fPIC -c -o unictype/libunistring_la-pr_ids_binary_operator.lo `test -f 'unictype/pr_ids_binary_operator.c' || echo './'`unictype/pr_ids_binary_operator.c
--
53907-mips-openwrt-linux-musl-gcc  -I. -Iinclude -Iproviders/common/include -Iproviders/implementations/include  -DAES_ASM -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -fPIC -pthread -mabi=32 -Wa,--noexecstack -Wall -O3 -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -msoft-float -fmacro-prefix-map=/home/username/openwrt/build_dir/target-mips_24kc_musl/openssl-3.0.13=openssl-3.0.13 -ffunction-sections -fdata-sections -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -DPIC -fpic -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -msoft-float -fmacro-prefix-map=/home/username/openwrt/build_dir/target-mips_24kc_musl/openssl-3.0.13=openssl-3.0.13 -ffunction-sections -fdata-sections -Wformat -Werror=format-security -fstack-protector -fpic -znow -zrelro -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSLDIR="\"/etc/ssl\"" -DENGINESDIR="\"/usr/lib/engines-3\"" -DMODULESDIR="\"/usr/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -D_FORTIFY_SOURCE=1 -DPIC -DOPENSSL_PREFER_CHACHA_OVER_GCM -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/usr/include -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/include/fortify -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/include -MMD -MF crypto/err/libcrypto-shlib-err.d.tmp -MT crypto/err/libcrypto-shlib-err.o -c -o crypto/err/libcrypto-shlib-err.o crypto/err/err.c
53908-../../../scripts/cd2nroff CURLOPT_FTP_USE_EPSV.md >CURLOPT_FTP_USE_EPSV.3
53909-cc1: note: someone does not honour COPTS correctly, passed 2 times
53910-mips-openwrt-linux-musl-gcc  -I. -Iinclude -Iproviders/common/include -Iproviders/implementations/include  -DAES_ASM -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -fPIC -pthread -mabi=32 -Wa,--noexecstack -Wall -O3 -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -msoft-float -fmacro-prefix-map=/home/username/openwrt/build_dir/target-mips_24kc_musl/openssl-3.0.13=openssl-3.0.13 -ffunction-sections -fdata-sections -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -DPIC -fpic -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -msoft-float -fmacro-prefix-map=/home/username/openwrt/build_dir/target-mips_24kc_musl/openssl-3.0.13=openssl-3.0.13 -ffunction-sections -fdata-sections -Wformat -Werror=format-security -fstack-protector -fpic -znow -zrelro -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSLDIR="\"/etc/ssl\"" -DENGINESDIR="\"/usr/lib/engines-3\"" -DMODULESDIR="\"/usr/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -D_FORTIFY_SOURCE=1 -DPIC -DOPENSSL_PREFER_CHACHA_OVER_GCM -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/usr/include -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/include/fortify -I/home/username/openwrt/staging_dir/toolchain-mips_24kc_gcc-12.3.0_musl/include -MMD -MF crypto/err/libcrypto-shlib-err_all.d.tmp -MT crypto/err/libcrypto-shlib-err_all.o -c -o crypto/err/libcrypto-shlib-err_all.o crypto/err/err_all.c
53911-mkdir .lib 2>/dev/null
53912:make[4]: [Makefile:322: libhogweed.so] Error 1 (ignored)
53913-(cd .lib \
53914-          && rm -f libhogweed.so \
53915-  && ln -s ../libhogweed.so libhogweed.so \
53916-  && [ -z "libhogweed.so.6" ] \
53917-  || { rm -f libhogweed.so.6 \
--
87961-Configuring uhttpd-mod-ubus.
87962-Configuring luci-light.
87963-Configuring luci.
87964-Configuring urngd.
87965-Configuring ppp-mod-pppoe.
87966:make[2]: *** [package/Makefile:71: package/install] Error 255
87967-make[2]: Leaving directory '/home/username/openwrt'
87968:make[1]: *** [package/Makefile:124: /home/username/openwrt/staging_dir/target-mips_24kc_musl/stamp/.package_install] Error 2
87969-make[1]: Leaving directory '/home/username/openwrt'
87970:make: *** [/home/username/openwrt/include/toplevel.mk:232: world] Error 2
username@ubuntu-32gb-fsn1-1:~/openwrt$ ls package/base-files/files/etc/
banner       device_info  fstab  hotplug.d  iproute2     os-release  profile    rc.common  shadow  sysctl.conf      uci-defaults
banner.failsafe  diag.sh      group  init.d openwrt_release  passwd      protocols  rc.local   shells  sysctl.d
board.d      ethers       hosts  inittab    openwrt_version  preinit     rc.button  services   shinit  sysupgrade.conf
username@ubuntu-32gb-fsn1-1:~/openwrt$ ls package/base-files/files/
bin/  etc/  lib/  rom/  sbin/ usr/  
username@ubuntu-32gb-fsn1-1:~/openwrt$ ls package/base-files/files/bin/
board_detect  config_generate  ipcalc.sh
username@ubuntu-32gb-fsn1-1:~/openwrt$ ls package/base-files/files/lib/
functions  functions.sh  preinit  upgrade