search

sudomesh / sudowrt-firmware

Scripts to build the sudo mesh OpenWRT firmware.
Other
73 stars 19 forks source link

make download in build_pre does not retreive all packages needed for offline build_only #116

Closed jhpoelen closed 6 years ago

jhpoelen commented 6 years ago

In an effort to stabilize firmware build process, two phases have been introduced: build_pre and build_only. The first gathers all (external) resources needed to run the build, and the second assembles the firmware using only build_pre.

Most packages, tools etc, are gathered using the make download target (see build_lib, however, when running

sudo docker run --net=none -v $PWD/firmware_images:/firmware_images sudowrt/firmware

the ar71xx build failed on 

make[3]: Entering directory `/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/package/libs/lzo'
mkdir -p /usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/dl
/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/scripts/download.pl "/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/dl" "lzo-2.08.tar.gz" "fcec64c26a0f4f4901468f360029678f" "http://www.oberhumer.com/opensource/lzo/download/"
--2017-12-05 18:28:00--  http://www.oberhumer.com/opensource/lzo/download/lzo-2.08.tar.gz
Resolving www.oberhumer.com (www.oberhumer.com)... failed: Name or service not known.
wget: unable to resolve host address 'www.oberhumer.com'
Download failed.
--2017-12-05 18:28:00--  http://mirror2.openwrt.org/sources/lzo-2.08.tar.gz
Resolving mirror2.openwrt.org (mirror2.openwrt.org)... failed: Name or service not known.
wget: unable to resolve host address 'mirror2.openwrt.org'
Download failed.
--2017-12-05 18:28:00--  http://downloads.openwrt.org/sources/lzo-2.08.tar.gz
Resolving downloads.openwrt.org (downloads.openwrt.org)... failed: Name or service not known.
wget: unable to resolve host address 'downloads.openwrt.org'
Download failed.
No more mirrors to try - giving up.
make[3]: *** [/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/dl/lzo-2.08.tar.gz] Error 2
make[3]: Leaving directory `/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/package/libs/lzo'
make[2]: *** [package/libs/lzo/compile] Error 2
make[2]: Leaving directory `/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx'
make[1]: *** [/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/staging_dir/target-mips_34kc_uClibc-0.9.33.2/stamp/.package_compile] Error 2
make[1]: Leaving directory `/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx'
make: *** [world] Error 2
make: Leaving directory `/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx'
jhpoelen commented 6 years ago

@wigyori - I tried to figure out why libs packages like package/libs/openssl are included in the make download target whereas package/libs/lzo is not. Any suggestions on where to look?

paidforby commented 6 years ago

After poking around inside a container a little I think I found the problem. My guess is we just need to explicitly request lzo as an openwrt package, you can see that openssl is inlcuded here as libopenssl. Testing this theory now, will report back.

paidforby commented 6 years ago

So just adding lzo to the packages does not appear to solve the problem, despite it appearing to be installed in the container in `packages/libs/lzo. Build still fails in same way after 53mins. Potential, non-ideal solution is to manually perform the command while you have internet,

/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/scripts/download.pl "/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/dl" "lzo-2.08.tar.gz" "fcec64c26a0f4f4901468f360029678f" "http://www.oberhumer.com/opensource/lzo/download/"

I'm going to try inserting this into the pre-build and then running the build with net-none. will report back in another 53 minutes.

paidforby commented 6 years ago

I'm noticing that during make download it's doing the following,

 make[2] -C feeds/wlanslovenija/net/tunneldigger download
 make[2] -C package/kernel/gpio-button-hotplug download
 make[2] -C package/kernel/linux download
 make[2] -C package/kernel/mac80211 download
 make[2] -C package/libs/libjson-c download
 make[2] -C package/libs/libnl download
 make[2] -C package/libs/libnl-tiny download
 make[2] -C package/libs/libpcap download
 make[2] -C package/libs/libubox download
 make[2] -C package/libs/openssl download
 make[2] -C package/libs/polarssl download
 make[2] -C package/libs/toolchain download
 make[2] -C package/libs/uclibc++ download
 make[2] -C package/libs/ustream-ssl download
 make[2] -C package/libs/zlib download
 make[2] -C package/network/config/firewall download

I'm wondering where it gets this list of packages from?

paidforby commented 6 years ago

Ok, so further investigation lead me to find where make download is defined, built_firmware/builder.ar71xx/include/download.mk. Good news is that is appears to perform the exact same command that I suggested to run manually. However, it's still unclear how it knows what deps to download, it looks like it just loops through the entire directory tree looking for possible packages it needs to download, so it's weird that it isn't getting lzo, since I'm almost sure that has a makefile in package/libs/lzo. Below is the partial definition of make download.

define Download
  $(eval $(Download/Defaults))
  $(eval $(Download/$(1)))
  $(foreach FIELD,URL FILE $(Validate/$(call dl_method,$(URL),$(PROTO))),
    ifeq ($($(FIELD)),)
      $$(error Download/$(1) is missing the $(FIELD) field.)
    endif
  )

  $(foreach dep,$(DOWNLOAD_RDEP),
    $(dep): $(DL_DIR)/$(FILE)
  )
  download: $(DL_DIR)/$(FILE)

  $(DL_DIR)/$(FILE):
        mkdir -p $(DL_DIR)
        $(if $(DownloadMethod/$(call dl_method,$(URL),$(PROTO))),$(DownloadMethod/$(call dl_method,$(URL),$(PROTO))),$(DownloadMethod/unknown))

endef
paidforby commented 6 years ago

Haha. Manually downloading lzo during the pre-build works. It no longer gets stuck at downloading lzo. Of course, now it fails after 64 minutes when it tries to download ncurses. Further investigation to find the root cause of these download failures.

paidforby commented 6 years ago

make package/libs/ncurses/compile V=s appears to manually trigger the download and installation of the package. I can't seem to find how to force make download to pick up these stray packages. Depending on how many there are, it may not be so terrible to trigger them manually?

paidforby commented 6 years ago

Amazingly, I haven't given up on this yet. So far I've found that the following packages are not downloaded by make download:

However, it is becoming unbearable to work on this any longer. It requires running the full build for 50-60mins, finding which package fails, adding the package to the build_lib like so, rebuilding the container and trying again. There must be a better way of either finding which packages haven't been downloaded or looping through all the problem directories and force downloading the packages (shouldn't that be what make download does...sigh)

paidforby commented 6 years ago

So proposed solution is something like this,

for d in $(find package/ -mindepth 1 -maxdepth 2 -type d); do
    if [ -e $d/Makefile ]
    then
      make $d/download V=s
    else
      echo "not a package"
   fi
done

But it errors out on some directories that are not packages or on dead links. Not sure how to avoid/ignore these?

gobengo commented 6 years ago

@paidforby You may need to call readlink on the file, not just check if it exists, in case it's a link. https://linux.die.net/man/1/readlink

paidforby commented 6 years ago

Hmmm, @gobengo I'm not sure how you are suggesting to use readlink. Here's the error produced by the proposed solution,

make[2]: Entering directory `/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/package/kernel/ar7-atm'
mkdir -p /usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/dl
/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/scripts/download.pl "/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/dl" "sangam_atm--.tar.bz2" "unknown" "http://mirror2.openwrt.org/sources"
--2018-05-23 18:21:11--  http://mirror2.openwrt.org/sources/sangam_atm--.tar.bz2
Resolving mirror2.openwrt.org (mirror2.openwrt.org)... 46.4.11.11, 2a01:4f8:221:3d45::2
Connecting to mirror2.openwrt.org (mirror2.openwrt.org)|46.4.11.11|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-05-23 18:21:12 ERROR 404: Not Found.

Download failed.
--2018-05-23 18:21:12--  http://mirror2.openwrt.org/sources/sangam_atm--.tar.bz2
Resolving mirror2.openwrt.org (mirror2.openwrt.org)... 46.4.11.11, 2a01:4f8:221:3d45::2
Connecting to mirror2.openwrt.org (mirror2.openwrt.org)|46.4.11.11|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-05-23 18:21:12 ERROR 404: Not Found.

Download failed.
--2018-05-23 18:21:12--  http://downloads.openwrt.org/sources/sangam_atm--.tar.bz2
Resolving downloads.openwrt.org (downloads.openwrt.org)... 148.251.78.235, 2a01:4f8:202:43ea::2
Connecting to downloads.openwrt.org (downloads.openwrt.org)|148.251.78.235|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://sources.openwrt.org/sangam_atm--.tar.bz2 [following]
--2018-05-23 18:21:13--  http://sources.openwrt.org/sangam_atm--.tar.bz2
Resolving sources.openwrt.org (sources.openwrt.org)... 148.251.78.235, 2a01:4f8:202:43ea::3
Reusing existing connection to downloads.openwrt.org:80.
HTTP request sent, awaiting response... 404 Not Found
2018-05-23 18:21:13 ERROR 404: Not Found.

Download failed.
No more mirrors to try - giving up.
make[2]: *** [/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/dl/sangam_atm--.tar.bz2] Error 2
make[2]: Leaving directory `/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx/package/kernel/ar7-atm'
make[1]: *** [package/kernel/ar7-atm/download] Error 2
make[1]: Leaving directory `/usr/local/sudowrt-firmware/built_firmware/builder.ar71xx'
make: *** [package/kernel/ar7-atm/download] Error 2
paidforby commented 6 years ago

Updates to the build process described in https://github.com/sudomesh/sudowrt-firmware/issues/137#issuecomment-410417514 address this problem. You can now run with --net=none after pulling the latest docker image,

sudo docker pull sudomesh/sudowrt-firmware:latest
sudo docker run --net=none -v $PWD/firmware_images:/firmware_images sudomesh/sudowrt-firmware:latest

However, we shouldn't use docker run anymore, since that inhibits the ability to rebuild the firmware, you can replicate its functionality with

sudo docker create --net=none -v $PWD/firmware_images:/firmware_images -name sudowrt-container sudomesh/sudowrt-firmware:latest 
sudo docker start -a sudowrt-container

Closing issue, as I feel it has been adequately addressed, if @jhpoelen disagrees, please reopen.