search

OnionIoT / openwrt-imagebuilder-wrapper

Easily build firmware images using the OpenWRT Build System
MIT License
1 stars 3 forks source link

Upgrade to using Onion customized OpenWRT 22.03.3 image builder #4

Closed greenbreakfast closed 1 year ago

greenbreakfast commented 1 year ago

Motivation

Use the Onion-customized OpenWRT SDK and Image Builder to build openwrt-22.03 firmware. This will allow for additional packages like omega2-ctrl and NodeJS to be compiled and installed on device.

Description

Update this repo and OnionIoT/openwrt-sdk-wrapper to use the Onion-customized OpenWRT 22.03.3 of the SDK and image builder.

The Onion-customized SDK and Image Builder can be found here: http://downloads.onioniot.com/releases/22.03.3/targets/ramips/mt76x8/

For any device-side things that need to change, please update the openwrt-22.03 branch in OnionIoT/openwrt-packages.

Output

The output firmware should have opkg configured to:

More details to follow on the above once https://github.com/OnionIoT/openwrt-buildsystem-wrapper/issues/7 is complete Updated with URLs for package repos

Dependencies

  1. ✅ Complete https://github.com/OnionIoT/openwrt-imagebuilder-wrapper/issues/3 - migrate to vanilla 22.03.3 to shake out any migration issues
  2. ✅ Complete https://github.com/OnionIoT/openwrt-buildsystem-wrapper/issues/7 - Setup CI for OnionIoT/openwrt-buildsystem-wrapper repo and host output binaries and package files - @greenbreakfast to take care of this
greenbreakfast commented 1 year ago

@jempatel please take this up next

jempatel commented 1 year ago

@jempatel please take this up next

@greenbreakfast There are some issues with imagebuilder. It generates Python host-side softlink to Python virtual environment. The original OpenWrt Image builder picks the Python path from the standard path. This build machine is set up a bit differently. This causes the build to fail through Onion Image builder.

greenbreakfast commented 1 year ago

Is this something configuration related? 

If so, please make a bug report issue with a description of the problem, current behaviour, expected behaviour, proposed solution, etc.  And I’ll ask our CD guy to take a look and implement a solution.

greenbreakfast commented 1 year ago

@jempatel The python path issue is resolved - https://github.com/OnionIoT/openwrt-buildsystem-wrapper/issues/12

Please try again with the latest release of the SDK and image builder: http://downloads.onioniot.com/releases/22.03.3/targets/ramips/mt76x8/

greenbreakfast commented 1 year ago

Merged https://github.com/OnionIoT/openwrt-sdk-wrapper/pull/4 and https://github.com/OnionIoT/openwrt-imagebuilder-wrapper/pull/6 PRs. Will build new firwmare, test it out, and report back.

greenbreakfast commented 1 year ago

@jempatel hmm CD is getting an error when building OnionIoT/openwrt-imagebuilder-wrapper:

Configuring libpthread.
Configuring opkg.
ConfigurCollected errors:
 * opkg_install_cmd: Cannot install package onion-repo-keys.
 * opkg_install_cmd: Cannot install package omega2-base.
 * opkg_install_cmd: Cannot install package omega2-base-files.
 * opkg_install_cmd: Cannot install package omega2-base-passwd.
ing kmod-nls-base.
Configuring kmod-usb-core.
Configuring libjson-c5.
...
make[2]: *** [Makefile:169: package_install] Error 255
make[1]: *** [Makefile:124: _call_image] Error 2
make: *** [Makefile:242: image] Error 2

But it does seem to read the Onion package repo correctly:

Downloading http://repo.onioniot.com/omega2/packages/openwrt-22.03.3/onion/Packages.gz
Updated list of available packages in /codebuild/output/src508862575/src/openwrt-imagebuilder/build_dir/target-mipsel_24kc_musl/root-ramips/../../../../../../../../codebuild/output/src508862575/src/openwrt-imagebuilder/dl/onion_packages
Downloading http://repo.onioniot.com/omega2/packages/openwrt-22.03.3/onion/Packages.sig
Signature check passed.

Can you please look into this? Attached the full log: log-events-viewer-result.csv

jempatel commented 1 year ago

http://repo.onioniot.com/omega2/packages/openwrt-22.03.3/onion/Packages.gz

This seems to be issue with http://repo.onioniot.com/omega2/packages/openwrt-22.03.3/onion/Packages.gz. Packages.gz file should contain package metadata but there is no content in Packages file.

For verification, I used local path for ipk file and it was passed.

jempatel commented 1 year ago

There is also problem with SDK, when I used SDK If failed in toolchain compilation with an error

/media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk/staging_dir/host/bin/fakeroot /media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk/staging_dir/host/bin/bash /media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk/scripts/ipkg-build -m "" /media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk/build_dir/target-mipsel_24kc_musl/toolchain/ipkg-mipsel_24kc/libgcc /media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk/bin/targets/ramips/mt76x8/packages
/media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk/staging_dir/host/bin/bash: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk/staging_dir/host/lib/libfakeroot.so)
make[2]: *** [Makefile:737: /media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk/bin/targets/ramips/mt76x8/packages/libgcc1_11.2.0-4_mipsel_24kc.ipk] Error 1
make[2]: Leaving directory '/media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk/package/toolchain'
time: package/toolchain/compile#0.68#0.71#1.12
    ERROR: package/toolchain failed to build.
make[1]: *** [package/Makefile:116: package/toolchain/compile] Error 1
make[1]: Leaving directory '/media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk'
make: *** [/media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk/include/toplevel.mk:223: package/onion-repo-keys/compile] Error 2
make: Leaving directory '/media/Data/Source/gitrepo/OnionIoT/openwrt-sdk-wrapper/openwrt-sdk'

I think this, build env still needs other fixes or we need to defined what os needs to be used for it to work. The system I use, its using libc-2.31.

greenbreakfast commented 1 year ago

@jempatel the SDK + Image builder CD has been fixed and a new firmware with these changes is available: http://repo.onioniot.com.s3.amazonaws.com/omega2/images/openwrt-22.03/onion_omega2p-22.03.3-20230504.bin

jempatel commented 1 year ago

@jempatel the SDK + Image builder CD has been fixed and a new firmware with these changes is available: http://repo.onioniot.com.s3.amazonaws.com/omega2/images/openwrt-22.03/onion_omega2p-22.03.3-20230504.bin

I found an issue in this firmware, It keeps rebooting with kernel panic message.

[    1.042888] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[    1.057286] Rebooting in 1 seconds..

I will take a look into this.

greenbreakfast commented 1 year ago

@jempatel I tried out the vanilla firmware built by the buildsystem wrapper (http://downloads.onioniot.com.s3.amazonaws.com/releases/22.03.3/targets/ramips/mt76x8/openwrt-22.03.3-ramips-mt76x8-onion_omega2p-squashfs-sysupgrade.bin), works fine:

image

I guess this issue is introduced with the imagebuilder?

greenbreakfast commented 1 year ago

@jempatel could the issue be caused by the imagebuilder still using the kernel modules compiled by OpenWRT?

Screen Shot 2023-05-08 at 4 43 13 PM
jempatel commented 1 year ago

I tried removing and replacing it with onion_core but it's still getting kernel panic.

greenbreakfast commented 1 year ago

Hi @jempatel, good work! I've merged in the 3 PRs and a new build has been triggered. When it's done I'll try it out and post an update.

EDIT: Build complete and fw available at: http://repo.onioniot.com/omega2/images/openwrt-22.03/onion_omega2p-22.03.3-20230517.bin Tested it on a device and it works fine boots but has some issues. EDIT 2: See notes on firmware issues in the comment below

I've opened issues in the sdk and image builder wrappers for some minor items, please take a look when you get a chance.

Questions

Also, I had a few questions about the changes to the omega2-base package in OnionIoT/OpenWRT-Packages:

1. Dependencies Removed

I noticed https://github.com/OnionIoT/OpenWRT-Packages/pull/55 removes the following dependencies from the omega2-base package:

How come we remove these packages? Do they cause issues with SDK compilation?

2. Opkg distfeeds change for openwrt_core repo

The distfeeds.conf will now be modified to set the URL for the openwrt_core repo to http instead of https:

https://github.com/OnionIoT/OpenWRT-Packages/blob/84b5ff204d31e6ed7597cd41f381a420774cf798/omega2-base/files/etc/uci-defaults/98_opkg#L14-L15

Is this to avoid reading the package list from this repo? If so, wouldn't the removal of the openwrt repo keys stop the Omega from reading this repo anyway?

greenbreakfast commented 1 year ago

Issues with Latest Firmware

@jempatel please take a look at these issues with the latest firmware built by the sdk and image builder

Context

Issue 1

Cannot ping any URLs:

root@Omega-f195:/tmp# ping www.google.com
ping: bad address 'www.google.com'

Issue 2

wget not able to download any files

root@Omega-f195:/tmp# wget http://repo.onioniot.com.s3.amazonaws.com/omega2/imag
es/openwrt-22.03/onion_omega2p-22.03.3-20230517.bin
Downloading 'http://repo.onioniot.com.s3.amazonaws.com/omega2/images/openwrt-22.03/onion_omega2p-22.03.3-20230517.bin'
Failed to send request: Operation not permitted
root@Omega-f195:/tmp# wget https://archive.openwrt.org/releases/22.03.3/targets/
ramips/mt76x8/packages/Packages
Downloading 'https://archive.openwrt.org/releases/22.03.3/targets/ramips/mt76x8/packages/Packages'
Failed to send request: Operation not permitted

Issue 3

opkg update doesn't work:

root@Omega-f195:/tmp# opkg update
Downloading http://downloads.onioniot.com/releases/22.03.3/targets/ramips/mt76x8/packages/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.onioniot.com/releases/22.03.3/targets/ramips/mt76x8/packages/Packages.gz

Downloading http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/base/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/base/Packages.gz

Downloading http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/packages/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/packages/Packages.gz

Downloading http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/routing/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/routing/Packages.gz

Downloading http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/telephony/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/telephony/Packages.gz

Downloading http://repo.onioniot.com/omega2/packages/openwrt-22.03.3/onion/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://repo.onioniot.com/omega2/packages/openwrt-22.03.3/onion/Packages.gz

Downloading http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/base/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/base/Packages.gz

Downloading http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/luci/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/luci/Packages.gz

Downloading http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/packages/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/packages/Packages.gz

Downloading http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/routing/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/routing/Packages.gz

Downloading http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/telephony/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/telephony/Packages.gz

Collected errors:
 * opkg_download: Failed to download http://downloads.onioniot.com/releases/22.03.3/targets/ramips/mt76x8/packages/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/base/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/packages/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/routing/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/telephony/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://repo.onioniot.com/omega2/packages/openwrt-22.03.3/onion/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/base/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/luci/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/packages/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/routing/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.openwrt.org/releases/22.03.3/packages/mipsel_24kc/telephony/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.
DocHardinger commented 1 year ago

Just a suggestion, we had this problem with ipv6 enabled. If we disabled ipv6 we can run opkg and ping google. But dont know if this is the same issue

greenbreakfast commented 1 year ago

@DocHardinger thanks for the suggestion, I just tried it out but no luck. Did I miss something when disabling ipv6?

Disabling ipv6:

uci set network.lan.ipv6=0
uci set network.wan.ipv6=0
uci set network.wwan.ipv6=0
uci set network.wlan.ipv6=0
uci commit
/etc/init.d/network restart

Seems like it took effect since no interfaces have ipv6 addresses:

root@Omega-f195:/# ifconfig
apcli0    Link encap:Ethernet  HWaddr 40:A3:6B:C0:F1:95
          inet addr:192.168.0.39  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:762 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:272170 (265.7 KiB)  TX bytes:1074 (1.0 KiB)

eth0      Link encap:Ethernet  HWaddr 40:A3:6B:C0:F1:96
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:5

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:3280 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3280 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:315512 (308.1 KiB)  TX bytes:315512 (308.1 KiB)

ra0       Link encap:Ethernet  HWaddr 42:A3:6B:C0:F1:95
          inet addr:192.168.3.1  Bcast:192.168.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

But it still can't successfully ping google or do an opkg update

root@Omega-f195:/# ping www.google.com
ping: bad address 'www.google.com'
root@Omega-f195:/# ping google.com
ping: bad address 'google.com'
root@Omega-f195:/# opkg update
Downloading http://downloads.onioniot.com/releases/22.03.3/targets/ramips/mt76x8/packages/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.onioniot.com/releases/22.03.3/targets/ramips/mt76x8/packages/Packages.gz

Downloading http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/base/Packages.gz
Failed to send request: Operation not permitted
*** Failed to download the package list from http://downloads.onioniot.com/releases/22.03.3/packages/mipsel_24kc/base/Packages.gz
DocHardinger commented 1 year ago

Hmm than it seems to be another problem. We had this issue when using original openwrt 22.03. Firmware with custom packages.

By now im on holidays bit will be back next week, then i will also do some testing with the new firmware builder.

jempatel commented 1 year ago

The latest node Makefile depends on HAS_FPU, and when the firmware is built using the HAS_FPU config, it compiles everything with FPU, including toolchain, SDK, image builder, and packages. Because of that, when the packages are installed from the upstream openwrt repo, the binaries installed are not usable because they can not detect executables and fail with the error file not found.

Even a few of the default packages are not working even after being compiled natively with HAS_FPU. i.e. awk command was not working, because of which, when omega2 firmware is installed, It does not resolve the DNS name. When omega2 firmware boots first time, it starts the dnsmasq service. dnsmasq service init scripts internally use the awk command to generate the dnsmasq daemon config, because of the misbehavior of awk, It generates the wrong config, causing failure in starting the dnsmasq service. Due to this, it does not generate correct /etc/resolv.conf which ultimately causes failure in DNS resolution on omega2.

In the working version of the node, It was only dependent on HAS_FPU or KERNEL_MIPS_FPU_EMULATOR. If we follow the same in a newer node version, It fails in compilation with an error in __detect_fp64_mode.

__detect_fp64_mode, executes assembly code, but as omega2 does not have hardware FPU, we had to add a patch always to return 0 from the __detect_fp64_mode and add gcc flag --with-mips-float-abi=soft in the compilation.

With these changes, the node gets compiled with soft float abi.

TT-E1NFACH commented 1 year ago

@greenbreakfast

i now build the firmware and installed it on an omega2 pro. Established a connection to my wifi and can successfully ping google and run opkg update. Changed nothing else except adding some packages like python3 to the firmware. So for me it works fine.

TT-E1NFACH commented 1 year ago

Sorry for this, my statement seems not to be correct, im investigating...ignore this for now!

I discovered an issue when using the firmware builder.

if i put some config files under additions/files/etc/config for example "network" all other files which usually will be created automatically disappear in the firmware. When using original OPenWrt firmware builder the files will be created automatically except the one i put inside files/etc/config (network).

So when trying to boot with the created firmware it stops booting because files are missing.

... [ 11.315550] block: attempting to load /etc/config/fstab [ 11.330432] block: unable to load configuration (fstab: Entry not found) [ 11.337766] block: no usable configuration [ 11.343222] mount_root: jffs2 not ready yet, using temporary tmpfs overlay [ 11.368881] urandom-seed: Seed file not found (/etc/urandom.seed) ...

can anyone confirm this?

greenbreakfast commented 1 year ago

i now build the firmware and installed it on an omega2 pro. Established a connection to my wifi and can successfully ping google and run opkg update. Changed nothing else except adding some packages like python3 to the firmware. So for me it works fine.

@TT-E1NFACH great to hear. And yep, this is the result of @jempatel's great work:

The 22.03.3-20230526 firmware (available at http://repo.onioniot.com/omega2/images/openwrt-22.03/) resolves the issues mentioned above and can successfully connect to wifi, ping remote servers, opkg update, and download with wget.

I think we can safely say the Onion openwrt-22.03 beta firmware has been upgraded to using the Onion customized OpenWRT 22.03.3 SDK and image builder, so I am going to close this issue.

@TT-E1NFACH regarding your problem with the image builder, please make a new issue in this repo so we can better track the bug. Please include a "Steps to reproduce" section for best results.