arduino / openwrt-yun

A custom version of OpenWrt, targeted to the Arduino Yún
GNU General Public License v2.0
148 stars 71 forks source link

ImageBuilder possible for YUN ? #20

Closed MaStr closed 10 years ago

MaStr commented 10 years ago

Hi guys, thank you for pushing the Yun forward. It is a nice piece of hardware.

Do you think it is possible to supply for your current release the thing called "ImageBuilder"? It is a packed stripped down image-build environment, which can be used to create custom images. PirateBox & LibraryBox use this package and a clever Makefile to create images containing our packages. (see https://github.com/PirateBox-Dev/openwrt-image-build/tree/AA-with-installer)

With this kind of prepared images, the installation of LibraryBox/PirateBox becomes more a "wait until it is done" scenario over hacking stuff in the command line. In the same time, we stay compatible to the official kernel modules etc.

edit: What the hell is ImageBuilder? http://wiki.openwrt.org/doc/howto/obtain.firmware.generate ; A Elegant way to refit the stock binary firmwares with other software then install without recompiling everything.

What do you think?

Matthias

MaStr commented 10 years ago

Providing an working imagebuilder package would save trees. (less compile time all over the world :) )

ffissore commented 10 years ago

I'm building it right now. Is the imagebuilder a native package? That is: if I compile it on a 64 bit ubuntu, will it run on a 32 bit redhat?

ffissore commented 10 years ago

Just built and no. Image builder is built with the host architecture, making it less useful. Hints?

MaStr commented 10 years ago

Yes, this is the limitation of the imagebuilder, but to be honest - I used the 32bit openwrt-imagebuilder on my 64bit machine without issues for one year. OpenWRT-Trunk imagebuilder is now based upon 64bit.

As the OpenWrt-Yun compile worked only on my 32bit-Debian VM inside my 64Bit machine, I don't think that is a really problematic situation.

About cross distribution&32/64bit ? I think it depends on the 32bit Libraries you have installed. The stuff OpenWRT delivers on stock, worked on my boxes usually without additional work.

I haven't noticed limitations beside the 62/32bit limitation. I tried Debian64 , Debian32, Arch64, gentoo32, gentoo64- I can't remember an additional installed packaged to get it working.

regards Matthias

comp-hx commented 10 years ago

How do I use the imagebuilder to build my own application that can be loaded onto and run (flashed) on the openwrt part of the yun?

MaStr commented 10 years ago

Hi comp-hy, first you need to compile the complete stuff via the yun-build chain described here at the repository. After this you run make menuconfig and select imagebuilder, save & exit. Then run the command make to rerun the compile process. This time the compile is much faster, because all the stuff is already compiled and at the end of the process you get a tar.bz (?) file with the name "imagebuilder" in it (You find it in the bin/ar71xx folder).

Copy the imagebuilder file to a different place and extract it. When you simply run make you get some instructions. More information can be found here: http://wiki.openwrt.org/doc/howto/obtain.firmware.generate

You can copy your own files into the image and/or install additional packages from the stock repository (and additional repositories you configure). So, If you want to install your own software, my recommended way is to create a packages for the software and a repository. (That would extend the range of the issue here).

You have to be aware of the point, that with imagebuilder only a "bare-bone" image of the Yun-image is shipped. There is no web-interface and all the custom Yun packages are not included into that image, only the needed drivers to get the yun working. (That is the reason for asking in #21 what the original Yun packages are).

Hope that helps Matthias

ffissore commented 10 years ago

Can you give a spin to this? http://downloads.arduino.cc/openwrtyun/1/OpenWrt-ImageBuilder-ar71xx_generic-for-linux-x86_64.tar.bz2

MaStr commented 10 years ago

Default Packages:

Default Packages: base-files libc libgcc busybox dropbear mtd uci opkg hotplug2 netifd kmod-leds-gpio kmod-gpio-button-hotplug kmod-wdt-ath79 swconfig kmod-ledtrig-default-on kmod-ledtrig-timer kmod-ledtrig-netdev kmod-ath9k wpad-mini uboot-envtools dnsmasq iptables ppp ppp-mod-pppoe kmod-ipt-nathelper firewall

And the specific profile (additional to packages above):

Yun:
    Arduino Yun based on Atheros AR9331
    Packages: kmod-usb-core kmod-usb2

(Between all the other listed possible profiles).

At least the minimum Profile is there (OK). If you don't have a "must have" requirement, that is missing above, this can be used yes. I haven't tested it for myself yet.

You maybe can exclude the other profiles from your build-chain, because - as far as I recognized it - do changes in the "filesystem" without a target-profile dependency, right? These may fail, like this change

wifi-live-or-reset
boot-complete-notify

That leads me directly to my question in: #21 What packages do you recommend for the Yun-Arduino functionality. What should be installed? > would be nice, if you can come back with this question with a wiki page (list, which of your packages does what on the yun?)

Thanks for you support, I really appreciate it!

Matthias

ffissore commented 10 years ago

@MaStr I now fully understand your need for a meta package. I'm not expert with the ImageBuilder, so I'm sorry but I won't be able to help much.

You can get the complete list of built-in packages with grep '=y' .config. Many entries are not packages: they are options for other packages. For example CONFIG_BUSYBOX_CONFIG_TAR=y means that busybox (the whole-in-one binary that provides the base utilities) must support tar command. The real package is CONFIG_PACKAGE_busybox=y

Can you help us here?

MaStr commented 10 years ago

@ffissore It is not a need , it is more like it is good to have, it would be better to have a description of the components you use on your image for specific things.

I just tested you config file. I would start from the end of the file and figure out the flagged packages, for this I just figured that command out grep PACKAGE .config | grep \=y . All packages seems to have the keyword "PACKAGE".

Then you get a long list with enabled packages:

CONFIG_PACKAGE_base-files=y
CONFIG_PACKAGE_block-mount=y
CONFIG_PACKAGE_busybox=y
CONFIG_PACKAGE_dnsmasq=y
CONFIG_PACKAGE_dropbear=y
CONFIG_PACKAGE_firewall=y
CONFIG_PACKAGE_hotplug2=y
CONFIG_PACKAGE_libc=y
CONFIG_PACKAGE_libgcc=y
CONFIG_PACKAGE_libpthread=y
CONFIG_PACKAGE_librt=y
CONFIG_PACKAGE_mtd=y
CONFIG_PACKAGE_netifd=y
CONFIG_PACKAGE_opkg=y
CONFIG_PACKAGE_swconfig=y
CONFIG_PACKAGE_uSDaemon=y
CONFIG_PACKAGE_ubus=y
CONFIG_PACKAGE_ubusd=y
CONFIG_PACKAGE_uci=y
CONFIG_PACKAGE_wireless-tools=y
CONFIG_PACKAGE_yun-conf=y
CONFIG_PACKAGE_yun-scripts=y
CONFIG_PACKAGE_luci=y
CONFIG_PACKAGE_luci-mod-admin-core=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-app-arduino-webpanel=y
CONFIG_PACKAGE_luci-app-firewall=y
CONFIG_PACKAGE_luci-theme-base=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
CONFIG_PACKAGE_luci-theme-openwrt=y
CONFIG_PACKAGE_luci-i18n-english=y
CONFIG_PACKAGE_luci-proto-core=y
CONFIG_PACKAGE_luci-proto-ppp=y
CONFIG_PACKAGE_luci-sgi-cgi=y
CONFIG_PACKAGE_luci-lib-core=y
CONFIG_PACKAGE_luci-lib-core_compile=y
CONFIG_PACKAGE_luci-lib-ipkg=y
CONFIG_PACKAGE_luci-lib-json=y
CONFIG_PACKAGE_luci-lib-nixio=y
CONFIG_PACKAGE_luci-lib-nixio_cyassl=y
CONFIG_PACKAGE_luci-lib-sys=y
CONFIG_PACKAGE_luci-lib-web=y
CONFIG_PACKAGE_kmod-loop=y
CONFIG_PACKAGE_kmod-scsi-core=y
CONFIG_PACKAGE_kmod-crypto-aes=y
CONFIG_PACKAGE_kmod-crypto-arc4=y
CONFIG_PACKAGE_kmod-crypto-core=y
CONFIG_PACKAGE_kmod-crypto-hash=y
CONFIG_PACKAGE_kmod-crypto-manager=y
CONFIG_PACKAGE_kmod-fs-ext4=y
CONFIG_PACKAGE_kmod-fs-hfs=y
CONFIG_PACKAGE_kmod-fs-hfsplus=y
CONFIG_PACKAGE_kmod-fs-ntfs=y
CONFIG_PACKAGE_kmod-fs-reiserfs=y
CONFIG_PACKAGE_kmod-fs-vfat=y
CONFIG_PACKAGE_kmod-fuse=y
CONFIG_PACKAGE_kmod-i2c-core=y
CONFIG_PACKAGE_kmod-leds-gpio=y
CONFIG_PACKAGE_kmod-ledtrig-default-on=y
CONFIG_PACKAGE_kmod-ledtrig-netdev=y
CONFIG_PACKAGE_kmod-ledtrig-timer=y
CONFIG_PACKAGE_kmod-lib-crc-ccitt=y
CONFIG_PACKAGE_kmod-lib-crc16=y
CONFIG_PACKAGE_kmod-nls-base=y
CONFIG_PACKAGE_kmod-nls-utf8=y
CONFIG_PACKAGE_kmod-ipt-core=y
CONFIG_PACKAGE_kmod-ipt-conntrack=y
CONFIG_PACKAGE_kmod-ipt-nat=y
CONFIG_PACKAGE_kmod-ipt-nathelper=y
CONFIG_PACKAGE_kmod-ppp=y
CONFIG_PACKAGE_kmod-pppoe=y
CONFIG_PACKAGE_kmod-pppox=y
CONFIG_PACKAGE_kmod-gpio-button-hotplug=y
CONFIG_PACKAGE_kmod-wdt-ath79=y
CONFIG_PACKAGE_kmod-usb-core=y
CONFIG_PACKAGE_kmod-usb-ohci=y
CONFIG_PACKAGE_kmod-usb-storage=y
CONFIG_PACKAGE_kmod-usb2=y
CONFIG_PACKAGE_kmod-video-core=y
CONFIG_PACKAGE_kmod-ath=y
CONFIG_PACKAGE_ATH_DFS=y
CONFIG_PACKAGE_kmod-ath9k=y
CONFIG_PACKAGE_kmod-ath9k-common=y
CONFIG_PACKAGE_kmod-cfg80211=y
CONFIG_PACKAGE_kmod-mac80211=y
CONFIG_PACKAGE_MAC80211_DEBUGFS=y
CONFIG_PACKAGE_MAC80211_MESH=y
CONFIG_PACKAGE_uboot-ar71xx-linino=y
CONFIG_PACKAGE_curl=y
CONFIG_PACKAGE_wget=y
CONFIG_PACKAGE_iptables=y
CONFIG_PACKAGE_avahi-daemon=y
CONFIG_PACKAGE_avahi-dnsconfd=y
CONFIG_PACKAGE_avahi-utils=y
CONFIG_PACKAGE_uhttpd=y
CONFIG_PACKAGE_uhttpd-mod-lua=y
CONFIG_PACKAGE_uhttpd-mod-tls=y
CONFIG_PACKAGE_uhttpd-mod-tls_cyassl=y
CONFIG_PACKAGE_uhttpd-mod-ubus=y
CONFIG_PACKAGE_iw=y
CONFIG_PACKAGE_ppp=y
CONFIG_PACKAGE_ppp-mod-pppoe=y
CONFIG_PACKAGE_spacebrew=y
CONFIG_PACKAGE_temboo=y
CONFIG_PACKAGE_wpad-mini=y
CONFIG_PACKAGE_libip4tc=y
CONFIG_PACKAGE_libip6tc=y
CONFIG_PACKAGE_libxtables=y
CONFIG_PACKAGE_libcyassl=y
CONFIG_PACKAGE_libgnutls=y
CONFIG_PACKAGE_libopenssl=y
CONFIG_PACKAGE_argp-standalone=y
CONFIG_PACKAGE_libassuan=y
CONFIG_PACKAGE_libavahi=y
CONFIG_PACKAGE_libavahi-client=y
CONFIG_PACKAGE_libavahi-dbus-support=y
CONFIG_PACKAGE_libblkid=y
CONFIG_PACKAGE_libblobmsg-json=y
CONFIG_PACKAGE_libcurl=y
CONFIG_PACKAGE_libdaemon=y
CONFIG_PACKAGE_libdbus=y
CONFIG_PACKAGE_libelf=y
CONFIG_PACKAGE_libexpat=y
CONFIG_PACKAGE_libffi=y
CONFIG_PACKAGE_libftdi=y
CONFIG_PACKAGE_libgcrypt=y
CONFIG_PACKAGE_libgdbm=y
CONFIG_PACKAGE_libgpg-error=y
CONFIG_PACKAGE_libgpgme=y
CONFIG_PACKAGE_libiwinfo=y
CONFIG_PACKAGE_libjson=y
CONFIG_PACKAGE_libjson-c=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_libncurses=y
CONFIG_PACKAGE_libnl-tiny=y
CONFIG_PACKAGE_libpcre=y
CONFIG_PACKAGE_libreadline=y
CONFIG_PACKAGE_libubox=y
CONFIG_PACKAGE_libubus=y
CONFIG_PACKAGE_libubus-lua=y
CONFIG_PACKAGE_libuci=y
CONFIG_PACKAGE_libuci-lua=y
CONFIG_PACKAGE_libusb=y
CONFIG_PACKAGE_libusb-1.0=y
CONFIG_PACKAGE_libuuid=y
CONFIG_PACKAGE_terminfo=y
CONFIG_PACKAGE_zlib=y
CONFIG_PACKAGE_nano=y
CONFIG_PACKAGE_blkid=y
CONFIG_PACKAGE_swap-utils=y
CONFIG_PACKAGE_avrdude=y
CONFIG_PACKAGE_cpu-mcu-bridge=y
CONFIG_PACKAGE_dbus=y
CONFIG_PACKAGE_gnupg=y
CONFIG_PACKAGE_jshn=y
CONFIG_PACKAGE_libjson-script=y
CONFIG_PACKAGE_px5g=y
CONFIG_PACKAGE_rng-tools=y
CONFIG_PACKAGE_triggerhappy=y
CONFIG_PACKAGE_uboot-envtools=y
CONFIG_PACKAGE_usbutils=y
CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_python=y
CONFIG_PACKAGE_python-json=y
CONFIG_PACKAGE_python-mini=y

The problem is, that you only see the flagged packages, but not why they are flagged. i.e. python depends on python-mini (I think). So you select in menuconfig "python" and python-mini is flagged too.

Now, you have to remember, why you activate a package like python-mini . Is it because you need it in on of you "Yun" scripts, is it for practical reasons for the user? I think "nano" is a package that is installed for easier editing on the yun.

Do you have any self made packages in your repository? How did you apply your custom web-ui? Which packages are needed for the interaction between the OpenWrt and the Arduino?

Based upon those questions, I would try to create a dependency-list with additional meta-package like:

Yun-meta
    +--> Yun-UI (meta)
    |        +--->  uhttpd   (original openwrt package)
    |                     +-- (.. pulls its normal dependencies )
    |        +--->  lua
    |        +--->  Web-Interface 
    |        +--->  Python
    +--> Yun-Arduino - Hardware -Bridge support
    |        +--->  foo
    +--> Yun-Comfort-Packages
    |        +--->  nano
    |        +--->  python
    +--> Yun custom configuration
    |        +---> config files like rc.local 
    |                 maybe additional package repository??!!
.....

Sounds complicated in the first place compared to easily selecting it, but my idea behind is:

  1. Customizing your own image via ImageBuilder in a way, that the user can decide if he wants to have the UI packages or not .. and those (including the dependency) are not installed to save space
  2. You can later easily maintain your stuff, when you switch OpenWrt versions. Especially if OpenWrt maybe really drops python support, you know in which context you added it. As you can see in my example above, you can add python in two packages as dependency... like you use it in the UI and you want the user to have a better CLI user-experience.
  3. You don't need to remember where and why you added the package xy

Let my know what you think

Matthias

edit: You can create "module" packages for the Yun-meta package. Yun-Meta-UI , that calls the dependencies. These Modules can be created within the same Makefile like the yun-meta package. Example how to create it can be found here: https://github.com/PirateBox-Dev/openwrt-piratebox-feed/blob/master/utils/usb-config-scripts/Makefile and here https://github.com/PirateBox-Dev/openwrt-piratebox-feed/blob/master/utils/extendRoot/Makefile

ffissore commented 10 years ago

The problem is, that you only see the flagged packages, but not why they are flagged. i.e. python depends on python-mini (I think). So you select in menuconfig "python" and python-mini is flagged too.

That information is lost once you select a package from the text UI opened with make menuconfig: dependent packages are properly marked as "m" or "y", but you have no feedback or trace about which one and why

Do you have any self made packages in your repository?

Yes, they are in https://github.com/arduino/openwrt-packages-yun, "arduino" folder

How did you apply your custom web-ui?

It's a selected package https://github.com/arduino/openwrt-yun/blob/master/.config#L1480

Which packages are needed for the interaction between the OpenWrt and the Arduino?

Those marked with "y" from this list https://github.com/arduino/openwrt-packages-yun/tree/master/arduino

Based upon those questions, I would try to create a dependency-list with additional meta-package like:

That's super good: can't wait to see it. Thank you!