Board auto detection broken by design

ThomasKaiser commented 8 years ago

Relying on the presence of certain hardware features like internal Terminus USB hub turns out to be wrong.

A shortterm solution would be to either create images for every board individually again or tweak our build process so that a not yet implemented final install hook takes one base image and replaces script.bin/dtb stuff, cpufreq and /etc/modules (that should be all that's needed to serve H3 and A20 boards, haven't thought about the others yet) or do it as now and use another platform specific hook in check_first_login.sh to let the user choose the board in question.

A midterm solution could be something ssvb outlined/prepared over a year ago. We could already do this type of detection/choice from within u-boot (USB works now also with H3 [1]). But IMO it's no good idea to loose the ability to setup a device headless since this is one of the reasons I chose Armbian in the beginning. We're not in a hurry and should weigh the pros/cons with care.

But regarding the shortterm solution blowing up configuration.sh again (containing all Bananas/Oranges indivudally again) seems easy but still requires some hackish workarounds (we use one single .dtb for all Oranges != Plus since... it just works). So what about taking the time and exploring whether we could implement this sort of 'final step' and being able to just select 'H3', let a single image create and in a final step insert all the needed stuff for the individual boards in a few seconds?

[1] Console output with 2016.03 on Orange Pi PC:

U-Boot 2016.03-armbian (Mar 17 2016 - 11:23:31 +0100) Allwinner Technology

CPU:   Allwinner H3 (SUN8I)
...
starting USB...
USB0:   USB EHCI 1.00
USB1:   USB OHCI 1.0
USB2:   USB EHCI 1.00
USB3:   USB OHCI 1.0
USB4:   USB EHCI 1.00
USB5:   USB OHCI 1.0
scanning bus 0 for devices... 4 USB Device(s) found
scanning bus 2 for devices... 1 USB Device(s) found
scanning bus 4 for devices... 1 USB Device(s) found

ThomasKaiser commented 8 years ago

How to proceed with this? IMO should we come up with a solution prior to release of 5.10?

igorpecovnik commented 8 years ago

I vote for manual selection ... since it bring much less problems. Currently working on repacking board .deb packages so those things might be changed.

zador-blood-stained commented 8 years ago

For me the most reliable approach is "one board - one set of images" and no first boot config changes based on autodetect. Since this means increased space requirements for images hosting, it may be necessary to drop some image targets based on download statistics. Alternatively once all desktop related stuff is packed into packages it may be possible to drop prebuild desktop images (at least for some boards) and provide a script to convert CLI images into desktop ones.

zador-blood-stained commented 8 years ago

I vote for manual selection

At build time (multiple images) or at first boot by user (single image)?

ThomasKaiser commented 8 years ago

IMO it would be still worth a look to do one single build per ${LINUXFAMILY} and then customize everything board specific in a last step before closing the image (making use of btrfs snapshots and so on). But that needs a huge amount of time for development and testing (and also adoption of zador's template system to work efficiently).

Anyway: auto detection can't work reliably since some boards available now are more or less 99% clones of others so we need one image per board (or users that are able to walk through the OSM settings of their display to disable overscanning and stuff like that).

Regarding hosting requirements why not rethinking the torrent approach?

Regarding count of OS images (and differentiation between CLI and desktop images): It would be great if CLI images could be upgraded to desktop images simply by installing packages. But in fact I'm already thinking about another image variant per board: SD card tester (technically close to moronic but maybe a nice PR stunt -- ok, forget about it now)

zador-blood-stained commented 8 years ago

IMO it would be still worth a look to do one single build per ${LINUXFAMILY} and then customize everything board specific in a last step before closing the image (making use of btrfs snapshots and so on). But that needs a huge amount of time for development and testing (and also adoption of zador's template system to work efficiently).

With cached rootfs it won't save more than few minutes build time per board, so IMO it's not worth the efforts, at least now.

Regarding hosting requirements why not rethinking the torrent approach?

It will save bandwidth/traffic, but it won't help minimizing storage space requirements, especially considering xenial release -> new target for prebuilt images.

ThomasKaiser commented 8 years ago

With cached rootfs it won't save more than few minutes build time per board, so IMO it's not worth the efforts, at least now.

Agreed. And regarding hosting requirements: We did some ZFS trickery a few years ago which resulted in a storage overcommitment of +200 times the real storage needed (would apply to btrfs tricks with image creation in a batch and then using the build host also for serving torrents when using a more primitive compression method). But that's off-topic :)

Ok, question seems to be: Let the user decide when downloading the image or when powering up the board the 1st time? Based on feedback we got from our new user base I prefer the former.

igorpecovnik commented 8 years ago

With little more optimization and if we don't provide all possible combinations per board we can stick to dedicated images scenario, otherwise selecting a proper board at first login. Hardware detection will always be complicated so let's forget about this. With proper desktop packing, we could skip desktop creation too.

Alberto1398 commented 8 years ago

What about using 'fw_printenv fdtfile' to get the name of the dtb used by u-boot?

igorpecovnik commented 8 years ago

We were trying to narrow down the number of images with auto selecting proper board config at first boot ... so dtb is fixed and known. Unfortunately it's not reliable method so only alternative left is to ask user - choose your board at first login ...

zador-blood-stained commented 8 years ago

What about using 'fw_printenv fdtfile' to get the name of the dtb used by u-boot?

fdtfile is defined by u-boot board configuration at compile-time, so it is useless for any kind of autodetect

ThomasKaiser commented 8 years ago

To continue discussion from forum thread: I second @zador-blood-stained 's suggestion to strip down armhwinfo from the auto detection stuff and use one file eg. /etc/armbian-boards.conf to be the unique source for hardware configuration.

What about doing it this way:

First line contains simply $BOARD using the naming convention we now use for /run/machine.id ("Orange Pi One", "Beelink X2", "Banana Pi" and so on).
Since also the IRQ distribution is now based on wild guesses and weird code in armhwinfo we define for every board in the specific board templates the name of the modules used to be assigned to different CPU cores (we might have to differentiate between vanilla and legacy since names might vary. So for A20 devices we end with an array/list containing one entry per kernel, on H3 boards we have 3 entries and on ODROID XU4 for example up to 7 entries
These entries are then used to fill in the following lines of /etc/armbian-boards.conf: One line per board, containing board name and the array of modules for IRQ assignment.
In armhwinfo only /etc/armbian-boards.conf is consulted. First line determines the board hardware and then a lookup in the following table is used for IRQ assignments
(optionally the same could be done with LEDs to provide user feedback in firstrun or powering down situations -- but since this might require huge efforts to test better postpone it to get rid auf auto detection soon)

Any thoughts?

zador-blood-stained commented 8 years ago

First line contains simply $BOARD using the naming convention we now use for /run/machine.id ("Orange Pi One", "Beelink X2", "Banana Pi" and so on).

This requires mapping $BOARD to /run/machine.id format at compilation time and then mapping machine.id to sets of tweaks at armhwinfo execution time, so why not use $BOARD (and $LINUXFAMILY) in the first place? I'm assuming that we will split H3 and bananapi/pro rather sooner than later.

Board configuration files can define custom function which can be easily included in firstrun script (like it's currently done with write_uboot_platform here, so armhwinfo can include tweaks only for one LINUXFAMILY

ThomasKaiser commented 8 years ago

I'm open for any suggestion (especially since I won't help coding this stuff since build hosts still unavailable and test boards too except of Pine64) so IMO we should implement it as you like to be able to switch away from auto detection soon.

I just wanted to ensure that we don't forget IRQ distribution since this is one of the superiour Armbian features compared to other distros out there :)

zador-blood-stained commented 8 years ago

Then my suggestion would be:

Use /etc/default/armhwinfo to define variables like BOARD (for internal logic), ID (user-friendly name for logs and MOTD), VERSION (for Armbian release version);
source /etc/default/armhwinfo in /etc/init.d/armhwinfo and /etc/init.d/firstrun to access those variables
/etc/init.d/armhwinfo executes generic tweaks and LINUXFAMILY specific tweaks
LINUXFAMILY specific tweaks are stored in lib/config/sources and are included into /etc/init.d/armhwinfo at build time or are stored in separate .sh file in /etc
No autodetection of any kind. One board = one image with exception of fully compatible boards

ThomasKaiser commented 8 years ago

Perfectly fine with me!

BTW: Now I remember why I wanted all the boards configured in a single file: To ease switching with one installation between different boards (eg. between OPi One and PC). But that doesn't matter that much and the whole scenario was present 2 months ago since Orange Pi users had two boards and started to test with Armbian on both of them and ran into troubles. Nothing we now have to take care of any longer. The remaining issues if one wants to switch with a single installation between different boards are stuff that has to be covered by improving documentation.

zador-blood-stained commented 8 years ago

Now it's a question whether this

One board = one image with exception of fully compatible boards

is fine with Igor and his hosting storage space...

svkatielee commented 8 years ago

If you choose to go with a file of board descriptions, I suggest considerable thought. I watched the Arduino folks go through several iterations of boards.txt format and now that the IDE is popular with several other platforms like the ESP8266 families they have gone to a mixed environment with a load time JSON file (XML'ish) describing all the boards for a platform family.

It is a little different since the users select the board before compiling in the IDE, but My point is the platforms grew and they re-wrote the board config code several times.

zador-blood-stained commented 8 years ago

Arduino file formats affect 3rd party libraries and additional hardware cores - yes, there are some problems with backwards compatibility. Here, though, changes are fully internal and mostly do not affect end users, except manual upgrade will be required to switch from more generic images (i.e. orangepih3) to more specific ones (i.e. orangepipc).

ThomasKaiser commented 8 years ago

is fine with Igor and his hosting storage space...

Regarding the latter: @igorpecovnik please keep in mind that lvrp16 offered hosting/mirroring (and would be glad to do so if I understood him correctly since his customers complained about download speed from lauri's NAS). And now since he also sends his ODROID customers to get Armbian...

igorpecovnik commented 8 years ago

It's o.k. regarding space usage @hosting and yes we need bandwidth upgrade, here in EU and also in Asia. I was offered from more sources and perhaps now we got a real need to expand. I am also in the middle of forum server migration to XU4, since it's getting slow. Traffic is going up all the time. I guess it must be powered by ARMbian ;)

zador-blood-stained commented 8 years ago

@igorpecovnik

Is toradex / colibri i.MX6 still supported or should it be disabled? It's missing some critical properties in board and sources config files.
Are Cubox i2eX and i4 fully compatible regarding u-boot and kernel configuration?

igorpecovnik commented 8 years ago

Toradex, don't have any confirmation that it works, disable
AFAIK yes, unless they didn't broke something (uboot autodetection) while adding new boards. I haven't done any tests with Cubox for a while since I am using it as a media center :) HB2 4x4 is working out of the box.

ThomasKaiser commented 8 years ago

BTW: Somewhat related to dropping auto configuration: How to deal with onboard leds? I started to add led user feedback at booting and firstrun stage for sun8i, it got extended to sun7i more by accident and currently it seems we have a firstrun showstopper on all sun7i boards that do not feature a red led.

I'm not really concerned about consistent behaviour accross all boards we support (eg. all start with green led that starts to blink for 10 seconds when login is possible) since the kind of users that deal with more than one or two boards together with Armbian should be experienced enough to deal with different behaviour.

I'm fine with removing the blinking behaviour from firstrun at all since this was a concession to our 'new' H3 users back then. But since as soon as we add Pine64 to the list of supported boards even more inexperienced users will start to use Armbian on a board that allows zero user feedback... it doesn't matter that much.

And to provide a consistent user experience on H3 boards it would be necessary to activate the green led directly after powering up and not like it's now only after hardware initialisation has been done in u-boot. Since H3 u-boot allows scanning for USB devices the delay between powering on the board and the led starting to light is already too long. And it seems with 5.10 another issue this time caused by armhwinfo is present: Blinking not just for 10 seconds but endlessly. Any thoughts (still no H3 devices available to test with)?

zador-blood-stained commented 8 years ago

BTW: Somewhat related to dropping auto configuration: How to deal with onboard leds?

For example we could define LED names in board configuration files, and new variable would go into /etc/default/armbian (in board support package) to be used by firstrun and armhwinfo scripts.

ThomasKaiser commented 8 years ago

I also thought about this but feared a bit that we need info about each and every board we support to do it right. Or we simply define the special word 'none' to skip led stuff for boards that aren't confirmed to work now. Another problem are different sysfs paths for legacy and vanilla kernel and that we need already some special u-boot code to deal really correctly with the leds (so that the led we want to use as 'power led' immediately starts to light after providing power to the board and not with +5 secs delay).

zador-blood-stained commented 8 years ago

Or we simply define the special word 'none' to skip led stuff for boards that aren't confirmed to work now.

Here it's enough to skip LED activation if variable is empty or not defined.

Another problem are different sysfs paths for legacy and vanilla kernel

Then it may be easier to include LED specific code into sources family configuration files, same way I'm planning to do interrupts distribution and similar tweaks.

zador-blood-stained commented 8 years ago

Well, set -e here may be the reason why missing LED kills firstrun script before FS resizing and may cause a lot of "fun" issues.

ThomasKaiser commented 8 years ago

Right, ommited definitions serve better than 'none' :)

Then it may be easier to include LED specific code into sources family configuration files

Unfortunately this won't work (fully) since device tree led definitions are based on the fantasy of the .dts author (your only hope is wildcard matching like /sys/foo/bar/*green*) and since board vendors choose different led colors for the same function (blue vs. red when comparing Orange Pi One with its clone NanoPi M1 for example) we would need this stuff defined on a per board basis or start with 'per board' alias definitions.

Maybe it's the best to try to fix things for sun8i now, disable led blinking in firstrun for both sun7i/sun8i and leave everything else as it's now.

IMO the whole 'provide user feedback' approach should be dependant on the target audience. H3 users need(ed) some support, users of Clearfog Pro are able to read the docs and have the necessary patience to wait for one automated reboot. And with the absolutely clueless kickstarter crowd that might appear here when we start supporting Pine64 we're already lost due to the design flaws of the board :)

zador-blood-stained commented 8 years ago

Unfortunately this won't work (fully) since device tree led definitions are based on the fantasy of the .dts author (your only hope is wildcard matching like /sys/foo/bar/green)

This will work - we define a function for board family, and inside that function we deal with per-board and per-kernel LEDs configurations based on BOARD variable from /etc/default/armbian.

ThomasKaiser commented 8 years ago

Well, set -e here may be the reason why missing LED kills firstrun script before FS resizing and may cause a lot of "fun" issues.

Good catch. What about simply removing this line for 5.11 release?

we define a function for board family, and inside that function we deal with per-board and per-kernel LEDs configurations based on BOARD variable from /etc/default/armbian

Ah, now understood. Ok, then we should start with sun8i (currently the most inexperienced users we have) and expand support to more board families later. But still the need to tweak something in u-boot to get 'perfect' user feedback (immediately switching on power led for example)

zador-blood-stained commented 8 years ago

Right now I'm waiting for

Armbian 5.11 release - to have more time since changes affect board support package creation code, MOTD code, firstrun and armhwinfo scripts and documentation.
U-boot 2016.05 release (should've happened a week ago)

Edit:

Good catch. What about simply removing this line for 5.11 release?

should work, I don't see why it was needed in the first place.

zador-blood-stained commented 8 years ago

But still the need to tweak something in u-boot to get 'perfect' user feedback (immediately switching on power led for example)

Please test u-boot with this patch - it turns on any LED connected to PL10 on H3 boards.

Disabling USB support in u-boot will prevent users from using USB keyboards in u-boot, so for now feedback by turning on LED should be good enough for improving user experience.

ThomasKaiser commented 8 years ago

Disabling USB support in u-boot will prevent users from using USB keyboards in u-boot

Just curious: what should a H3 board user do with his keyboard in u-boot stage (except of breaking autoboot by accident)? Since we're not providing Pine64 images now our H3 users can be considered the most inexperienced we have while the few experienced know how to use serial console.

I would really love to here of a good example for this stuff but otherwise would fear that it only delays boot for no reason. BTW: People are even creative enough to revert back to Allwinner's BSP u-boot just because of that. Please see the 2nd Alpine Linux variant promoted here

zador-blood-stained commented 8 years ago

  | Symbol: USB [=y]                                                        |
  | Type  : boolean                                                         |
  | Prompt: USB support                                                     |
  |   Location:                                                             |
  | (1) -> Device Drivers                                                   |
  |   Defined at drivers/usb/Kconfig:1                                      |
  |   Selected by: ARCH_SUNXI [=y] && <choice>

Right now USB support is forced for all sunxi boards. So either we disable USB for all sunxi boards (quick&dirty) or only for H3 (slightly more complicated)

ThomasKaiser commented 8 years ago

Hmm... I just commented CONFIG_USB_EHCI_HCD=y out (setting to n might be smarter) in defconfig and retried it:

 U-Boot SPL 2016.05-armbian (Jun 05 2016 - 18:29:33)
 DRAM: 1024 MiB
 Card did not respond to voltage select!
 Trying to boot from MMC2

 U-Boot 2016.05-armbian (Jun 05 2016 - 18:29:33 +0200) Allwinner Technology

 CPU:   Allwinner H3 (SUN8I 1680)
 Model: Xunlong Orange Pi PC
 DRAM:  1 GiB
 MMC:   Card did not respond to voltage select!
 SUNXI SD/MMC: 1, SUNXI SD/MMC: 0 (eMMC)
 *** Warning - bad CRC, using default environment

 In:    serial
 Out:   serial
 Err:   serial
 Net:   No ethernet found.
 starting USB...
 No controllers found
 Hit any key to stop autoboot:  0

Seems to work -- at least no USB busses are scanned for peripherals. Regarding your patch I let create a new OS image right now since the image I'm using now was for DRAM testing on BPi M2+ and I fear that I messed up too much in the meantime maybe (since led behaviour is not as expected on OPi PC Plus -- green led doesn't light at boot, red led starts to light as soon as kernel loads and when the board will be shutdown green led stays on).

zador-blood-stained commented 8 years ago

Hmm... I just commented CONFIG_USB_EHCI_HCD=y out (setting to n might be smarter) in defconfig and retried it

This is good enough, but we'll have to update documentation parts that refer to using USB keyboard in u-boot (i.e. unbricking procedure).

(since led behaviour is not as expected on OPi PC Plus -- green led doesn't light at boot, red led starts to light as soon as kernel loads and when the board will be shutdown green led stays on).

During my THS tests both LEDs were set to heartbeat, so we need to re-check firstrun and armhwinfo scripts too.

ThomasKaiser commented 8 years ago

Ok, I would prefer updating documentation and disabling USB through the defconfig to save the few seconds in u-boot.

Then regarding led behaviour:

your patch works fine, green led almost immediately starts to light
when the kernel is loaded at approx sec 1 green led will be set to heartbeat trigger and red led to default-on (really no idea why and I would believe this is the result of exchanging our legacy kernel sources)
further behaviour almost as expected. When firstrun is executed red led gets flashed due to assigning heartbeat trigger. The green led will be set to heartbeat trigger from within armhwinfo (pretty useless at the moment since it's already set when kernel started) and then switched to default-on (after 120 seconds when 1st login happens, at further logins after only 10 seconds later).

So open questions are: Where to define led behaviour (must be a kernel feature) and how? If we don't get the source of new default led behaviour at kernel start most simple solution would be to also set red led to 'none' from within armhwinfo (only exception: BPi M2+ where it should stay default-on since it's the only available led). Then we would get the green led as indicator that the board boots, when green led starts flashing and red lights solid the user knows that the kernel starts and when red led goes off and green changes to solid he knows it's time to login (system completely booted).

zador-blood-stained commented 8 years ago

when the kernel is loaded at approx sec 1 green led will be set to heartbeat trigger and red led to default-on (really no idea why and I would believe this is the result of exchanging our legacy kernel sources)

https://github.com/igorpecovnik/linux/blob/sun8i/drivers/leds/leds-sunxi.c#L17-L34

Patching this should work.

ThomasKaiser commented 8 years ago

Yes, you're right. Setting all three to none as it was before should be enough to let fex definitions take over. Will try that out later with an Orange Pi and BPi M2+

ThomasKaiser commented 8 years ago

Yep, setting all three to none we get back to former (and desired) behaviour: LEDs behave like defined in fex file. At the moment I'm getting mad failing to comment out +# CONFIG_USB_EHCI_HCD is not set

When I boot OPi PC Plus USB buses are scanned. Stuff for next week I guess :)

ThomasKaiser commented 8 years ago

Update: # CONFIG_USB_EHCI_HCD is not set works as expected on Banana Pi M2+ but not on Orange Pi PC Plus. There it looks like this

 starting USB...
 USB0:   USB EHCI 1.00
 USB1:   USB OHCI 1.0
 USB2:   USB EHCI 1.00
 USB3:   USB OHCI 1.0
 USB4:   USB EHCI 1.00
 USB5:   USB OHCI 1.0
 scanning bus 0 for devices... 1 USB Device(s) found
 scanning bus 2 for devices... 1 USB Device(s) found
 scanning bus 4 for devices... 1 USB Device(s) found

ThomasKaiser commented 8 years ago

Since with release of 5.15 this issue is resolved closing now.

armbian / build

Board auto detection broken by design #231