ODroid XU4 won't boot

[nalipaz]

Home Assistant release with the issue: 2.10

Description of problem: I cannot get my xu4 to boot after flashing the emmc with hass.io. It starts to boot, I hear the fan whir, then it restarts, it does this repeatedly. I have had no issue with this device in the past and have confirmed files on the emmc look good. I flashed using the installation instructions. Any help or guidance on troubleshooting is appreciated.

[mad-ady]

I tested it on my XU4 with orange eMMC as well. Console output is:

MMC Device 0 (eMMC): 29.1 GiB
Info eMMC rst_n_func status = enabled
Card did not respond to voltage select!
mmc_init: -95, time 11
*** Warning - bad CRC, using default environment

Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
** Invalid partition 2 **

There's something wrong with the bootloader for eMMC. If needed I can do further tests. The Odroid C2 image boots fine from eMMC on the C2 (but it's a different SOC).

[nalipaz]

I ended up using a rpi model 2 b+ so won't be continuing to try this on the xu4. I will likely use that board for something else. Before I utilize it for another project if any testing is needed let me know. I should say I went back a version on hassio and the same issue occurred.

[pvizeli]

I don't have any eMMC. I use expensive SD cards.

Make an issue on HassOS repository. Maybe someone with eMMC can fix that.

[mad-ady]

I'd like to help - I have the necessary hardware, but I need to be pointed at the build guide - where the image is created, to see what's up.

[rbray89]

@nalipaz @mad-ady : Read the documentation: https://github.com/home-assistant/hassos/blob/dev/Documentation/boards/odroid-xu4.md#emmc

u-boot exists in a special hidden sector on the XU4 eMMC. It can only be accessed on the XU4 boards, you won't be able to write to it via SD adapters. Follow the instructions outlined in the documentation and you should be able to get it working. An additional note, if you get permissions issues when running the dd command, use echo 0 > /sys/block/mmcblk0boot0/force_ro to mark the sector as writable.

[escoand]

Armbian provides images which are booting up correctly: https://github.com/armbian/build/blob/master/config/bootscripts/boot-odroid-xu4-default.ini Don't know if this will help.

[sander76]

@rbray89 working with the link you provided above. Unfortunately no success yet. Is the following correct?

dd if=/dev/mmcblk0 of=/dev/mmcblk0boot0 bs=512 skip=63 seek=62 count=1440

especially the if=/dev/mmcblk0 part ? Shouldn't that be 1 instead of 0 ?

[rbray89]

@sander76 What is your setup? (eg. do you have an SD card? what is on it? what's on the eMMC?)

[sander76]

@rbray89 sdcard is mmcblk1. eMMC is mmcblk0

I mean, mmcblk0 and mmcblk0boot0 are physically the same devices ? Shouldn't I be copying from sd to eMMC ?

A bit of history: I first did this (sounded easiest to me):

Alternatively, you can try flash HassOS to both an SD and eMMC, then boot off the SD with the eMMC also plugged in, then run dd if=/dev/mmcblk1 of=/dev/mmcblk0boot0 bs=512 skip=1 seek=0 count=16381 at the Linux prompt. Note that this is untested, but in theory should work..

Unfortunately that didn't work. The device frooze during bootup.

Now I am doing your first suggestion (with hassos on an SD card first). But I am confused about the mmcblk0....

Thanks for the quick reply !

[sander76]

hmmm... I am now in hassos console. going to dev folder and can only find mmcblk1 devices. No mmcblk0 devices.

[rbray89]

@sander76 /dev/mmcblk0boot0 is a "hidden" boot sector on the eMMC device. We copy from mmcblk0, as when we use etcher to write the eMMC image, it never touches the "hidden" boot sector, so we need to copy what we DID write to the boot sector.

Are you flipping the DIP switch to select SD boot with the eMMC device attached?

[mwbehr1]

@sander76 What worked for me:

1. Flash both SD and EMMC with HASSOS 3.1
2. Boot to SD with EMMC also plugged in
3. /sys/block/mmcblk0boot0/force_ro
4. dd if=/dev/mmcblk0 of=/dev/mmcblk0boot0 bs=512 skip=63 seek=62 count=1440
5. Reboot to EMMC

Without first flashing the EMMC, mmcblk0 wouldn't be pointing to a HASSOS boot sector.

[rbray89]

@mwbehr1 Thanks for the succinct instructions. This should be exactly what is needed.

[sander76]

@mwbehr1 Thanks for the description. Going to give it another try.

I am a bit confused about the version indication. Going here: https://www.home-assistant.io/hassio/installation/ it forwards me to version 2.12

https://github.com/home-assistant/hassos/releases/tag/3.1 shows the correct version.

I am going to use the latter.

[sander76]

@sander76 /dev/mmcblk0boot0 is a "hidden" boot sector on the eMMC device. We copy from mmcblk0, as when we use etcher to write the eMMC image, it never touches the "hidden" boot sector, so we need to copy what we DID write to the boot sector.

Are you flipping the DIP switch to select SD boot with the eMMC device attached?

Ok. I guess I understand. But then, what's the point of having to boot the SD card with HassOS on it ? It could be any linux os ?

[sander76]

@mwbehr1 Can't get it to work. It boots up, but gets stuck with the message:

mmc0: tuning execution failed: -5
mmc0: error -5 whilst initializing MMC card

Do you have any idea what to do now ?

[rbray89]

@sander76 One small edit, note the changed "/sys/block/mmcblk0boot0/force_ro" line. Flash both SD and EMMC with HASSOS 3.1 Boot to SD with EMMC also plugged in echo 0 > /sys/block/mmcblk0boot0/force_ro dd if=/dev/mmcblk0 of=/dev/mmcblk0boot0 bs=512 skip=63 seek=62 count=1440 Reboot to EMMC

Are you booting from the SD card? you have to flip a DIP switch to do so. Also make sure that the eMMC module is fully seated in its socket and no pins are damaged.

[sander76]

@rbray89

Thanks for the reply. Ok. Here is what I've done with my odroid xu4 (among many other things. without success)

1. To recover the boot sector I followed the hardkernel procedure. With success. Odroid boots to android using MMC. https://forum.odroid.com/viewtopic.php?f=53&t=6173
2. flashed both SD card and MMC card with hassos.
3. Booting to SD card. With success. One clue for failure that the log already says:

   mmc0: tuning execution failed: -
   mmc0: error -5 whilst initializing MMC card

4. There is no mmcblk0 device in the dev folder. Somehow MMC is not discovered. I am sure in my previous attempts it was there. But now it just doesn't. Also after several attempts/reboots/flashes etc.
5. I flash an ubuntu image to the sd card. And boot from that. With success.
6. mmcblk0 device is also present.
7. I execute the following commands.

   echo 0 > /sys/block/mmcblk0boot0/force_ro
   dd if=/dev/mmcblk0 of=/dev/mmcblk0boot0 bs=512 skip=63 seek=62 count=1440
   echo 1 > /sys/block/mmcblk0boot0/force_ro

   No failures reported.

8. Shutdown odroid, power off. flip boot switch, remove sd card and power on.
9. Odroid boots. blue led starts solid and follows with blinking (activity behaviour)
10. I can see al instructions passing by on my connected monitor. So all seems to be going fine until the following appears:

    mmc0: tuning execution failed: -5
    mmc0: error -5 whilst initializing MMC card

11. Boot process freezes. No progress.

I have done some googling on the error message and I found it could be either the kernel requiring a patch/update or incorrect boot image. I just don't know which one.

I just did an installation of Armbian (https://www.armbian.com/). After following the instructions it is booting on the EMMC and running Hassio without problems. Glad to see my MMC and Odroid can at least run A instance of hass.

But I do prefer to run Hassos. I have a feeling there is something wrong with the current release of hassos (3.1 AND 2 actually).

[rbray89]

Ah, thanks for the perfect summary of what you've tried so far.

I agree, I think there is a patch needed for your SPECIFIC eMMC. From what I've read, hardkernel released a couple eMMC versions, one of which was much more sensitive to higher clock speeds and unable to be used at the faster rates. I'll have to look into it further. I thought I included those patches in the build but I must have been mistaken if you're seeing this.

[sander76]

Thanks. Let me know if you need any testing on my side. I don't have my odroid operational. Yet ;-)

[sander76]

@rbray89 Are you making any progress ? I understand it's all in your free time, but I'd like to get started with my new hass installation. If not, I'll go for the armbian road...

[rbray89]

Maybe? I'm looking at some of the device tree patches that Hardkernel have applied to their 4.14 kernel branch. I think I found a potential fix, but I have no way to verify the change as I don't have the eMMC that triggers the issue.

For documentation's sake, Hardkernel's patch:https://github.com/hardkernel/linux/commit/257a634ae3fd9fb0a261071315d9027d7fd5d10e#diff-7475e0a481aa08535923f67344cbae8e location in linux/master: https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/exynos5422-odroidxu3-common.dtsi#L405

[sander76]

Is it possible to give me some instructions how to apply these? I'm happy to try it out.

[rbray89]

I'll try to get an image built tomorrow that you can try out.

[sander76]

Thanks!

[rbray89]

@sander76 https://github.com/rbray89/hassos/releases/tag/3.3-xu4-emmc-strobe-fix

[rbray89]

@sander76 did you get a chance to try the new image out?

[sander76]

@rbray89 sorry. Not yet. Planning to do it this weekend.

[sander76]

@rbray89 Same error again. 😞

I also see this error:

s5p-mfc 11000000.codec: Direct firmware load for s5p-mfc-v8.fw failed with error -2
s5p_mfc_load_firmware:73: Firmware is not present in the /lib/firmware directory nor compiled in kernel

[rbray89]

@sander76 Hmm... I found a 5.0 kernel thread on HK's forum that describes a user having the same issue: https://forum.odroid.com/viewtopic.php?f=95&t=33510&start=100#p250303 Though in their case it seemed to be an incomplete flashing.

The two possibilities I can think of are: a) It is possible that we're missing something in our DTB file (device tree blob). I've poured over the dtbs though in both the mainline kernel and the one linked in the thread above and can't find a change that would address this. b) The flashing procedure isn't working for some reason, leaving "bad" dtbs or such in the way.

If you do ls -la /mnt/boot what do you see?

[rbray89]

So on second thought, I just ran an update on my XU4 and things did not go well... Wondering if a change was made to the repo that messed up eMMC booting.

[sander76]

@rbray89 Ah! Now your hass is broken too! That sucks

[sander76]

@rbray89 So I did a bootup. And this time it found the mmc0 devices. Just to see whether it was persistent it did a reboot again: mmc0 device gone. And not able to make it appear again after several reboots.

I do see the following message appear. I wonder if it is of any help:

dwmmc_exynos 12220000.mmc: Unexpected interrupt latency

When I do a ls -la /mnt/boot I get the following files:

System Volume Information
boot.scr
cmdline.txt
config.txt
exynos5422-odroidxu4.dtb

[sander76]

@rbray89 Should I go for Armbian? Or do you have plans to attack this issue once more ?

[rbray89]

I'm planning on working on it tonight.

[rbray89]

@sander76 So I can't replicate the issue. I was hoping my upgrade attempt was failing and reverting to the default, but now that I look back on it, I think I may have just been impatient. Using this as a guide https://wiki.odroid.com/accessory/emmc/reference_chart which eMMC module are you using?

[sander76]

@rbray89 I have the orange eMMC. rev 0.4

[rbray89]

@sander76 That's what I'm running with too. I'm thinking this may still just be an install issue. From what you have posted though, it looks like you're doing all the correct steps.

Would you mind posting the full log on something like pastebin?

[sander76]

@rbray89 See below:

https://pastebin.com/2JM7e0M0

Some of the errors I found:

Dec 21 18:53:34 hassio kernel: s5p_mfc_load_firmware:73: Firmware is not present in the /lib/firmware directory nor compiled in kernel
Dec 21 18:53:34 hassio kernel: mmc0: tuning execution failed: -5
Dec 21 18:53:34 hassio kernel: mmc0: error -5 whilst initialising MMC card
Dec 21 18:53:34 hassio kernel: dwc3 12000000.dwc3: Failed to get clk 'ref': -2

Thanks for putting in so much effort !

[rbray89]

@sander76: are you sure it is fully seated and the connector is fully engaged? I recall having a similar issue (eMMC instability) when I didn't get the eMMC module fully seated. I can't seem to trigger the issues you have with any image. (2.12, 3.2, or 3.3 I built)

[rbray89]

Just uploaded a fix for the s5p_mfc fw not being found. Shouldn't impact your problem though. https://github.com/home-assistant/hassos/pull/425 https://github.com/rbray89/hassos/releases/tag/3.3-sp5c

[sander76]

@rbray89 I'll give it another try and pay special attention to the seating. But I doubt that's the case. Other images (original, armbian) have all just worked properly.

[rbray89]

Oh, one other thing you can try, would be to interrupt the boot process and run your "own" boot command. This would let us boot uboot from the SD card and linux off the eMMC. This would bypass the boot sector of the eMMC to check to see if it is a boot sector issue. Are you running connected to the serial port? Or have you just been using ssh/hdmi?

Added a boot.scr to the above release if you want to try that out.

[rbray89]

Acutally, I don't think you'd have to muck with a boot.scr file. While booting with the SD card and eMMC car both flashed and plugged in, you could run (on host) fw_setenv device 0 Then continue "booting" from the SD card. uboot will still live on the SD card, but it should load everything else off the eMMC. Only downside is that this isn't a long-term solution as I think this has the potential to mess up RAUC (uboot environment vars won't be able to be read in host linux ). But this would tell us if the boot on eMMC needs work (this hack works) or if there is something else at play (eMMC frequency selection? Why don't I see it with the same chip?).

[sander76]

@rbray89 Currently I am running with everything (keyboard/monitor) connected to the device itself. I'd love to go ssh as it removes a lot of practical hassle, but haven't bothered trying as I thought the hassos image runs with ssh disabled by default. Or doesn't it ? For the serial port I need a special cable don't I ?

I will try your suggestion this weekend...

[rbray89]

Yeah. Also, in digging around,I may have found something that could be contributing to errors, but have yet to fix it. Will update if it turns out like I suspect.

[rbray89]

So I discovered why my upgrade was failing, we weren't storing/importing the uboot environment properly. That should be fixed in my pull request and in the latest image here: https://github.com/rbray89/hassos/releases/tag/3.3-odroid-xu4-env-fix Though I don't know if it will fix your issue @sander76

[sander76]

@rbray89 No luck. mmc0 is just not recognized. Somewhere the Hassos image is failing where the Armbian image is not.

[sander76]

@rbray89 unless you really want to solve this and need me to try things out I'm going for the Armbian installation. Please let me know...

[rbray89]

@sander76 I'd love to solve this, but I think remote debugging this just isn't going to realistically get this soved. I ordered a larger eMMC for my XU4, maybe (hopefully? ) I'll be able to replicate the issue on it.