Odroid-N2+ doesn't boot from eMMC (11.5+, other than Kingston eMMC)

scoobydrvr commented 4 months ago

Describe the issue you are experiencing

Clean install by writing image to eMMC card. Device does not boot to HA OS with version 11.5 or 12.0. If 11.4 is flashed, device will boot and behave as expected. Updating to 11.5 causes device to fail to boot. Further information - https://github.com/home-assistant/operating-system/discussions/3226

What operating system image do you use?

odroid-n2 (Hardkernel ODROID-N2/N2+)

What version of Home Assistant Operating System is installed?

11.4

Did you upgrade the Operating System.

Yes

Steps to reproduce the issue

Flash 11.5 or 12.0 image to eMMC and attempt to boot OR
Flash 11.4 image to eMMC and allow HA OS to update to 11.5

Anything in the Supervisor logs that might be useful for us?

Unable to access after update

Anything in the Host logs that might be useful for us?

Unable to access after update

System information

No response

Additional information

Noticed that there is a kernel change between 11.4 and 11.5. Could this be the root cause?

agners commented 4 months ago

With HAOS 11.5 we updated the U-Boot bootloader to 2024.01, so this is probably bootloader related.

FWIW, my ODROID-N2 test device here boots fine with HAOS 11.5 and 12.0, so this must be related to your particular unit.

Do you have access to the serial console (e.g. using the USB-UART adapter from Hardkernel) to observe the output at boot time?

scoobydrvr commented 4 months ago

Unfortunately, no. I feel like I was lucky to have bought the eMMC writer just in case. For what it's worth, this is a brand new unit so it sounds like Hardkernel has changed something in their file loads at the factory. Would there be anything I could help provide by flashing back to Ubuntu and stopping during Petitboot or files I could share?

Since it took about a week and a half to get the delivery from Korea, would something like this work, provided the jumpers can be made to fit inside the header (and the appropriate TTL level is selected)?

scoobydrvr commented 3 months ago

@agners I purchased a USB serial module and got the following the terminal

U-Boot 2024.01 (Mar 13 2024 - 12:36:37 +0000) odroid-n2/n2-plus

Model: Hardkernel ODROID-N2
SoC:   Amlogic Meson G12B (S922X) Revision 29:c (40:2)
DRAM:  1 GiB (effective 3.8 GiB)
Core:  400 devices, 30 uclasses, devicetree: separate
MMC:   sd@ffe05000: 0, mmc@ffe07000: 1
Loading Environment from nowhere... OK
In:    usbkbd,serial
Out:   vidconsole,serial
Err:   vidconsole,serial
Board variant: n2-plus
Net:   eth0: ethernet@ff3f0000
Card did not respond to voltage select! : -110
unable to select a mode : -5
MMC Device 2 not found
no mmc device at slot 2
starting USB...
Bus usb@ff500000: Register 3000140 NbrPorts 3
Starting the controller
USB XHCI 1.10
scanning bus usb@ff500000 for devices... 3 USB Device(s) found
usb - USB sub-system

However, moving back to 11.4 the following is spat out

U-Boot 2023.01 (Jan 09 2024 - 11:00:06 +0000) odroid-n2/n2-plus

Model: Hardkernel ODROID-N2
SoC:   Amlogic Meson G12B (S922X) Revision 29:c (40:2)
DRAM:  1 GiB (effective 3.8 GiB)
Core:  387 devices, 27 uclasses, devicetree: separate
MMC:   sd@ffe05000: 0, mmc@ffe07000: 1
Loading Environment from nowhere... OK
In:    serial
Out:   serial
Err:   serial
Board variant: n2-plus
Net:   eth0: ethernet@ff3f0000
Card did not respond to voltage select! : -110
switch to partitions #0, OK
mmc1(part 0) is current device
Scanning mmc 1:1...
Found U-Boot script /boot.scr
3491 bytes read in 2 ms (1.7 MiB/s)
## Executing script at 08000000
switch to partitions #0, OK
mmc1(part 0) is current device
loading env...

scoobydrvr commented 3 months ago

Looks like there is a similar issue with 12.1 maybe

https://github.com/home-assistant/operating-system/issues/3250

piercy commented 3 months ago

@scoobydrvr I see you linked my post, I actually got a similar output as you today. Ive been trying all sorts to get it working but having no luck. going to try 11.4 now and see if it works.

Here's what I managed to get on output , looks very similar to what you are seeing.

edit: 11.4 doesnt work for me either, same error about it not finding the mmc. If it flash ubuntu it works no problem.

scoobydrvr commented 3 months ago

What size is your MMC? I remembered I have a 16Gb card from an old Odroid device. I just threw 12.1 on it an, low and behold, the thing boots up! I had hoped that society had moved past storage device limits but it seems to somehow have an effect here...

piercy commented 3 months ago

What size is your MMC? I remembered I have a 16Gb card from an old Odroid device. I just threw 12.1 on it an, low and behold, the thing boots up! I had hoped that society had moved past storage device limits but it seems to somehow have an effect here...

It's 32Gb, I just found another thread with a similar issue.

What I noticed in your logs was that when its successful its booting from MMC 1, but when it fails its booting from MMC 2. Mine always seems to try boot from MMC 2 (hence its always failing).

I don't know if there is a way to force this, but feels like its trying to boot the wrong device.

piercy commented 3 months ago

Not sure it matters but I also noticed my mmc card revision is 0.5, a lot of the pictures ive seen show 0.4. So wonder if the new revision is having issues or something.

I might have to install with SD card as right now I have no automation running, so my house is extra dumb.

scoobydrvr commented 3 months ago

Not sure it matters but I also noticed my mmc card revision is 0.5, a lot of the pictures ive seen show 0.4. So wonder if the new revision is having issues or something.

I might have to install with SD card as right now I have no automation running, so my house is extra dumb.

My "non-functional" 32Gb is also a rev 0.5 hmmm...

piercy commented 3 months ago

Not sure it matters but I also noticed my mmc card revision is 0.5, a lot of the pictures ive seen show 0.4. So wonder if the new revision is having issues or something. I might have to install with SD card as right now I have no automation running, so my house is extra dumb.

My "non-functional" 32Gb is also a rev 0.5 hmmm...

Yeah, it doesnt make too much sense, if i can boot ubuntu I don't really see how the card could be the issue. However, maybe some kind of incompatibility has occurred and is causing issues.

Its way above my knowledge level really, but as other people seem to be fine, it feels like I must have something different to them.

scoobydrvr commented 3 months ago

Maybe someone with more intimate knowledge can sort out how v0.4 and v0.5 differentiate and why v0.5 doesn't seem to be working?

https://wiki.odroid.com/accessory/emmc/reference_chart#orange_emmc_module

agners commented 3 months ago

I have at least one eMMC module with version v0.5 and it seems to work here, so I don't think it is the PCB revision.

What PCB colors is yours? it seems that PCB color gives some indication what eMMC chip is on the PCB. I do have orange PCBs here.

piercy commented 3 months ago

@agners Mine is an orange module

piercy commented 3 months ago

I should add, ive tried both flashing with the USB 3.0 reader board, and flashing direct via the cable plugged into the N2+, both flash successfully but wont boot.

scoobydrvr commented 3 months ago

I have at least one eMMC module with version v0.5 and it seems to work here, so I don't think it is the PCB revision.

What PCB colors is yours? it seems that PCB color gives some indication what eMMC chip is on the PCB. I do have orange PCBs here.

Both of mine, the v0.4 16Gb and v0.5 32Gb, are orange cards

peter-dolkens commented 3 months ago

I actually have an old model Home Assistant Blue which I based my recommendation to piercy on when he was selecting hardware.

I went through the fresh install process just days before piercy and had no issues

It's currently deployed 2 hours away, so I'll have to wait a few days before I can get any covers off to get photos / part numbers for comparison

BarBaar44 commented 3 months ago

Happy to see this one, as there's nothing wrong my my board and/or eMMC after all :)

https://github.com/home-assistant/operating-system/issues/3150#issuecomment-2031611953

I can confirm I was using an orange 32gb v0.5 eMMc

DavidRSullivan commented 2 months ago

Since it took about a week and a half to get the delivery from Korea, would something like this work, provided the jumpers can be made to fit inside the header (and the appropriate TTL level is selected)?

Did this work?

scoobydrvr commented 2 months ago

Since it took about a week and a half to get the delivery from Korea, would something like this work, provided the jumpers can be made to fit inside the header (and the appropriate TTL level is selected)?

Did this work?

Yes, thankfully. See output in the comment above. Only caveat is the power and ground cables had to have the plastic connectors removed so they could fit inside the serial header on the board. Should also wrap them in electrical tape or similar to prevent shorts. Additionally, the dongle is powered by the board so you'll need to be quick with PuTTY or whatever serial terminal you use to capture the boot messages from the beginning.

TibsD commented 2 months ago

Any solution for this? I am having the same problem wit my N2+

I had a corrupted install of HA OS on an 32Gb eMMC card and tried reflashing the HA OS to the eMMC but do not see a blue light during boot.

Is it possible that the partitions created by the flashing of HA create a problem? How can I "reset" the eMMC card so it is completely empty and has no partitions?

DavidRSullivan commented 2 months ago

My problem was the SD card, and after purchasing a New ODROID N2+, everything is fine. I am using a 256GB eMMC version .5 without any issue.

TibsD commented 2 months ago

Weird issue. Mine just booted up without any problems after leaving it disconnected from power for a full night. Connected it up this morning and it booted up right away.

piercy commented 2 months ago

Weird issue. Mine just booted up without any problems after leaving it disconnected from power for a full night. Connected it up this morning and it booted up right away.

I had mine boot once too, figured it had been resolved by an update (i'd tried a different versions). When i then shut it down to move it to another room, it didn't boot again.

It certainly is a weird issue, seems like its starting to happen more often now though. Hopefully more examples can help diagnose and resolve it .

piercy commented 2 months ago

My problem was the SD card, and after purchasing a New ODROID N2+, everything is fine. I am using a 256GB eMMC version .5 without any issue.

While i could potentially do this, the fact I can boot into ubuntu fine suggest thats it isn't a problem with my eMMC or the N2+. I am currently using SD card rather that eMMC for HAOS but hoping a solution comes along soon.

TibsD commented 2 months ago

Weird issue. Mine just booted up without any problems after leaving it disconnected from power for a full night. Connected it up this morning and it booted up right away.

I had mine boot once too, figured it had been resolved by an update (i'd tried a different versions). When i then shut it down to move it to another room, it didn't boot again.

It certainly is a weird issue, seems like its starting to happen more often now though. Hopefully more examples can help diagnose and resolve it .

And if you wait for a night or so and then try starting it up?

agners commented 2 months ago

I think with #3319 we might have a solution. I've created a development build with that change specifically, it can be downloaded from this GitHub Action run: https://github.com/home-assistant/operating-system/actions/runs/8741265657 (image named haos_odroid-n2-12.3.dev1713456781.img.xz)

Can someone affected by the problem test that build and report if it indeed fixes the problem for you?

piercy commented 2 months ago

@agners yes, will test it shortly 👍

agners commented 2 months ago

@piercy can you access/download the artifacts? I just realized, at least when logged out, it is not possible. Not sure if this is tied to my account.

piercy commented 2 months ago

@agners yes already downloaded it :)

agners commented 2 months ago

Probably obvious, but just to be sure: The artifacts are zipped again (.zip). Make sure to unzip first, to get to the tar.xz (which depending on the tool, might need another unxz, as usual).

piercy commented 2 months ago

@agners sorry took a bit longer than expected.. work got in the way.

Anyway, it booted up first time. Blue boot lights straight away and minute or so later I had the setup screen. Attempting to restore my backup now, and then will cycle it a few times to see how it does

sairon commented 2 weeks ago

I'm reopening this issue because now I got my hands on a Kingston eMMC and I realized we might have two different issues:

Kingston eMMC modules that never worked correctly (the card correctly initializes only one in many boots), until we introduced a patch in #3319. This is tracked in #3250.
However, that might have triggered #3351, where the faster frequency causes another regression, or actually increses its incidence. The root cause seems to be the update of U-Boot. As a workarond we have downgraded U-Boot to pre-11.5 state in #3409 but that's not a long-term solution, as it also breaks Kingston again. If true, it means that the above dev build, and the 12.3 release could never work correctly for @scoobydrvr and others that do not have a Kingston eMMC.

I'd like to ask everyone here to provide us with as many details about the modules used as possible (ideally with the photos of the eMMC module with the sticker removed). Eventually, if you're located in the EU, I could arrange shipping you a new eMMC module and sourcing the faulty one from you - please get in touch with me if you are willing to help with that.

piercy commented 2 weeks ago

@sairon See below. Since the update in april I have been working completely fine. Also, if you need to chat about it more, or need more information, im piercy (Car with pixar eyes for a profile photo) on the HA discord. Feel free to DM or tag me.

Kingston
2401076-001.A006-A  (it could be 00G-A at the end, hard to tell)
2338  4014187
EMMC32G-PJ30

4014187.09
Taiwan

sairon commented 2 weeks ago

@piercy Thanks, that's the same model as I have here. Unless I'm missing something very obvious, that only works with the 12.3 release (which we pulled back shortly after publishing it). Very likely upgrading to 12.4 release will soft-brick it. If you're seeing something different, let's keep it in #3250.

In any case, be warned about the next release, we currently had to treat Kingston eMMCs as unsupported as they're not that widespread and only a few of you managed to install HAOS on them.

roblamoreaux commented 2 weeks ago

I just saw this thread.

A couple weeks ago my N2+ stopped working after booting to the point is says "Loading Kernel..." on the hdmi screen, Is this the same symptom?

I flashed a new image on the spare eMMC and it did the same thing. I didn't have time to deal with it and wanted more RAM so I replaced it with a Raspberry pi 5 8GB. I have the backup files for it on my NAS and can try to pull the log files off it if it will help. I will try to flash the spare emmc with the latest image containing OS 12.4 to see if it fixes the issue for me. While I am enjoying the better [performance of the Pi 5 with NVME drive it would be nice to have the Odroid available as a backup I could get going quickly.

Rob

arjoh commented 2 weeks ago

I have noticed that updating to certain versions renders the eMMC unbootable as well. 12.5 does not boot for me.

Also:

Petitboot menu crashes with a "bad" flashed eMMC
Sometimes (but not all the time) a longer power cycle helps. So leave it of the power for about a minute it will boot again.
It seems to matter if I use balenaEtcher or passmark ImageUSB to flash. The latter seems to yield better results. But this could also be because incidentally the power cycle was longer.
A simple "reboot" never works. I have to disconnect the power and wait a bit.
Flashing an older image (os version 11.3) and restoring a full backup works. But I'm reluctant now to update OS, because this is my "production" unit.

Having tried so many things, it's hard to give a coherent summary of everything I've done and what works and what doesn't. Flashing an older image and restoring a full backup always works, so I'm certainly willing to try out a few things and report back if that helps fixing this issue. Just tell me what would be helpful.

smarthomefamilyverrips commented 2 weeks ago

@all Anyone successful did update to 12.4 on Odroid N2+? I am on 12.2 and honestly am a bit afraid to do the update to 12.4 😳

arjoh commented 2 weeks ago

@all Anyone successful did update to 12.4 on Odroid N2+? I am on 12.2 and honestly am a bit afraid to do the update to 12.4 😳

Would not boot for me.

smarthomefamilyverrips commented 2 weeks ago

@ALL Anyone successful did update to 12.4 on Odroid N2+? I am on 12.2 and honestly am a bit afraid to do the update to 12.4 😳

Would not boot for me.

@arjoh you where already on 12.2? Did see in your previous remark you flashed manually, I never done this I normally just use the install option in the UI when it shows me a update is avaliable.

piercy commented 2 weeks ago

@ALL Anyone successful did update to 12.4 on Odroid N2+? I am on 12.2 and honestly am a bit afraid to do the update to 12.4 😳

I haven't tried yet. Based on sairons comments my emmc isn't supported. Which sucks because I don't know how we're supposed to tell when we just order a emmc and get what we get. Without lifting the label , I didn't even know it was a Kingston one.

At the moment I'll be staying on 12.3 dev build , but it is quite worrying

It's also my first time with an odroid and haos. Hasn't been the best first impression but hopefully I'm just unlucky

arjoh commented 2 weeks ago

@ALL Anyone successful did update to 12.4 on Odroid N2+? I am on 12.2 and honestly am a bit afraid to do the update to 12.4 😳

Would not boot for me.

@arjoh you where already on 12.2?

I have been restoring a lot lately so I'm not 100% sure but I think I recall updating from 12.2 to 12.4 yes.

smarthomefamilyverrips commented 2 weeks ago

@ALL Anyone successful did update to 12.4 on Odroid N2+? I am on 12.2 and honestly am a bit afraid to do the update to 12.4 😳

Would not boot for me.

@arjoh you where already on 12.2?

I have been restoring a lot lately so I'm not 100% sure but I think I recall updating from 12.2 to 12.4 yes.

@arjoh you used the install function in UI when it showed you update was available? Or you flashing manually? I am on a HA Blue and never did do a OS update manually and would have no idea what to do if not boots up anymore, so that is why I am holding back to do update. 🫣

smarthomefamilyverrips commented 2 weeks ago

@ALL Anyone successful did update to 12.4 on Odroid N2+? I am on 12.2 and honestly am a bit afraid to do the update to 12.4 😳

I haven't tried yet. Based on sairons comments my emmc isn't supported. Which sucks because I don't know how we're supposed to tell when we just order a emmc and get what we get. Without lifting the label , I didn't even know it was a Kingston one.

At the moment I'll be staying on 12.3 dev build , but it is quite worrying

@piercy I also really have no idea what I should do, as stated in my other comment I normally just chose install in UI whenever I got prompted a update is available, 12.3 was already pulled again before I got the chance to push install, what feels like I was lucky, but now as said I not know if should push install on the 12.4 install, I really would not know what to do when not will boot as said I have the HA blue what already came pre-installed with HA so I never loaded/flashed myself anything. 🙄

smarthomefamilyverrips commented 2 weeks ago

Is it difficult to restore in case my HA blue will not boot anymore?

peter-dolkens commented 2 weeks ago

If you bought HA Blue, I believe you'll have older stock of the eMMC module and should be ok

On Saturday, June 22, 2024, sesame_madness @.***> wrote:

Is it difficult to restore in case my HA blue will not boot anymore?

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/operating-system/issues/3227#issuecomment-2183204571, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADSVIW7CCN3GDBSGIRQOPTZIRTLDAVCNFSM6AAAAABEJ5AVD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBTGIYDINJXGE . You are receiving this because you commented.Message ID: @.***>

smarthomefamilyverrips commented 2 weeks ago

If you bought HA Blue, I believe you'll have older stock of the eMMC module and should be ok … On Saturday, June 22, 2024, sesame_madness @.> wrote: Is it difficult to restore in case my HA blue will not boot anymore? — Reply to this email directly, view it on GitHub <#3227 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADSVIW7CCN3GDBSGIRQOPTZIRTLDAVCNFSM6AAAAABEJ5AVD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBTGIYDINJXGE . You are receiving this because you commented.Message ID: @.>

@peter-dolkens I have the HA blue for about 2.5 years and came with 128 GB eMMC, so I also assume it is not the newest type of eMMC.... I think just will need to gather some courage, cross my fingers and bite the bullet in the end, normally I like to keep my install up to date.

smarthomefamilyverrips commented 1 week ago

@all just to share, I managed to successful update my Home Assistant Blue from 12.2 to 12.4, all seems to work fine. I not did flash the eMCC , I just used the install option on the update notification within the HA UI.

ajfriesen commented 1 week ago

I'm reopening this issue because now I got my hands on a Kingston eMMC and I realized we might have two different issues:

Kingston eMMC modules that never worked correctly (the card correctly initializes only one in many boots), until we introduced a patch in ODROID-N2/C2/C4: Force 40MHz instead of 24MHz as eMMC clock #3319. This is tracked in ODROID-N2 doesn't boot from Kingston eMMC modules #3250.

However, that might have triggered Upgrade Odroid N2+ to 12.3, now unable to boot. #3351, where the faster frequency causes another regression, or actually increses its incidence. The root cause seems to be the update of U-Boot. As a workarond we have downgraded U-Boot to pre-11.5 state in Temporary revert ODROID-N2 U-Boot to 2023.01 to fix eMMC issues #3409 but that's not a long-term solution, as it also breaks Kingston again. If true, it means that the above dev build, and the 12.3 release could never work correctly for @scoobydrvr and others that do not have a Kingston eMMC.

I'd like to ask everyone here to provide us with as many details about the modules used as possible (ideally with the photos of the eMMC module with the sticker removed). Eventually, if you're located in the EU, I could arrange shipping you a new eMMC module and sourcing the faulty one from you - please get in touch with me if you are willing to help with that.

@sairon I have a Home Assistant Blue like forever. I did not update yet. Still on 12.2 because I do not have a second device I could run Home Assistant right now without much hassle.

Would those emmcs be affected by this bug? Would those be helpful for your testing?

PXL_20240624_213553490

PXL_20240624_213540194 MP

TibsD commented 1 week ago

Updating my Odroid n2+ with Hardkernel eMMC caused it to fail on boot. I will try to disconnect the Odroid and let it reboot this evening to, hopefully, solve the issue.

Edit: I tried to update to 12.4. A simple reboot did not work and also a 30 minute power cut did not work. I will leave it unplugged for the night and otherwise will reflash the eMMC.

cvasantrao commented 1 week ago

Hi I am a similar issue but booting from SDCard after a fresh install. Attaching the log with errors during setup. https://dpaste.org/YCfqv

home-assistant / operating-system