Closed CodePhase closed 1 year ago
Are the errors on all OS versions similar looking?
The error above looks like some type of hardware error, possible memory issue.
trying the official odroid ubuntu OS's (both the mate and minimal latest releases work fine)
Did you also try to stress those systems a bit? HAOS creates quite some stress as it downloads the latest version of Home Assistant Core on first boot (which leads to 1GB+ downloads, extraction and lots of writes to the eMMC).
How old is that device? What is the board revision?
I tried the stress-ng suite for CPU and memory and didn't see anything unusual. If there's some output that would be particularly helpful let me know and I can run and post the results. As far as errors on the other OS versions, I'd say they are similar but not identical. For example, here is an excerpt from an attempt with 8.5:
[ 3.827019] ------------[ cut here ]------------
[ 3.832381] meson8b-dwmac ff3f0000.ethernet: Ring mode enabled
[ 3.839351] WARNING: CPU: 2 PID: 270 at kernel/rcu/tree_plugin.h:725 rcu_sched_clock_irq+0x664/0xd80
[ 3.839368] Modules linked in: crct10dif_ce panfrost gpu_sched meson_dw_hdmi(+) dw_hdmi rc_odroid cec dwmac_meson8b(+) meson_drm meson_ir rtc_meson_vrtc stmmac_platform rc_core stmmac pcs_xpcs snd_soc_simple_amplifier fuse
[ 3.839398] CPU: 2 PID: 270 Comm: systemd-udevd Not tainted 5.15.60 #1
[ 3.839404] Hardware name: Hardkernel ODROID-N2 (DT)
[ 3.839407] pstate: 400000c5 (nZcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 3.845415] meson8b-dwmac ff3f0000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[ 3.850817] pc : rcu_sched_clock_irq+0x664/0xd80
[ 3.850827] lr : rcu_sched_clock_irq+0x464/0xd80
[ 3.850830] sp : ffff80000966bce0
[ 3.850832] x29: ffff80000966bce0 x28: ffff800008129ad0 x27: ffff0000e07b52c0
[ 3.850839] x26: 0000000000000000 x25: 0000000000000001 x24: 00000000000000c0
[ 3.850844] x23: 00000000e3edac47 x22: ffff0000e07b5320 x21: ffff0000078d3a00
[ 3.850848] x20: 0000000000000001 x19: ffff80000929b2c0 x18: 0000000000000000
[ 3.850852] x17: ffff8000d751f000 x16: ffff80000966c000 x15: 0000000000004000
[ 3.850856] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[0;32m OK ] Found device /dev/ttyAML0.
[ 3.850860] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[ 3.976751] x8 : 0000000000000000 x7 : 0000000000000000 x6 : ffff800009495170
[ 3.976755] x5 : ffff0000078d3a00 x4 : ffff80000966bce0 x3 : ffff0000078d3a00
[ 3.976760] x2 : ffff0000078d3a00 x1 : 0000000000000000 x0 : 00000000ffffffff
[ 3.976764] Call trace:
[ 3.976768] rcu_sched_clock_irq+0x664/0xd80
[ 3.976774] update_process_times+0x98/0xf0
[ 3.976779] tick_sched_handle.isra.0+0x34/0x60
[ 3.976784] tick_sched_timer+0x4c/0xb0
[ 3.976787] __hrtimer_run_queues+0x114/0x1c0
[ 3.976790] hrtimer_interrupt+0xe8/0x250
[ 3.976793] arch_timer_handler_phys+0x34/0x50
[ 3.976799] handle_percpu_devid_irq+0x84/0x140
[ 3.976804] handle_domain_irq+0x60/0x90
[ 3.976807] gic_handle_irq+0xa4/0xe0
[ 3.976812] call_on_irq_stack+0x28/0x58
[ 3.976817] do_interrupt_handler+0x54/0x60
[ 3.976821] el0_interrupt+0x34/0xc0
[ 3.976825] __el0_irq_handler_common+0x18/0x30
[ 3.976829] el0t_64_irq_handler+0x10/0x20
[ 3.976831] el0t_64_irq+0x1a0/0x1a4
[ 3.976835] ---[ end trace 5afd489c0731d9d8 ]---
and
[ 5.298724] Unable to handle kernel execute from non-executable memory at virtual address ffff0000e07d2f48
[ 5.303941] Mem abort info:
[ 5.303945] ESR = 0x8600000f
[[ 5.303948] EC = 0x21: IABT (current EL), IL = 32 bits
[ 5.314993] SET = 0, FnV = 0
[ 5.317971] EA = 0, S1PTW = 0
[ 5.321040] FSC = 0x0f: level 3 permission fault
[ 5.325795] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000093dd000
[ 5.332437] [ffff0000e07d2f48] pgd=18000000f4e52003, p4d=18000000f4e52003, pud=18000000e4864003, pmd=18000000e4760003, pte=00680000e07d2707
[ 5.346016] Internal error: Oops: 8600000f [#1] PREEMPT SMP
The board is a couple of years old, 2019 I think. The revision is 0.4 20190327. Thank you for having a look!
Hm, that error looks quite a bit different again. The problem reminds me of #1471.
These type of errors are usually caused by memory problems. Now since it works on a different OS, it seems to suggest that we use different memory settings than they are using. The memory timings are set by the bootloader, and last time I checked the official ODROID image and ours was using the exact same firmware there.
What official release did you test? Can you share the image link?
Did you boot the offical image and HAOS image from the same boot media?
Just for good measure, here's some stress-ng results:
stress-ng --brk 2 --stack 2 --bigheap 2 -t 2m
stress-ng: info: [3304] setting to a 120 second (2 mins, 0.00 secs) run per stressor
stress-ng: info: [3304] dispatching hogs: 2 brk, 2 stack, 2 bigheap
stress-ng: info: [3304] successful run completed in 120.20s (2 mins, 0.20 secs)
stress-ng --matrix 0 -t 2m --times
stress-ng: info: [3347] setting to a 120 second (2 mins, 0.00 secs) run per stressor
stress-ng: info: [3347] dispatching hogs: 6 matrix
stress-ng: info: [3347] successful run completed in 120.00s (2 mins, 0.00 secs)
stress-ng: info: [3347] for a 120.00s run time:
stress-ng: info: [3347] 720.02s available CPU time
stress-ng: info: [3347] 719.40s user time ( 99.91%)
stress-ng: info: [3347] 0.05s system time ( 0.01%)
stress-ng: info: [3347] 719.45s total time ( 99.92%)
stress-ng: info: [3347] load average: 5.85 3.39 1.51
stress-ng --class memory --all 1 -t 2m
stress-ng: info: [3356] setting to a 120 second (2 mins, 0.00 secs) run per stressor
stress-ng: info: [3356] dispatching hogs: 1 atomic, 1 bad-altstack, 1 bsearch, 1 context, 1 full, 1 heapsort, 1 hsearch, 1 judy, 1 list, 1 lockbus, 1 lsearch, 1 malloc, 1 matrix, 1 matrix-3d, 1 mcontend, 1 membarrier, 1 memcpy, 1 memfd, 1 memrate, 1 memthrash, 1 mergesort, 1 mincore, 1 misaligned, 1 null, 1 numa, 1 pipe, 1 pipeherd, 1 prefetch, 1 qsort, 1 radixsort, 1 randlist, 1 remap, 1 resources, 1 rmap, 1 shellsort, 1 skiplist, 1 sparsematrix, 1 stack, 1 stackmmap, 1 str, 1 stream, 1 tlb-shootdown, 1 tmpfs, 1 tree, 1 tsearch, 1 vm, 1 vm-addr, 1 vm-rw, 1 vm-segv, 1 wcs, 1 zero, 1 zlib
stress-ng: info: [3378] stress-ng-memthrash: starting 6 threads on each of the 1 stressors on a 6 CPU system
stress-ng: info: [3381] stress-ng-misaligned: skipping method int16atomic, misaligned operations tripped signal 7 'SIGBUS'
stress-ng: info: [3387] stress-ng-prefetch: using built-in defaults as no suitable cache found
stress-ng: info: [3381] stress-ng-misaligned: skipping method int32atomic, misaligned operations tripped signal 7 'SIGBUS'
stress-ng: info: [3383] stress-ng-numa: system has 1 of a maximum 4 memory NUMA nodes
stress-ng: info: [3381] stress-ng-misaligned: skipping method int64atomic, misaligned operations tripped signal 7 'SIGBUS'
stress-ng: info: [3411] stress-ng-sparsematrix: 5000 items in 500 x 500 sparse matrix (2.00% full)
stress-ng: info: [3424] stress-ng-stream: using built-in defaults as no suitable cache found
stress-ng: info: [3424] stress-ng-stream: stressor loosely based on a variant of the STREAM benchmark code
stress-ng: info: [3424] stress-ng-stream: do NOT submit any of these results to the STREAM benchmark results
stress-ng: info: [3424] stress-ng-stream: Using CPU cache size of 2048K
stress-ng: info: [3381] stress-ng-misaligned: skipping method int128tomic, misaligned operations tripped signal 7 'SIGBUS'
stress-ng: info: [3381] stress-ng-misaligned: exercised int16rd int16wr int16inc int32rd int32wr int32inc int64rd int64wr int64inc int128rd int128wr int128inc
stress-ng: info: [3411] stress-ng-sparsematrix: hash 161.4K 280708.79 Get/s 134882.43 Put/s
stress-ng: info: [3411] stress-ng-sparsematrix: judy 85.6K 210287.73 Get/s 70618.53 Put/s
stress-ng: info: [3411] stress-ng-sparsematrix: list 174.6K 11763.04 Get/s 8481.36 Put/s
stress-ng: info: [3411] stress-ng-sparsematrix: mmap 1.9M 161663.47 Get/s 39406.55 Put/s
stress-ng: info: [3411] stress-ng-sparsematrix: qhash 162.4K 462007.66 Get/s 232900.91 Put/s
stress-ng: info: [3411] stress-ng-sparsematrix: rb 232.7K 109434.05 Get/s 50993.22 Put/s
stress-ng: info: [3424] stress-ng-stream: memory rate: 95.28 MB/sec, 38.11 Mflop/sec (instance 0)
stress-ng: info: [3377] stress-ng-memrate: write1024: 88.10 MB/sec
stress-ng: info: [3377] stress-ng-memrate: write512: 80.99 MB/sec
stress-ng: info: [3377] stress-ng-memrate: write256: 81.33 MB/sec
stress-ng: info: [3377] stress-ng-memrate: write128: 79.87 MB/sec
stress-ng: info: [3377] stress-ng-memrate: write64: 91.02 MB/sec
stress-ng: info: [3377] stress-ng-memrate: write32: 84.02 MB/sec
stress-ng: info: [3377] stress-ng-memrate: write16: 67.07 MB/sec
stress-ng: info: [3377] stress-ng-memrate: write8: 42.75 MB/sec
stress-ng: info: [3377] stress-ng-memrate: read64pf: 123.98 MB/sec
stress-ng: info: [3377] stress-ng-memrate: read1024: 101.65 MB/sec
stress-ng: info: [3377] stress-ng-memrate: read512: 138.15 MB/sec
stress-ng: info: [3377] stress-ng-memrate: read256: 138.58 MB/sec
stress-ng: info: [3377] stress-ng-memrate: read128: 130.77 MB/sec
stress-ng: info: [3377] stress-ng-memrate: read64: 107.59 MB/sec
stress-ng: info: [3377] stress-ng-memrate: read32: 109.55 MB/sec
stress-ng: info: [3377] stress-ng-memrate: read16: 103.49 MB/sec
stress-ng: info: [3377] stress-ng-memrate: read8: 66.54 MB/sec
stress-ng: info: [3480] stress-ng-zlib: instance 0: compression ratio: 60.92% (0.03 MB/sec)
stress-ng: info: [3387] stress-ng-prefetch: using a 4096 KB L3 cache, 1408 benchmark rounds
stress-ng: info: [3387] stress-ng-prefetch: non-prefetch read rate @ 0.26 GB/s
stress-ng: info: [3387] stress-ng-prefetch: best prefetch read rate @ 4.66 GB/s at offset 6976 (~1394.77 nanoseconds)
stress-ng: info: [3386] stress-ng-pipeherd: 3.10 context switches per bogo operation (123442.60 per second)
stress-ng: info: [3356] successful run completed in 121.25s (2 mins, 1.25 secs)
stress-ng --mq 0 -t 2m --times --perf
stress-ng: info: [23948] setting to a 120 second (2 mins, 0.00 secs) run per stressor
stress-ng: info: [23948] dispatching hogs: 6 mq
stress-ng: info: [23948] successful run completed in 120.01s (2 mins, 0.01 secs)
stress-ng: info: [23948] Cannot read perf counters, do not have CAP_SYS_ADMIN capability or /proc/sys/kernel/perf_event_paranoid is set too high (3)
stress-ng: info: [23948] for a 120.01s run time:
stress-ng: info: [23948] 720.04s available CPU time
stress-ng: info: [23948] 135.47s user time ( 18.81%)
stress-ng: info: [23948] 548.77s system time ( 76.21%)
stress-ng: info: [23948] 684.24s total time ( 95.03%)
stress-ng: info: [23948] load average: 30.89 49.90 24.32
Hm, that error looks quite a bit different again. The problem reminds me of #1471.
These type of errors are usually caused by memory problems. Now since it works on a different OS, it seems to suggest that we use different memory settings than they are using. The memory timings are set by the bootloader, and last time I checked the official ODROID image and ours was using the exact same firmware there.
What official release did you test? Can you share the image link?
Did you boot the offical image and HAOS image from the same boot media?
Yeah, I saw that issue and tried different network cables and even different switches just in case. I tried two Odroid images, minimal version 20220622 and mate version 20220629
I can check the checksums of the firmware you mentioned in both official Odroid images and the HAOS if you think that would help. Do you know the locations of the files?
I can check the checksums of the firmware you mentioned in both official Odroid images and the HAOS if you think that would help. Do you know the locations of the files?
Unfortunately it is not that easy: These firmware files are bundled at build time and part of the U-Boot boot loader. Since we build our own version of U-Boot, these things won't match up. You basically have to figure out how the images are being built. Unfortunately these things are not well documented (or I am unable to find the documentation).
Things are better documented/clearer for Armbian, they get the firmware files from https://github.com/LibreELEC/amlogic-boot-fip. I was considering switching to those anyways.
Can you try Armbian on that particular board (along with stressing a bit)? https://www.armbian.com/odroid-n2/
From the explanations in this slide deck, it seems that acs.bin
is the responsible file for memory/DDR timings.
At least Armbian seems to use the very same timings as we are using, so if that is the problem, Armbian should fail on your platform as well.
Interesting. I do get similar errors from Armbian version 22.08:
[ 6.356893] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000019
[ 6.365645] Mem abort info:
[ 6.365650] ESR = 0x0000000096000044
[ 6.365652] EC = 0x25: DABT (current EL), IL = 32 bits
[ 6.365654] SET = 0, FnV = 0
[ 6.365656] EA = 0, S1PTW = 0
[ 6.365657] FSC = 0x04: level 0 translation fault
[ 6.365659] Data abort info:
[ 6.365660] ISV = 0, ISS = 0x00000044
[ 6.365661] CM = 0, WnR = 1
[ 6.365663] user pgtable: 4k pages, 48-bit VAs, pgdp=00000000081ec000
[ 6.365666] [0000000000000019] pgd=0000000000000000, p4d=0000000000000000
[ 6.365671] Internal error: Oops: 96000044 [#1] PREEMPT SMP
[ 6.365675] Modules linked in: sch_fq_codel ramoops reed_solomon pstore_blk pstore_zone efi_pstore ip_tables x_tables autofs4 meson_gxl panfrost gpu_sched rtc_pcb
[ 6.365704] CPU: 3 PID: 868 Comm: loadkeys Tainted: G W 5.19.17-meson64 #22.08.7
[ 6.365708] Hardware name: Hardkernel ODROID-N2 (DT)
[ 6.365710] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 6.365714] pc : _raw_spin_unlock_irqrestore+0x10/0x50
[ 6.365725] lr : __stack_depot_save+0x300/0x4e0
[ 6.424588] systemd[1]: Finished Coldplug All udev Devices.
[ 6.443074] sp : ffff80000acebbc0
[ 6.443075] x29: ffff80000acebbc0 x28: ffff00003f35d880 x27: 0000000000000000
[ 6.443081] x26: 0000000000000001 x25: ffff00000000c400 x24: ffff00000000c400
[ 6.443085] x23: 0000000000000001 x22: 0000000000004b47 x21: 0000000000000000
[ 6.496035] x20: ffff80000945ecb8 x19: 0000ffffe5750328 x18: 0000000000000000
[ 6.496040] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[ 6.496044] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 6.517252] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[ 6.520367] systemd[1]: Starting Helper to synchronize boot up for ifupdown...
[ 6.524320]
[ 6.524322] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
[ 6.524326] x5 : 0000000000400100 x4 : ffff00003f35d880 x3 : 0000000000000000
[ 6.524330] x2 : 0000000000000000 x1 : ffff80000945ecb8 x0 : 0000000000000019
[ 6.524334] Call trace:
ting Restore / save the[ 6.563654] _raw_spin_unlock_irqrestore+0x10/0x50
[ 6.563664] __stack_depot_save+0x300/0x4e0
[ 6.563669] vt_ioctl+0x7d0/0x14f0
[ 6.563674] tty_ioctl+0x2bc/0xc30
[ 6.563677] __arm64_sys_ioctl+0xa8/0xe8
[ 6.563681] invoke_syscall+0x44/0x108
[ 6.563687] el0_svc_common.constprop.3+0x94/0xf8
[ 6.563691] do_el0_svc+0x2c/0xb8
[ 6.563694] el0_svc+0x20/0x50
[ 6.563698] el0t_64_sync_handler+0x98/0xc0
[ 6.563701] el0t_64_sync+0x170/0x174
[ 6.563706] Code: d503233f a9bf7bfd 52800002 910003fd (089ffc02)
[ 6.563710] ---[ end trace 0000000000000000 ]---
[ 6.951242] Unable to handle kernel read from unreadable memory at virtual address fffffbfffdcec9de
[ 6.959628] Mem abort info:
[ 6.962385] ESR = 0x000000009600000e
[ 6.966094] EC = 0x25: DABT (current EL), IL = 32 bits
[ 6.971354] SET = 0, FnV = 0
[ 6.974373] EA = 0, S1PTW = 0
[ 6.977478] FSC = 0x0e: level 2 permission fault
[ 6.982222] Data abort info:
[ 6.985068] ISV = 0, ISS = 0x0000000e
[ 6.988863] CM = 0, WnR = 0
[ 6.991796] swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000003552f000
[ 6.998437] [fffffbfffdcec9de] pgd=00000000359d7003, p4d=00000000359d7003, pud=00000000359d8003, pmd=006000003e800781
[ 7.008961] Internal error: Oops: 9600000e [#2] PREEMPT SMP
[ 7.014481] Modules linked in: snd_soc_meson_axg_frddr(+) snd_soc_meson_axg_tdmout videobuf2_dma_contig(+) meson_ir(+) v4l2_mem2mem snd_soc_meson_axg_toddr rc_cob
[ 7.080722] CPU: 3 PID: 924 Comm: systemd-udevd Tainted: G D W 5.19.17-meson64 #22.08.7
[ 7.089776] Hardware name: Hardkernel ODROID-N2 (DT)
[ 7.094693] pstate: 400000c5 (nZcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 7.101592] pc : __pi_strcmp+0xe4/0x140
[ 7.105387] lr : __of_find_property+0x40/0x88
[ 7.109699] sp : ffff80000ad9b8f0
[ 7.112977] x29: ffff80000ad9b8f0 x28: 0000000000000000 x27: 0000000000000000
[ 7.120049] x26: ffff80000ad9bce0 x25: ffff0000e48593e0 x24: ffff000009b5d968
[ 7.127122] x23: ffff800001212120 x22: ffff800001212140 x21: 0000000000000000
[ 7.134194] x20: ffff8000093c1e58 x19: ffff0000e48594c8 x18: 0000000000000000
[ 7.141267] x17: 0000000000000000 x16: 0000000000000000 x15: 6143296c6c756e28
[ 7.148339] x14: 0000000000000040 x13: 0000000000000228 x12: 0000000000000000
[ 7.155412] x11: 0000000000000000 x10: ffff84000b6d5524 x9 : 0003ffffb61f0d20
[ 7.162484] x8 : 0101010101010101 x7 : 000000000000656c x6 : 7f7f7f7f7f7f7f7f
[ 7.169557] x5 : ffff84000b6d5528 x4 : 0000000000000000 x3 : 0000000000000070
[ 7.176629] x2 : 0000000000000070 x1 : ffff8000093c1e60 x0 : fffffbfffdcec938
[ 7.183703] Call trace:
[ 7.186117] __pi_strcmp+0xe4/0x140
[ 7.189568] __of_device_is_compatible+0x84/0x150
[ 7.194224] __of_match_node.part.6+0x2c/0x80
[ 7.198537] of_match_node+0x40/0x70
[ 7.202073] of_match_device+0x28/0x40
[ 7.205782] platform_match+0x4c/0xe8
[ 7.209404] __driver_attach+0x2c/0x1d8
[ 7.213199] bus_for_each_dev+0x74/0xc8
[ 7.216994] driver_attach+0x24/0x30
[ 7.220531] bus_add_driver+0x19c/0x240
[ 7.224325] driver_register+0x64/0x120
[ 7.228121] __platform_driver_register+0x28/0x38
[ 7.232778] axg_frddr_pdrv_init+0x20/0x1000 [snd_soc_meson_axg_frddr]
[ 7.239247] do_one_initcall+0x74/0x1e8
[ 7.243042] do_init_module+0x44/0x1e0
[ 7.246751] load_module+0x1950/0x1c48
[ 7.250459] __do_sys_init_module+0x1dc/0x250
[ 7.254772] __arm64_sys_init_module+0x1c/0x28
[ 7.259171] invoke_syscall+0x44/0x108
[ 7.262879] el0_svc_common.constprop.3+0x84/0xf8
[ 7.267537] do_el0_svc+0x2c/0xb8
[ 7.270814] el0_svc+0x20/0x50
[ 7.273833] el0t_64_sync_handler+0x98/0xc0
[ 7.277973] el0t_64_sync+0x170/0x174
[ 7.281599] Code: 540001e1 cb000025 d503201f f8656807 (f86a6803)
[ 7.287635] ---[ end trace 0000000000000000 ]---
[ 7.292205] note: systemd-udevd[924] exited with preempt_count 1
[ 67.303848] rcu: INFO: rcu_preempt self-detected stall on CPU
[ 67.304498] rcu: 2-....: (15000 ticks this GP) idle=bb3/1/0x4000000000000000 softirq=1321/1323 fqs=7488
[ 67.313986] (t=15002 jiffies g=-91 q=470 ncpus=6)
[ 67.318729] Task dump for CPU 0:
[ 67.321920] task:systemd-udevd state:R running task stack: 0 pid: 892 ppid: 1 flags:0x00000802
[ 67.331753] Call trace:
[ 67.334167] __switch_to+0xfc/0x1b8
[ 67.337617] 0xe20031668b4ee000
[ 67.340723] Task dump for CPU 2:
[ 67.343913] task:systemd-udevd state:R running task stack: 0 pid: 927 ppid: 892 flags:0x0000080a
[ 67.353746] Call trace:
[ 67.356161] dump_backtrace+0x12c/0x138
[ 67.359956] show_stack+0x18/0x50
[ 67.363233] sched_show_task+0x160/0x1a0
[ 67.367114] dump_cpu_task+0x44/0x54
[ 67.370651] rcu_dump_cpu_stacks+0xf0/0x134
[ 67.374791] rcu_sched_clock_irq+0xa14/0xe10
[ 67.379017] update_process_times+0x68/0xb0
[ 67.383157] tick_sched_handle.isra.21+0x40/0x58
[ 67.387728] tick_sched_timer+0x4c/0xa8
[ 67.391523] __hrtimer_run_queues+0x11c/0x1b0
[ 67.395836] hrtimer_interrupt+0xe8/0x240
[ 67.399803] arch_timer_handler_phys+0x30/0x40
[ 67.404202] handle_percpu_devid_irq+0x84/0x138
[ 67.408687] generic_handle_domain_irq+0x2c/0x48
[ 67.413258] gic_handle_irq+0x98/0xd0
[ 67.416881] call_on_irq_stack+0x2c/0x58
[ 67.420762] do_interrupt_handler+0x84/0x88
[ 67.424902] el1_interrupt+0x38/0x70
[ 67.428438] el1h_64_irq_handler+0x18/0x28
[ 67.432492] el1h_64_irq+0x64/0x68
[ 67.435856] smp_call_function_many_cond+0x174/0x390
[ 67.440772] smp_call_function+0x40/0x78
[ 67.444653] kick_all_cpus_sync+0x24/0x30
[ 67.448621] load_module+0x15a8/0x1c48
[ 67.452329] __do_sys_init_module+0x1dc/0x250
[ 67.456642] __arm64_sys_init_module+0x1c/0x28
[ 67.461041] invoke_syscall+0x44/0x108
[ 67.464749] el0_svc_common.constprop.3+0x84/0xf8
[ 67.469407] do_el0_svc+0x2c/0xb8
[ 67.472684] el0_svc+0x20/0x50
[ 67.475703] el0t_64_sync_handler+0x98/0xc0
[ 67.479843] el0t_64_sync+0x170/0x174
[ 67.483466] Task dump for CPU 3:
[ 67.486657] task:systemd-udevd state:R running task stack: 0 pid: 941 ppid: 892 flags:0x00000802
[ 67.496490] Call trace:
[ 67.498905] __switch_to+0xfc/0x1b8
[ 67.502354] 0xe20031668b4ee000
[ 67.505460] Task dump for CPU 4:
[ 67.508651] task:systemd-udevd state:R running task stack: 0 pid: 960 ppid: 892 flags:0x00000802
[ 67.518483] Call trace:
[ 67.520898] __switch_to+0xfc/0x1b8
[ 67.524348] 0xffff0000087d0245
I can't get far enough to run the stress test because the system crashes at startup.
Hm interesting indeed, so probably something with the new kernel.
Can you try a 5.10.x based build from their archive, e.g. https://imola.armbian.com/archive/odroidn2/archive/Armbian_21.05.4_Odroidn2_buster_current_5.10.43.img.xz
We did had issues at one point with frequency scaling, but currently we HAOS uses the performance scaling governor, which keeps the CPU clock on a high frequency.
I've not heard similar on the N2+ yet...
I'm currently running 6.0.5 on the N2+.
sbc-bench
results: http://ix.io/4eQP
Today's testing images: https://github.com/rpardini/armbian-release/releases/download/20221103a/Armbian_20221103a-rpardini_Odroidn2_kinetic_edge_6.0.6.img.xz
using the Armbian 21.05.4 image I still get errors at startup and a failed boot. I attached the log.
odroidN2-armbian21_05-emmc-2.log
Seems to be a similar error
trying the 6.0.6 Armbian image also gives me virtual memory errors.
odroidN2-armbian6_0_6-emmc-2.log
I think I'll try the Odroid image again and see if there's some other way to do another memory test. I'll report back afterwards.
I ran the stress-ng tests again as well as the sbc-bench (thanks @rpardini I didn't know about this). I can't see any issues with memory in any of the test results. Here's the link to the sbc-bench results: http://ix.io/4eSN. Here's a complete log with the bootup, testing, dmesg, and complete journal in case anyone is interested:
Yeah, I'm stumped. I've not the original N2, and my N2+ works with the 6.0.6 image. Looking around, there's a lot of similar stuff, https://github.com/home-assistant/operating-system/issues/1096 and https://github.com/home-assistant/operating-system/issues/1471 at least...
The fact vendor kernel 4.9 works doesn't help much; did you ever get a 5.10+ kernel working?
That 21.05.4
's logs indicate a vendor u-boot trying and failing to boot a kernel... it's an unrelated error. That image might have been messed up 🤦 or maybe you've used an uboot in SPI?
@CodePhase Maybe try Armbian_22.08.1_Odroidn2_jammy_current_5.10.139? (make sure switch is in eMMC position)
Thanks @rpardini but that image doesn't work either. I can confirm that I have the boot selector switch to the 'MMC' option on the board. Looks like the uboot on the image has the same issue where it can't boot the kernel. To answer your previous question, no I have never gotten a 5.10+ kernel to boot. I usually only use the Odroid Ubuntu images. Maybe I'll try a simple one like CoreElec or something.
The latest build of CoreElec works (https://github.com/CoreELEC/CoreELEC/releases/download/20.0-Nexus_alpha3/CoreELEC-Amlogic-ng.arm-20.0-Nexus_alpha3-Odroid_N2.img.gz) however it is running kernel 4.9.269:
uname -r 4.9.269
I suppose there is some issue with kernels in the 5.x and above series for N2 (non-plus) boards?
I wonder if it's just time to give up on getting this to work and just install HomeAssistant on the Odroid Ubuntu Minimal image. I am also stumped that I seem to be the only person having this issue. Surely others have attempted to install this on a non-plus N2 board. Do either of you (@agners or @rpardini) see any value in continuing to troubleshoot this or is it time to throw in the towel?
Thanks for testing @CodePhase. Maybe some Manjaro kernels? I'm out of ideas, sorry.
There are definitely other non-N2+ installations out there, I think @pvizeli is running one. But it seems that your particular device is very special 😰. Without having a failing HW here, it's hard to debug. If using their Odroid image works for you, probably the best way forward, I am sorry.
Yes, my Parents have an N2. I think it should be one of the latest HW revisions because it was the Prototype for the Blue and Hardkernel switch 2 months later over to N2+
What I can say, is it seems to work stable, but I can take a look after my parents are back
There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
Describe the issue you are experiencing
I have been trying to install HAOS on my Odroid N2 (not the plus) 4 gig SBC for some time now without success. I have tried multiple versions (9.3, 9.2, 8.5, 6.0, 5.13) all with the same issue. I have followed the install documentation using Etcher on https://www.home-assistant.io/installation/odroid for every attempt. I have also tried using 'dd' to copy the image file after extracting it from xz format.
I have tried ruling out my SBC as the issue by:
The full log for the 9.3 attempt is in the host log section below
What operating system image do you use?
odroid-n2 (Hardkernel ODROID-N2/N2+)
What version of Home Assistant Operating System is installed?
9.3
Did you upgrade the Operating System.
No
Steps to reproduce the issue
Anything in the Supervisor logs that might be useful for us?
Anything in the Host logs that might be useful for us?
System information
No response
Additional information
No response