search

motioneye-project / motioneyeos

A Video Surveillance OS For Single-board Computers
Other
7.84k stars 897 forks source link

Odroid C2 Will Not Boot After Initial Boot #1892

Open rwmccoy opened 5 years ago

rwmccoy commented 5 years ago

I am setting up an Odroid C2 to be used as a hub. It has no peripherals attached and no configuration has yet been done.

After flashing the memory card, the C2 boots normally and is pingable with the web interface and SSH available.

When I reboot having made no changes to the initial configuration, boot fails. The device is not pingable or reachable by SSH. There is no web interface.

Preliminary Docs

I confirm that I have read the CONTRIBUTING guide before opening this issue.

I confirm that I have read the FAQ before opening this issue.

motionEyeOS Version

I am running motionEyeOS version: dev20190408

Board Model

I am using the following board/model: Odroid C2.

Camera

I am using the following type of camera: No camera yet attached. (This will be a hub.)

My camera model is: None.

Network Connection

My motionEyeOS unit is connected to the network via: Ethernet

Peripherals

I am using the following peripherals that I consider relevant to this issue:

Log Files

I consider the following log files relevant to this issue: I am attaching all log files as they exist after the first boot. After first boot, the board does not boot again. Looking at the memory card on a laptop, the Log directory is not available, saying: This link cannot be used because its target “/data/log” doesn’t exist. dmesg.log boot.log messages.log motioneye.log motion.log

ccrisan commented 5 years ago

@rwmccoy thanks for testing our nightlies. I have just tried to reproduce your problem using current nightly build (dev20190413) but everything seems to work fine here after many reboots.

You could try dev20190413 but there aren't any significant changes since dev20190408 that could have fixed your problem.

From the top of my head, your issue could be related to the SD card (size/speed/model). The fact that you have no /data/log after first boot is definitely not ok, given that the first boot procedure successfully takes care of it (from boot.log):

 * Creating data partition: done
 * Formatting data partition: done
 * Mounting data partition: done
 * Creating required data files: done

Here are my suggestions (random order):

  1. Try older versions and see if the problem (using your exact hardware configuration) is still there.
  2. Try a different SD card model.
  3. Determine if this issue is reproducible.
  4. Dump the "corrupted" SD card contents to a file (e.g. using dd), compress it and put it somewhere where we could download it and take a look at it.
  5. Connect a USB2serial cable and see what it says during boot.
rwmccoy commented 5 years ago

Thanks for an actionable plan.

Since you can't reproduce the problem, I'm going to try some different models of SD card first, given their past track record. Then, if that doesn't correct the problem, I'll DD the SD card to a file and send you the URL.

In the meantime, I'm waiting on a USB2serial adapter that will be here on Tuesday. I thought I had the parts to cobble one together at the right voltage level, but it seems not. If it's still failing on Tuesday, we'll have some console logs to look at.

rwmccoy commented 5 years ago

I've tried the following SD Cards:

And the following releases:

In all cases the initial boot was successful. When rebooted immediately, the Odroid was no longer responsive.

I have attached console logs from the initial boot and from the reboot. There are errors in both. Hopefully this will lead you to the problem.

This URL points to an image of the SD card after the failure.

Let me know if I can gather any more information or try something else. OdroidC2 Initial Boot.txt OdroidC2 Reboot.txt

ccrisan commented 5 years ago

Thanks for the detailed report. A preliminary test on your OS image indicates that the data partition becomes corrupted right after the first boot, which is consistent with what your boot logs say. A simple e2fsck fixes (to some extent) the partition.

Given that you have ruled out SD cards and versions, I'd suggest trying a different power supply.

I would love to hear from other Odroid C2 users, what they'd have to say about this issue. According to GitHub stats, there should be about 70 of them.

rwmccoy commented 5 years ago

Thanks

I've been using the "official" Odroid power supply with the barrel. connector.

I tried it with a Raspberry Pi USB 2.5 amp power supply and got the same symptoms.

Strange, since your C2 is working. Could there have been changes to the design along the way? Mine was ordered in September of 2018.

ccrisan commented 5 years ago

@rwmccoy could you please try powering it via microUSB? That's what I do here. Otherwise, I'm out of ideas.

Could there have been changes to the design along the way? Mine was ordered in September of 2018.

I seriously doubt that.

rwmccoy commented 5 years ago

I did power it with microUSB when I tried the Raspberry Pi 2 amp power supply. I moved the jumper to accept power from USB and powered it up. This morning I tried another Raspberry Pi supply that is rated at 5.25 volts instead of the normal 5.00. Still no change in behavior.

Looking at the board, this one is Rev0.2 20171114.

I've only ever used this board as an iPerf endpoint so I suppose there could be some problem with the board that didn't show up until now. If we give up on this one, what's your favorite SBC that you support that will give me the best performance as I continue to add cameras?

ccrisan commented 5 years ago

Could you try other OSses and do some power/rboot cycles to see if the partitions ever get corrupted?

I favor the Odroid XU4 since it's a powerful octacore board and its opensource support is a lot better than the one for Odroid C2.

rwmccoy commented 5 years ago

Will do. There will be a slight delay while family is around for Easter.

Out of curiosity, I've got an eMMC card on order to see if the problem persists with a new type of media.

Thanks for the tip on the XU4. Right now the Raspberry Pi 3B+ is working well for testing. I'll order an XU4 as I start ordering cameras.

Thank you for all the time you are spending on this.

ccrisan commented 5 years ago

Out of curiosity, I've got an eMMC card on order to see if the problem persists with a new type of media.

Unfortunately there's currently no (easy) way to dictate which "disk" to be used by the OS. It's hardcoded to /dev/mmcblk0 and, in the case of an eMMC, it's probably going to be /dev/mmcblk1.

At some point we'll need to address this limitation in thingOS, but for now, you'll have to manually edit your /etc/fstab.sys and possibly some stuff in /boot.

Nevertheless, I'd be happy to know how this is going to play out.

rwmccoy commented 5 years ago

Back from the family trip. Thanks for your patience.

I ordered preloaded Linux SDCard and eMMC from Hardkernel.

Both card formats loaded and reloaded multiple times with no issues.

I loaded MotioneyeOS on to SDCard and eMMC card. Booted from both and, in both cases, reboots failed. I've attached console logs from both attempts.

FYI, I didn't have to change anything to boot from eMMC. I used Etcher to load the MotioneyeOS to the eMMC card using a USB adapter. The copy went fine but the verify failed. Even so, the image booted the first time without issue and failed the same way on reboot.

In all cases the Odroid was powered by USB to match your setup.

OdroidC2 SDCard Reboot Failure Console.txt OdroidC2 eMMC Reboot Failure Console.txt

ccrisan commented 5 years ago

@rwmccoy thanks to you for your patience! At this point we can rule out any kind of SD card/storage-induced problem. The problem can be highlighted by the following log message:

Mounting filesystems: mount: mounting /dev/mmcblk0p3 on /data failed: Input/output error

The rest of the errors doesn't even matter, since the data partition could not be mounted.

Honestly I don't have any further ideas on this issue, other than trying with a different board.

I'll try to follow the community's recent efforts and update the kernel to mainline: https://forum.odroid.com/viewtopic.php?f=135&t=22717, but I doubt that could be the source of your problem.

You could try to mess up with the /boot/boot.ini file. This is what a recent Ubuntu build comes with:

ODROIDC2-UBOOT-CONFIG

########################################################################
# Changes made to this are overwritten every time there's a new upgrade
# To make your changes permanent change it on 
# boot.ini.default
# After changing it on boot.ini.default run the bootini command to
# rewrite this file with your personal permanent settings.
# Documentation: http://odroid.com/dokuwiki/doku.php?id=en:c2_persistent_bootini
########################################################################

# Possible screen resolutions
# Uncomment only a single Line! The line with setenv written.
# At least one mode must be selected.

# Custom modeline!
# To use custom modeline you need to disable all the below resolutions
# and setup your own! 
# For more information check our wiki: 
# http://odroid.com/dokuwiki/doku.php?id=en:c2_hdmi_autosetting
# Example below:
# setenv m "custombuilt" 
# setenv modeline "1920,1200,154000,74040,60,1920,1968,2000,2080,1200,1202,1208,1235,1,0,1"

# Auto Detection of Monitor settings based on your Screen information
# See: http://odroid.com/dokuwiki/doku.php?id=en:c2_auto_detect_display
setenv display_autodetect "true"

# 480 Lines (720x480)
# setenv m "480i60hz" # Interlaced 60Hz
# setenv m "480i_rpt" # Interlaced for Rear Projection Televisions 60Hz
# setenv m "480p60hz" # 480 Progressive 60Hz
# setenv m "480p_rpt" # 480 Progressive for Rear Projection Televisions 60Hz

# 576 Lines (720x576)
# setenv m "576i50hz" # Interlaced 50Hz
# setenv m "576i_rpt" # Interlaced for Rear Projection Televisions 50Hz
# setenv m "576p50hz" # Progressive 50Hz
# setenv m "576p_rpt" # Progressive for Rear Projection Televisions 50Hz

# 720 Lines (1280x720)
# setenv m "720p50hz" # 50Hz
# setenv m "720p60hz" # 60Hz

# 1080 Lines (1920x1080)
# setenv m "1080i60hz" # Interlaced 60Hz
setenv m "1080p60hz" # Progressive 60Hz
# setenv m "1080i50hz" # Interlaced 50Hz
# setenv m "1080p50hz" # Progressive 50Hz
# setenv m "1080p24hz" # Progressive 24Hz

# 4K (3840x2160)
# setenv m "2160p30hz"    # Progressive 30Hz
# setenv m "2160p25hz"    # Progressive 25Hz
# setenv m "2160p24hz"    # Progressive 24Hz
# setenv m "smpte24hz"    # Progressive 24Hz SMPTE
# setenv m "2160p50hz"    # Progressive 50Hz
# setenv m "2160p60hz"    # Progressive 60Hz
# setenv m "2160p50hz420" # Progressive 50Hz with YCbCr 4:2:0 (Requires TV/Monitor that supports it)
# setenv m "2160p60hz420" # Progressive 60Hz with YCbCr 4:2:0 (Requires TV/Monitor that supports it)

### VESA modes ###
# setenv m "640x480p60hz"
# setenv m "800x480p60hz"
# setenv m "480x320p60hz"
# setenv m "480x800p60hz"
# setenv m "800x600p60hz"
# setenv m "1024x600p60hz"
# setenv m "1024x768p60hz"  
# setenv m "1280x800p60hz"
# setenv m "1280x1024p60hz"
# setenv m "1360x768p60hz"
# setenv m "1440x900p60hz"
# setenv m "1600x900p60hz"
# setenv m "1680x1050p60hz"
# setenv m "1600x1200p60hz"
# setenv m "1920x1200p60hz"
# setenv m "2560x1080p60hz"
# setenv m "2560x1440p60hz"
# setenv m "2560x1600p60hz"
# setenv m "3440x1440p60hz"

# HDMI BPP Mode
setenv m_bpp "32"
# setenv m_bpp "24"
# setenv m_bpp "16"

# HDMI DVI/VGA modes
# By default its set to HDMI, if needed change below.
# Uncomment only a single Line.
# setenv vout "dvi"
# setenv vout "vga"

# HDMI HotPlug Detection control
# Allows you to force HDMI thinking that the cable is connected.
# true = HDMI will believe that cable is always connected
# false = will let board/monitor negotiate the connection status
setenv hpd "true"
# setenv hpd "false"

# Monitor output
# Controls if HDMI PHY should output anything to the monitor
setenv monitor_onoff "false" # true or false

# Server Mode (aka. No Graphics)
# Setting nographics to 1 will disable all video subsystem
# This mode is ideal of server type usage. (Saves ~300Mb of RAM)
setenv nographics "0"

# Meson Timer
# 1 - Meson Timer
# 0 - Arch Timer 
# Using meson_timer improves the video playback however it breaks KVM (virtualization).
# Using arch timer allows KVM/Virtualization to work however you'll experience poor video
setenv mesontimer "1"

# UHS (Ultra High Speed) MicroSD mode enable/disable
setenv disableuhs "false"

# MicroSD Card Detection enable/disable
# Force the MMC controlled to believe that a card is connected.
setenv mmc_removable "true"

# USB Multi WebCam tweak
# Only enable this if you use it.
setenv usbmulticam "false"

# Disable/Enable ODROID-VU7 Touchscreen
setenv disable_vu7 "false"

# Default Console Device Setting
setenv condev "console=ttyS0,115200n8 console=tty0"   # on both

# CPU Frequency / Cores control
###########################################
### WARNING!!! WARNING!!! WARNING!!!
# Before changing anything here please read the wiki entry: 
# http://odroid.com/dokuwiki/doku.php?id=en:c2_set_cpu_freq
#
# MAX CPU's
# setenv maxcpus "1"
# setenv maxcpus "2"
# setenv maxcpus "3"
setenv maxcpus "4"

# MAX Frequency
# setenv max_freq "2016"  # 2.016GHz
# setenv max_freq "1944"  # 1.944GHz
# setenv max_freq "1944"  # 1.944GHz
# setenv max_freq "1920"  # 1.920GHz
# setenv max_freq "1896"  # 1.896GHz
# setenv max_freq "1752"  # 1.752GHz
# setenv max_freq "1680"  # 1.680GHz
# setenv max_freq "1656"  # 1.656GHz
setenv max_freq "1536"  # 1.536GHz

###########################################

# Boot Arguments
if test "${display_autodetect}" = "true"; then usb pwren; hdmitx edid; fi
if test "${m}" = "custombuilt"; then setenv cmode "modeline=${modeline}"; fi
if test "${disable_vu7}" = "false"; then setenv hid_quirks "usbhid.quirks=0x0eef:0x0005:0x0004"; fi

setenv bootargs "root=UUID=e139ce78-9841-40fe-8823-96a304a09859 rootwait ro ${condev} no_console_suspend hdmimode=${m} ${cmode} m_bpp=${m_bpp} vout=${vout} fsck.repair=yes net.ifnames=0 elevator=noop disablehpd=${hpd} max_freq=${max_freq} maxcpus=${maxcpus} monitor_onoff=${monitor_onoff} disableuhs=${disableuhs} mmc_removable=${mmc_removable} usbmulticam=${usbmulticam} ${hid_quirks}"

# Booting

setenv loadaddr "0x11000000"
setenv dtb_loadaddr "0x1000000"
setenv initrd_loadaddr "0x13000000"

fatload mmc 0:1 ${initrd_loadaddr} uInitrd
fatload mmc 0:1 ${loadaddr} Image
fatload mmc 0:1 ${dtb_loadaddr} meson64_odroidc2.dtb
fdt addr ${dtb_loadaddr}

if test "${mesontimer}" = "0"; then fdt rm /meson_timer; fdt rm /cpus/cpu@0/timer; fdt rm /cpus/cpu@1/timer; fdt rm /cpus/cpu@2/timer; fdt rm /cpus/cpu@3/timer; fi
if test "${mesontimer}" = "1"; then fdt rm /timer; fi

if test "${nographics}" = "1"; then fdt rm /reserved-memory; fdt rm /aocec; fi
if test "${nographics}" = "1"; then fdt rm /meson-fb; fdt rm /amhdmitx; fdt rm /picdec; fdt rm /ppmgr; fi
if test "${nographics}" = "1"; then fdt rm /meson-vout; fdt rm /mesonstream; fdt rm /meson-fb; fi
if test "${nographics}" = "1"; then fdt rm /deinterlace; fdt rm /codec_mm; fi

booti ${loadaddr} ${initrd_loadaddr} ${dtb_loadaddr}
rwmccoy commented 5 years ago

Well, that was different, at least.

I substituted your boot.ini contents for the one that comes with MotioneyeOS and got an infinite reboot. Console log attached.

I also realized that by using Hardkernal's preloaded SDCard and eMMC card, I wasn't testing my image burn process. So, I used Etcher on both Mac and Linux with the same results as before.

My first take would be it's the board too if I hadn't had successful boots and reboots with any image other than MotioneyeOS. Any ideas on additional tests I can make?

ccrisan commented 5 years ago

I managed to reproduce the problem myself. However only one in 3 cases ended up in this situation.

I'll try to see how things go with the mainline kernel.

rwmccoy commented 5 years ago

Interesting. Again, I'll be glad to do any testing I can to provide additional insight.

hrabcakm commented 5 years ago

Hi there, I would like to confirm that I have exactly same issue like you are mentioning here. After initial boot it works, but after follow up reboot OdroidC2 not responding. Running on OdroidC2 with 8GB eMMC.

Thanks a lot for and advice.

rwmccoy commented 5 years ago

Exactly correct. The initial boot works fine but it does not work after the first reboot. I've tested with various sizes of SD cards and also eMMC with no change in symptoms.

Thanks for reproducing the problem.

ccrisan commented 5 years ago

Guys, can you please try and see if the the latest nightly build fixes this problem? https://github.com/ccrisan/motioneyeos/releases/tag/nightly-dev

rwmccoy commented 5 years ago

Unfortunately, the same failure.

Let me know if I can provide any debugging info.

hrabcakm commented 5 years ago

Hi Guys,

I've tested dev release - dev20190610. Unfortunately same issue. After first restart, Odroid doesn't boot anymore.

I'm using Original AC Adapter as well as original eMMC drive. I've tried also several microSD Cards (Sandisk), without success. My OdroidC2 board revision - 0.2.

How can I Enable HDMI Output ? I can provide log files as well or at least some debugging info

Regards.

rwmccoy commented 5 years ago

Closed in error. (sorry!)

ccrisan commented 5 years ago

@hrabcakm you can't enable HDMI output because the "opensource" support for this board is in such a state that we don't even have a framebuffer. I tried to boot a mainline kernel but with no success. Your best bet is to use a USB to serial and capture the serial log. But I doubt it would be any different from what we've already got from @rwmccoy.

In any case, I still can't seem to manage to reproduce the issue at my end and I've also tried with different SD cards.

Here's another idea: could you please try and tweak the "disableuhs" setting that's now available in /boot/boot.ini?

rwmccoy commented 5 years ago

I altered the disableuhs setting and now I can't do the initial boot.

I'll come back to this tomorrow and verify this result with other cards, multiple vendors and multiple sizes.

rwmccoy commented 5 years ago

Reconfirmed I get no initial boot when setenv disableuhs "true"

Using Sandisk Ultra 16gb.