search

MinnowBoard-org / bugs-and-help

Ask QUESTIONS here. MinnowBoard.org issue and get help submission. See README for use.
14 stars 2 forks source link

Minnowboard Turbot quad core locks up randomly in Linux #64

Closed ftab closed 7 years ago

ftab commented 7 years ago

Referred to new issue via @Mark-van-der-Pol https://github.com/MinnowBoard-org/bugs-and-help/issues/60#issuecomment-329915920

I observed several instances of a hard lockup (no touch/keyboard response) at random intervals, ranging from minutes after booting up and starting and using multiple apps, to several hours before the issue occurred. Was logged into SSH, following journalctl, but didn’t see any messages. Also tried catching it over the serial console.

I thought it may have been the same issue that causes Memtest86 to fail in multi core mode, but the consensus is they’re two different things.

Unfortunately I no longer have the boards. After I reported the issue to Netgate, they instructed me to return them.

Tested on Automotive Grade Linux demo platform, and GENIVI development platform, both on SD card, and on two separate Turbot Quad core (single Ethernet) boards. Sorry, I didn’t think to try others before the RMA. Both of the ones I tried are built using Yocto, but they may’ve been at different branches (this was 2 months ago). I would be happy to do what I can to help walk someone else through setting either one up. From the looks of it, may be related to #57

Edited to mention I haven’t heard any reports from anyone with a dual core seeing this happen...

Mark-van-der-Pol commented 7 years ago

Great, thanks for the information - what capacity are the SD-Cards, and what make and class?

I've seen and heard similar reports to yours, and the one common factor is that the OS is running from SD-Cards. So I want to gather as much info about the different cards in use out there to see if we can identify a systemic issue with a particular brand, size or speed. Or if it is a general failure deep in an OS driver, agnostic of the card size, type or speed. Thanks, |\/|ark

ftab commented 7 years ago

Tried several SanDisk 8gb cards (black, class 4, no discernible "extreme" or "plus" branding - 5 MB typical dd write speed) as well as a Samsung Pro 64gb class 10/UHS-1, much faster

Mark-van-der-Pol commented 7 years ago

Great, thanks for the data points. Did all of these exhibit the freeze behaviour? |\/|.

ftab commented 7 years ago

Yeah. It didn’t seem to happen any sooner or later with either one

Mark-van-der-Pol commented 7 years ago

I have had boards of dual core and quad core pedigree running various linuxes of the SATA port, or the mSATA port or a USB-3 spinning HDD for weeks without issue,

I'm going to run some tests stressing the SD-card interface.

Thanks for the quick responses!

|\/ark.

BrianOttaway commented 7 years ago

Well Kudos to Netgate for taking the boards back so readily. But it would be interesting on the same boards to compare to OS install on mPCIe, mSATA, SATA, etc. The SD card size shouldn't have been an issue in your case. The fact that the boards work again after restart suggests something unique here....I'd normally think some kind of memory leak in SW or paging, but your Yocto and AGL images I'd be confident in.

BrianOttaway commented 7 years ago

Looks like this thread is stale. Closing it out. Do reopen if needed.