search

MycroftAI / hardware-mycroft-mark-II

Mycroft's Mark II Rpi mechanical, electrical and industrial designs
110 stars 32 forks source link

First boot fails. Where can I look for symptoms to debug? #87

Open enorberg opened 3 years ago

enorberg commented 3 years ago

Describe the bug Just put my Mark-II Dev Kit together. Trying to boot for the first time, but it fails.

To Reproduce Steps to reproduce the behavior:

  1. Apply power. RPi Red power led is on. RPi Green led is on for several seconds - as though reading USB Drive. Then goes out and after a few seconds, flickers once. SJ-201 shows blue leds circling for ~ 1 min. SJ-201 flashes red led circle. SJ-201 flashes yellow/white led circle. Then leds off. There is no other activity.
  2. Wait 5 min, then power off.

Expected behavior Boot and go into WiFi setup .. then device registration.

Log files N/A

I'm looking for diagnostic suggestions.

I've had this Dev Kit since sometime this summer and just now had time to work on it. I created the Image file with dd on my linux (ubuntu) laptop from MarkII_arm-rpi64_Prod_012-stable_2021-08-19.img.gz after unzipping.

Thanks for your help.

ChanceNCounter commented 3 years ago

Does the screen do anything? Maybe a screen test when power is applied? I'm wondering about the ribbon connector. Just a thought.

enorberg commented 3 years ago

There is nothing shown on the screen. I visually inspected the ribbon connector and it appears properly seated on both ends. Part of my problem is I don't know what to expect to see at boot. The only thing I know about is the RPi status leds from their docs.

Thanks.

On Sun, Nov 7, 2021 at 9:30 AM ChanceNCounter @.***> wrote:

Does the screen do anything? Maybe a screen test when power is applied? I'm wondering about the ribbon connector. Just a thought.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/MycroftAI/hardware-mycroft-mark-II/issues/87#issuecomment-962631111, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBZXUIWAWQ2MN2C2NPOFTUK2LRDANCNFSM5HQLNTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

krisgesling commented 3 years ago

Hey @enorberg - sorry to hear you're having trouble.

If you're not getting any audio output after boot then I think it points to more than a display issue.

Another thing to check is that the 40 pin connector between the pi and the SJ201 is nicely fitted. It must be getting power to the pi if that has LED's lit up but if some pins aren't connecting properly it won't successfully boot.

enorberg commented 3 years ago

Hi @krisgesling, thanks for the suggestions.

There is no audio at any point - only the SJ201 led displays (circling blue, momentary all red then momentary all yellow). I did check the 40 pin connector and it is completely snugged up between the RPi and the SJ201.

The only other thing I can think of is to re-image the USB drive.

The 1st time, I simply expanded the downloaded .gz file and used dd to copy the .img to the USB.

I am going to try the other method mentioned in the instructables.com page that involves downloading the Raspberry Pi Imager app. Then I'll use that to write the USB drive in case I was wrong that both methods are equivalent.

If that doesn't fix the issue, would there be useful output from the RPi mini-hdmi port at boot time? I could disassemble the Mark-II enough to hook up a monitor if there's something to see and no other diagnostic option.

MetaGamer commented 3 years ago

If the blue circle animation on the SJ201 LEDs stops, it means the HAL is at least working. Which makes me think it's booting fine and it's a display issue. I'd double check the ribbon cable to the display- it's easy to put it in backwards. As for sound- even without a resonating chamber you should be able hear the speakers.

Do the buttons do anything? The two front buttons should make the ring-LEDs temporarily light up with the current volume, and the left switch should cause on LED to turn red to indicate that the mic is disconnected.

enorberg commented 3 years ago

Hey @MetaGamer, thanks for the suggestion. This helped a lot! ... but now another issue.

I checked the ribbon cable and questioned the SJ201 end. It "looked" correct per the pix on the instructables.com page - see screen shot below. BUT that shows the exposed conductors facing outboard so they don't make contact with the board. I reversed this and the Dev Kit now boots with both audio and video output.

However, when prompted to join the temporary Mycroft network, I couldn't find it. I saw several other neighbor's wifi networks, but no "Mycroft". - I tried this with an iPhone8, a ubuntu desktop and a MacMini but none of them could see the Mycroft network.

There was a prompt to go to start.mycroft.ai, but that site appears to be down. I'm going to try again later tonight.

Screen Shot 2021-11-08 at 13 39 29
enorberg commented 3 years ago

...oops - not the SJ201 attachment - the LCD attachment - sorry.

krisgesling commented 3 years ago

Ah, that is a good thing for us to fix up!!

Did the wifi network show up after a reboot? If it's still happening it would be good if we could dig into it to figure out what's going wrong. Do you have an ethernet connection available at all?

enorberg commented 3 years ago

The network did not show up after reboot. I do have an ethernet connection that I can connect to. Let me know what I need to do.

On Tue, Nov 9, 2021 at 12:35 AM Kris Gesling @.***> wrote:

Ah, that is a good thing for us to fix up!!

Did the wifi network show up after a reboot? If it's still happening it would be good if we could dig into it to figure out what's going wrong. Do you have an ethernet connection available at all?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MycroftAI/hardware-mycroft-mark-II/issues/87#issuecomment-963857102, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBZXWWGO6I7O7N3H3AVITULC6KPANCNFSM5HQLNTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

krisgesling commented 3 years ago

Damn, but good that you have ethernet available.

First is a short list of the network manager connections:

  1. Plug in via ethernet and get paired
  2. Add an SSH key to that device in your account
  3. SSH into the awconnect container: 
    ssh -p 8222 awconnect@$IPADDRESS
  4. Send us the output of the following command: 
    nmcli c

Second, if you’re happy to temporarily share your device logs with us it would also be great if you could

  1. login to the main mycroft container: 
    ssh -p 8222 mycroft@$IPADDRESS
  2. Get your device-id and send that to us: 
    sudo cat /pantavisor/device-id
  3. Shutdown the device and plug the usb into a computer
  4. In the pvboot partition, edit cmdline.txt change the final 
    ph_log.push=0

    to

    ph_log.push=1

    This will push your device logs to our backend, so we then need to replicate the issue:

  5. shutdown the device
  6. remove the ethernet cable
  7. plug the device back in and see if the mycroft wifi AP has been created
  8. wait at least 3-5 minutes after it’s booted to ensure it’s definitely not showing up
  9. plug in the ethernet cable to connect to the net and have the logs uploaded.
  10. run nmcli c again to see if anything has changed there.

Thanks

krisgesling commented 3 years ago

The device-id does uniquely identify your device but it's not something anyone can use against you. Even so, if you'd prefer to send things to me privately my email is kris.gesling@mycroft.ai

enorberg commented 3 years ago

Here's the result of nmcli c

@.***:~# nmcli c NAME UUID TYPE DEVICE Wired connection 1 5e9542b9-59e6-3226-a672-7530649159fc ethernet eth0

lxcbr0 3cf08c2d-bae6-4cb2-92d5-8a4e8f3283f1 bridge lxcbr0

I'll be getting the logs to you next.

On Wed, Nov 10, 2021 at 12:00 AM Kris Gesling @.***> wrote:

The device-id does uniquely identify your device but it's not something anyone can use against you. Even so, if you'd prefer to send things to me privately my email is @.***

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MycroftAI/hardware-mycroft-mark-II/issues/87#issuecomment-964812651, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBZXV6KIN5FQGTB5HR6I3ULIC5ZANCNFSM5HQLNTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

enorberg commented 3 years ago

Here's the device-id

(.venv) @.:~$ (.venv) @.:~$ sudo cat /pantavisor/device-id 618c03e4790fa6000ad81c62 (.venv) @.***:~$

On Wed, Nov 10, 2021 at 1:01 PM Eric Norberg @.***> wrote:

Here's the result of nmcli c

@.***:~# nmcli c NAME UUID TYPE DEVICE Wired connection 1 5e9542b9-59e6-3226-a672-7530649159fc ethernet eth0

lxcbr0 3cf08c2d-bae6-4cb2-92d5-8a4e8f3283f1 bridge lxcbr0

I'll be getting the logs to you next.

On Wed, Nov 10, 2021 at 12:00 AM Kris Gesling @.***> wrote:

The device-id does uniquely identify your device but it's not something anyone can use against you. Even so, if you'd prefer to send things to me privately my email is @.***

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MycroftAI/hardware-mycroft-mark-II/issues/87#issuecomment-964812651, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBZXV6KIN5FQGTB5HR6I3ULIC5ZANCNFSM5HQLNTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

enorberg commented 3 years ago

I shut down the DevKit, edited the cmdline.txt file from my ubuntu desktop removed the ethernet cable from the DevKit & repowered it. After boot up, got the "connect to Mycroft network" display - but no such network visible from iPhone nor from desktop. Waited about 5min, still no Mycroft network Replaced ethernet cable ... after a few seconds, got Connect display and later normal Mycroft screen ssh back to awconnect and ran nmcli again - here is the output: @.:~# @.:~# nmcli c NAME UUID TYPE DEVICE Wired connection 1 5e9542b9-59e6-3226-a672-7530649159fc ethernet eth0

lxcbr0 546093ca-3842-47ce-a1b0-1854d1b708ae bridge lxcbr0 @.***:~#

====== Mycroft seems to respond to normal commands - eg "Hey Mycroft, what time is it" -- etc.

On Wed, Nov 10, 2021 at 1:03 PM Eric Norberg @.***> wrote:

Here's the device-id

(.venv) @.:~$ (.venv) @.:~$ sudo cat /pantavisor/device-id 618c03e4790fa6000ad81c62 (.venv) @.***:~$

On Wed, Nov 10, 2021 at 1:01 PM Eric Norberg @.***> wrote:

Here's the result of nmcli c

@.***:~# nmcli c NAME UUID TYPE DEVICE Wired connection 1 5e9542b9-59e6-3226-a672-7530649159fc ethernet eth0

lxcbr0 3cf08c2d-bae6-4cb2-92d5-8a4e8f3283f1 bridge lxcbr0

I'll be getting the logs to you next.

On Wed, Nov 10, 2021 at 12:00 AM Kris Gesling @.***> wrote:

The device-id does uniquely identify your device but it's not something anyone can use against you. Even so, if you'd prefer to send things to me privately my email is @.***

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MycroftAI/hardware-mycroft-mark-II/issues/87#issuecomment-964812651, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBZXV6KIN5FQGTB5HR6I3ULIC5ZANCNFSM5HQLNTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

enorberg commented 3 years ago

Just to check --- I have disconnected the DevKit from power and from the ethernet cable. If it's useful for it to remain powered on and connected, I can arrange that. - Let me know.

On Wed, Nov 10, 2021 at 1:37 PM Eric Norberg @.***> wrote:

I shut down the DevKit, edited the cmdline.txt file from my ubuntu desktop removed the ethernet cable from the DevKit & repowered it. After boot up, got the "connect to Mycroft network" display - but no such network visible from iPhone nor from desktop. Waited about 5min, still no Mycroft network Replaced ethernet cable ... after a few seconds, got Connect display and later normal Mycroft screen ssh back to awconnect and ran nmcli again - here is the output: @.:~# @.:~# nmcli c NAME UUID TYPE DEVICE Wired connection 1 5e9542b9-59e6-3226-a672-7530649159fc ethernet eth0

lxcbr0 546093ca-3842-47ce-a1b0-1854d1b708ae bridge lxcbr0 @.***:~#

====== Mycroft seems to respond to normal commands - eg "Hey Mycroft, what time is it" -- etc.

On Wed, Nov 10, 2021 at 1:03 PM Eric Norberg @.***> wrote:

Here's the device-id

(.venv) @.:~$ (.venv) @.:~$ sudo cat /pantavisor/device-id 618c03e4790fa6000ad81c62 (.venv) @.***:~$

On Wed, Nov 10, 2021 at 1:01 PM Eric Norberg @.***> wrote:

Here's the result of nmcli c

@.***:~# nmcli c NAME UUID TYPE DEVICE Wired connection 1 5e9542b9-59e6-3226-a672-7530649159fc ethernet eth0

lxcbr0 3cf08c2d-bae6-4cb2-92d5-8a4e8f3283f1 bridge lxcbr0

I'll be getting the logs to you next.

On Wed, Nov 10, 2021 at 12:00 AM Kris Gesling @.***> wrote:

The device-id does uniquely identify your device but it's not something anyone can use against you. Even so, if you'd prefer to send things to me privately my email is @.***

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MycroftAI/hardware-mycroft-mark-II/issues/87#issuecomment-964812651, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBZXV6KIN5FQGTB5HR6I3ULIC5ZANCNFSM5HQLNTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

krisgesling commented 3 years ago

Hey @enorberg sorry for the radio silence - just wanted to say thanks heaps for the logs - they came through perfectly.

The team have finally been able to replicate the issue and are working on a fix. I'll hopefully have more details on what went wrong soon too.

enorberg commented 3 years ago

Glad it's replacatable (and not just me). Let me know if there's anything else I can do.

On Sun, Nov 21, 2021 at 11:14 PM Kris Gesling @.***> wrote:

Hey @enorberg https://github.com/enorberg sorry for the radio silence - just wanted to say thanks heaps for the logs - they came through perfectly.

The team have finally been able to replicate the issue and are working on a fix. I'll hopefully have more details on what went wrong soon too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MycroftAI/hardware-mycroft-mark-II/issues/87#issuecomment-975134124, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBZXXPXZG5X7SQFYNEB3DUNHGR3ANCNFSM5HQLNTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

GameGuy1900 commented 2 years ago

Is there already a solution to this problem?

enorberg commented 2 years ago

Not that I'm aware of.

On Sun, Dec 26, 2021 at 12:16 PM GameGuy1900 @.***> wrote:

Is there already a solution to this problem?

— Reply to this email directly, view it on GitHub https://github.com/MycroftAI/hardware-mycroft-mark-II/issues/87#issuecomment-1001224006, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBZXSFYZT5QT4HL2RNIX3US5LZBANCNFSM5HQLNTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

GameGuy1900 commented 2 years ago

Okay, that's too bad