search

chirpstack / chirpstack-gateway-os

OpenWrt based gateway images including ChirpStack components.
https://www.chirpstack.io
130 stars 56 forks source link

Image does not boot on RPI without HDMI attached #9

Closed jeremyherbert closed 4 years ago

jeremyherbert commented 5 years ago

Hi,

I loaded this image onto the raspberry pi, but it doesn't seem to boot to ssh unless I have HDMI (or a HDMI dummy) attached to the PI.

There is also a reference to this on the forums: https://forum.loraserver.io/t/lora-gateway-os-not-booting-with-out-hdmi-or-keyboard/4126/3

brocaar commented 5 years ago

I can not reproduce this issue. My Pi boots fine without HDMI (which is how I'm using this most of the time).

cls02 commented 5 years ago

I have the same issue. Updating the Ethernet configuration as proposed in the forum post did not fix this issue for me.

brocaar commented 5 years ago

Maybe this is specific to a Raspberry Pi version that I have not tested against? Could you share the steps to reproduce this issue + Raspberry Pi version used?

but it doesn't seem to boot to ssh unless I have HDMI

Could you clarify this? Does this mean that the Raspberry Pi did get an IP address, but does not accept SSH connections?

jeremyherbert commented 5 years ago

I just burned the image onto a microsd and booted it up. Using raspberry pi 3B+

No, device does not get an IP.

cls02 commented 5 years ago

Confirm the above, also Raspberry Pi 3B+, IMST - iC880A on top, but I don't think the problem is with the concentrator boards.

jeremyherbert commented 5 years ago

Also, the RAK831 board is on top

azhaganandhan commented 5 years ago

I'm also facing the same issue ethernet is not enabled but I have not checked with HDMI. My Raspberry Configuration Raspberry Pi 3B+, RAK-831 developer kit The same issue had been posted by me in Lora server forum https://forum.loraserver.io/t/how-to-configure-gateway-using-base-os-image/4578

martinoorlandi commented 5 years ago

Same issue with a RPi 3B+ and a RAK2245 hat... Anyone has found a solution yet?

brocaar commented 5 years ago

I bought a Pi 3B+ for testing and can not reproduce the issue. My Pi (with RAK2245 shield) is only connected to ethernet + original Raspberry Pi power adapter. The SD card is flashed with https://artifacts.loraserver.io/downloads/lora-gateway-os/raspberrypi/raspberrypi3/2.0.0test4/lora-gateway-os-base-raspberrypi3--20190423145941.sdimg.gz without any modifications made.

Afterwards I run (on my laptop which is configured with a network bridge on 10.42.1.1):

sudo nmap -sP 10.42.1.1/24
Starting Nmap 7.70 ( https://nmap.org ) at 2019-07-02 17:22 CEST
Nmap scan report for 10.42.1.252
Host is up (0.00052s latency).
MAC Address: B8:27:EB:1B:7D:BF (Raspberry Pi Foundation)
Nmap scan report for 10.42.1.1
Host is up.
Nmap done: 256 IP addresses (2 hosts up) scanned in 9.60 seconds

Then I SSH to the returned IP admin@10.42.1.252 and I can login.

martinoorlandi commented 5 years ago

Thanks for your quick reply!

I've tried following your same steps, using Nmap and disconnecting everything but the power and ethernet cable. (the only difference is that I've connected the Pi to a router so that's on the same network with my MacBook).

I've tried also all the combinations of the following:

Still having the same issue! HDMI cable in and everything works fine, cable out and no ip!

This is head-scratcher!

brocaar commented 5 years ago

Could this be an issue related to power? The power-adapter seems to be the only difference that we have at this point. Is there a possibility that the HDMI plugged in has influence on the power-usage or is backfeeding power into the Pi (I'm not sure if this is even possible)?

Have you tried without the concentrator shield (to reduce the power usage)?

martinoorlandi commented 5 years ago

That’s a possibility.

I’ve tried without the concentrator but made no difference.

I’ve tried with a power adapter that is 5V - 2.1A what about yours?

fabioetec commented 5 years ago

I have the same problem, Raspberry 3B+, both with Ethernet or Wifi. ic880A Concentrator but I don't think makes a difference. I think could be relate to the order of boot of something because in some rare cases it works! Power adapter 5V - 2.5A

martinoorlandi commented 5 years ago

@fabioetec that’s interesting. Also in my case a couple of times it randomly worked.

A couple of things that may help narrow down the issue:

jeremyherbert commented 5 years ago

I have used a hdmi dummy and it solves the problem.

Is the raspberry pi firmware on the image the latest (ie the rpi foundation binary driver blobs)?

brocaar commented 5 years ago

Actually, this problem doesn't seem to be limited to LoRa Gateway OS: https://www.google.com/search?q=raspberry+pi+not+booting+without+hdmi.

What I will do at my end is update the meta-raspberrypi to the latest revision (thus latest bootstrap binary, u-boot and kernel) and create a new image. I was planning to do that anyway to create a release containing the v3 components.

martinoorlandi commented 5 years ago

Thanks for that @brocaar! That seems to explain it and I hope the updated kernel may do the trick.

I’m happy to give it a test when it’s ready.

cls02 commented 5 years ago

Ditto - great that this pesky problem might go away!

Casper

On Wed, Jul 3, 2019 at 2:05 PM martinoorlandi notifications@github.com wrote:

Thanks for that @brocaar https://github.com/brocaar! That seems to explain it and I hope the updated kernel may do the trick.

I’m happy to give it a test when it’s ready.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/brocaar/lora-gateway-os/issues/9?email_source=notifications&email_token=AAKHWQC3BCRLAI7BOAOBWK3P5SBXTA5CNFSM4HI7Y24KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZEDCZY#issuecomment-508047719, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKHWQEWZO6N3Z36EQQJQKLP5SBXTANCNFSM4HI7Y24A .

brocaar commented 5 years ago

Could one of you test this image: https://www.dropbox.com/sh/yofdcrwp6jqk4vc/AABN6IJWtb2-_eqPjITy7Vhba?dl=0&lst= ?

This:

(this is a build of https://github.com/brocaar/lora-gateway-os/tree/v3).

martinoorlandi commented 5 years ago

@brocaar Thanks for the new build. The ethernet issue seem to be solved for me!

I've tested on the same environment: RPi 3B+ with a RAK2245 concentrator, 5V 2.1A power adapter and ethernet cable plugged in (no HDMI or anything else). I've rebooted several times, tried to keep it off for a while and then back on...

Every time it boots up and connects to the network gets its IP fine, no issue at all.

I will keep an eye on it to see if it happens again, but it seem quite promising that I have not managed to reproduce the issue with the new build!

Thanks!

brocaar commented 5 years ago

That is great news @martinoorlandi and thanks for testing!

brocaar commented 5 years ago

This has been released: https://forum.loraserver.io/t/release-lora-gateway-os-v3-0-0test1/5299.

fabioetec commented 5 years ago

are you sure that it works?? for me doesn't work at all!!!

cls02 commented 5 years ago

Hi,

Works for me.

Casper

On Sun, Aug 4, 2019 at 11:24 AM fabioetec notifications@github.com wrote:

are you sure that it works?? for me doesn't work at all!!!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/brocaar/lora-gateway-os/issues/9?email_source=notifications&email_token=AAKHWQDY6BIDRIYKASHNG4TQC3YCZA5CNFSM4HI7Y24KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QD42A#issuecomment-518012520, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKHWQCQ72EAPO4J2INGARTQC3YCZANCNFSM4HI7Y24A .

splaw1k commented 4 years ago

Hello, I still have the problem, that mentioned above. My platform: Raspberry Pi 3 + RAK2245 ver. B I checked it out for the Gateway OS version 3.0 test1 (and test2 too).

brocaar commented 4 years ago

I'm finally able to reproduce the issue. It seems it is not only related to the HDMI, but also the speed of the SD card. I was only able to reproduce this using a faster / more expensive SD card. See also the above issue I have created.

brocaar commented 4 years ago

I think I have found the issue. I believe the boot fails because of the initramfs in some cases. When forcing the HDMI output, this error is unfortunately not visible. In most cases all is fine, however depending the speed of the sdcard, the block-device might not be available yet and the initramfs fails to mount the rootfs.

Somehow, connecting the HDMI affects timing of the boot.

splaw1k commented 4 years ago

So, do You have some idea to solve this issue (so that it would be works regardless of the speed of SD card) ?

brocaar commented 4 years ago

Yes, see the above commit. This will be included in the next release.

rogerdahl commented 4 years ago

Try another SD card. An older 8 GB SD card that works fine with Raspbian did not work for me with the ChirpStack full image. A newer 32 GB card works. This was with

chirpstack-gateway-os-full-raspberrypi3-20200127144146.rootfs.wic.gz
andershedberg commented 4 years ago

I see this is fixed so I will need to update, but as I could not find it when googling and I get it with the HDMI cable attached sometimes, I write a short note here if someone else needs to see it.

I use a Raspberry 3B+ with a Sandisk card detected "new high-speed SDIO card at address 0001" is detected. The error message is:

mount: mounting /dev/mmcblk0 on /mnt/root/ failed: no such file or directory

This happens about 1 time out of 4 reboots with a HDMI cable attached.

brocaar commented 4 years ago

@andershedberg actually, the HDMI is not the reason why it was failing. It was because of a race-condition in the boot process (maybe a HDMI cable changes the timing, but it is not the real fix). This was the real fix:

https://github.com/brocaar/chirpstack-gateway-os/blob/master/meta/recipes-filesystem/initramfs-readonly-rootfs-overlay/files/init-readonly-rootfs-overlay-boot.sh#L69

andershedberg commented 4 years ago

yes, my earlier solution to this was to change the display settings in the boot-configuration like in this thread https://www.raspberrypi.org/forums/viewtopic.php?t=11259 It helped, but not always. Glad it was something as simple as a 1 second sleep and the root cause was found! as the gateways caused me sleepless nights and got me into running Mikrotiks gateways this year ;-) As a side-note - I have got reports about "commercial" gateways that gets stuck in similar ways. Might be a common problem for embedded linux-systems.