search

OpenMediaVault-Plugin-Developers / installScript

script that installs omv-extras and openmediavault if not installed already.
1.07k stars 206 forks source link

Raspberry Pi 4b Installation bootloop #71

Closed illmaren closed 2 years ago

illmaren commented 2 years ago

Hardware: Fresh and new Raspberry 4b 8gb 32gb SD Card from Sandisk

Software: https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2021-11-08/ Flashed with Rufus 3.8.1580

What happens? After 

sudo apt update
sudo apt upgrade
wget https://github.com/OpenMediaVault-Plugin-Developers/installScript/raw/master/install
chmod +x install
sudo ./install -n

the Raspberry is stuck in a bootloop and Raspbian needs to be newly flashed to the SDCard

ryecoaaron commented 2 years ago

I would need the output from running the script to help. The script doesn’t do anything with the boot to cause a loop though. Is the SD card new?

illmaren commented 2 years ago

The SD Card is not old but not new. It was only used to help installing rasbian to a m.2 usb drive to another raspberry.

Right now the SD Card is hosting the vanilla rasbian without problems. Last thing in the script was something about applying configurations. Would need to restart the setup too see what exactly was the last.

Edit:/ As i'm still testing with this paticular raspberry I just started the installation new. Still a vanilla installation of rasbian 64bit lite.

ryecoaaron commented 2 years ago

The script applies many different salt modules near the end. But as I said, it doesn’t do anything with boot. And being on stable on vanilla Debian has far less going on than OMV installs. You could try the -f flag to skip installing the flashmemory plugin but I would run OmV for long on an SD card without it. If that fixes it, something strange is going on. There are hundreds of RPi users using the script and I have used it hundreds of times myself (have six rpi4 running right now).

illmaren commented 2 years ago

Strange things happen here for sure thats save... Running the Raspberry Pi on a Macbook 2020 Powerplug...so that should be more than enough power so no power outtages...

last thing happening before the script crashes and the raspberry goes into a bootloop

Creating configuration database ...
Setting up Salt environment ...
Setting up system ...
[WARNING ] /usr/lib/python3/dist-packages/salt/utils/files.py:396: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  f_handle = open(*args, **kwargs)  # pylint: disable=resource-leakage

[WARNING ] /usr/lib/python3/dist-packages/salt/utils/files.py:396: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  f_handle = open(*args, **kwargs)  # pylint: disable=resource-leakage

Deploying service configurations ...
jae-hyup commented 2 years ago

I got same issue. Both installation from SD card and USB drive has same issue. I was able to replicate this issue with 4GB pi and 8GB pi.

illmaren commented 2 years ago

Here a gist with the complete output: https://gist.github.com/illmaren/c332b923adaff680b94c70ae2342dc8c

jae-hyup commented 2 years ago

The script applies many different salt modules near the end. But as I said, it doesn’t do anything with boot. And being on stable on vanilla Debian has far less going on than OMV installs. You could try the -f flag to skip installing the flashmemory plugin but I would run OmV for long on an SD card without it. If that fixes it, something strange is going on. There are hundreds of RPi users using the script and I have used it hundreds of times myself (have six rpi4 running right now).

I retried with -f option and it crashed again. I was using this script a couple of days ago with same pi and memory and there was no issue. This issue is only observed starting today. Unfortunately, I was migrating OMV to this machine and wiped out old version.

ryecoaaron commented 2 years ago

I didn’t realize 6.0.9 had been released. This could be caused by the new systemd watchdog code. I will have to setup a test system.

ryecoaaron commented 2 years ago

The problem is happening (on two of my boards) here - https://github.com/openmediavault/openmediavault/blob/master/deb/openmediavault/debian/openmediavault.postinst#L212. Even when I removed the watchdog code from the install script, it still does the same thing. And I'm not home now. So, I can't look at the console. I will have to work on this more when I get home.

illmaren commented 2 years ago

Atleast you could confirm the problem.

ryecoaaron commented 2 years ago

I temporarily changed the script to install OMV 6.0.8 on RPis. If someone wants to test a fresh install using that, it shouldn't have any problems. I would curious to see if upgrading to 6.0.9 breaks the system when installing 6.0.8 first.

illmaren commented 2 years ago

Nope its not working

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Version '6.0.8' for 'openmediavault' was not found
failed to install openmediavault package.
ryecoaaron commented 2 years ago

completely different problem. I just typed the version number wrong. Fixed.

illmaren commented 2 years ago

still not a valid version number.

E: Version '6.0.8-1' for 'openmediavault' was not found

edit:/ I get only one version display of the debian package...I'll get the other repo...

ryecoaaron commented 2 years ago

The repos have changed to not include old files. I have temporarily disabled install on RPis running bullseye. Not much I can do until I can get home to fix this unless someone else figures it out.

flmaxey commented 2 years ago

I attempted to replicate the issue on an R-PI4, 2GB. After upgrading to 6.0.9-1 and rebooting, the login went into a loop. SSH was fine.

However, when clearing Firefox's cache (Cntrl+shift+R), the problem cleared. Using another browser, or the same browser in privacy mode is an indicator that cache needs to be cleared.

Edit: After running sudo omv-salt deploy run watchdog and rebooting, I have what appears to be a boot loop.

NeoTrace82 commented 2 years ago

I attempted to replicate the issue on an R-PI4, 2GB. After upgrading to 6.0.9-1 and rebooting, the login went into a boot loop. SSH was fine.

However, when clearing Firefox's cache (Cntrl+shift+R), the problem cleared. Using another browser, or the same browser in privacy mode is an indicator that cache needs to be cleared.

Edit: After running sudo omv-salt deploy run watchdog and rebooting, I have what appears to be a boot loop.

so this happens to existing installations that will update too? just did update, so should not at all reboot my pi?!

ryecoaaron commented 2 years ago

Until I know what the problem is, I wouldn't reboot your RPi if you need it. We are talking hours not weeks here.

NeoTrace82 commented 2 years ago

no worries i hardly reboot my pi but i have a testing one that i sometime do will avoid that one too till it is fixed ;)

AaveryG commented 2 years ago

Is it possible to switch the pi to running buster so version 5.x is used instead?

ryecoaaron commented 2 years ago

Is it possible to switch the pi to running buster so version 5.x is used instead?

Yep, download the legacy lite image.

ryecoaaron commented 2 years ago

Pretty sure the default of 5 mins is too much for the RPi watchdog. I need to test but I am going to try setting RuntimeWatchdogSec=10 in /etc/systemd/system.conf.d/openmediavault-watchdog.conf

because I see this message hundreds of times in syslog Jan 18 11:17:08 rpi4 systemd[1]: Failed to set timeout to 300s: Invalid argument

ryecoaaron commented 2 years ago

https://github.com/OpenMediaVault-Plugin-Developers/installScript/commit/f561189c6132427fb1dcef627666c89b2b0e78dc

ryecoaaron commented 2 years ago

To fix an existing system for upgrading or after upgrade but before reboot: sudo omv-env set "OMV_WATCHDOG_SYSTEMD_RUNTIMEWATCHDOGSEC" 14 sudo omv-salt stage run prepare sudo omv-salt deploy run watchdog

ryecoaaron commented 2 years ago

I don't recommend taking the time higher than 14 secs. It boot looped with 30 or 40.

ryecoaaron commented 2 years ago

I fixed both of my RPis by putting the SD card in my laptop (xubuntu) and mount the second partition. I think changed the value in /etc/systemd/system.conf.d/openmediavault-watchdog.conf. I ran the omv-* commands after they booted.

jae-hyup commented 2 years ago

I fixed both of my RPis by putting the SD card in my laptop (xubuntu) and mount the second partition. I think changed the value in /etc/systemd/system.conf.d/openmediavault-watchdog.conf. I ran the omv-* commands after they booted.

Unfortunately, install script is still setting watchdog configuration to 5min.

ryecoaaron commented 2 years ago

What is the output of: grep WATCH /etc/default/openmediavault

ryecoaaron commented 2 years ago

I did ask Volker to change the OMV default itself to 14 secs.

jae-hyup commented 2 years ago

Will the fix be released as 6.1.0?

ryecoaaron commented 2 years ago

Does the version matter? It will probably be 6.0.10.

jae-hyup commented 2 years ago

Does the version matter? It will probably be 6.0.10.

Nope. Just wondering new version to be released or workaround to be offered.

ryecoaaron commented 2 years ago

I am trying to fix the script. What workaround do you need? The script has always set a timeout for the rpi watchdog. It just changed how it is done. Can you get the output of the grep command?

ryecoaaron commented 2 years ago

I just realize the script is setting the watchdog after the openmediavault package is installed. But the installation of the openmediavault package breaks things. I will have to see if I can set it beforehand. Otherwise, the script will have to wait for 6.0.10.

jae-hyup commented 2 years ago

I am trying to fix the script. What workaround do you need? The script has always set a timeout for the rpi watchdog. It just changed how it is done. Can you get the output of the grep command?

grep command returns nothing.

ryecoaaron commented 2 years ago

Your system is evidently not being detected as an RPi. What is the output of: cat /proc/device-tree/model

jae-hyup commented 2 years ago

Your system is evidently not being detected as an RPi. What is the output of: cat /proc/device-tree/model

Raspberry Pi 4 Model B Rev 1.4

jae-hyup commented 2 years ago

FYI

I have managed installation following below step.

  1. Created /etc/default/openmediavault file with below contents before execute install script. OMV_WATCHDOG_SYSTEMD_RUNTIMEWATCHDOGSEC="14" OMV_WATCHDOG_DEFAULT_MODULE="bcm2835_wdt" OMV_WATCHDOG_CONF_WATCHDOG_TIMEOUT="14"

  2. execute sudo install -n ( ( if omit "-n" option, it broke eth0 interface. )

  3. at below step, type D to display difference and manually merge all other OMV_ configurations.

Configuration file '/etc/default/openmediavault' ==> File on system created by you or by a script. ==> File also in package provided by package maintainer. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version.

Add below contents in /etc/default/openmediavault and type "N" OMV_CONFIG_FILE="/etc/openmediavault/config.xml" OMV_CONFIG_TEMPLATE_FILE="/usr/share/openmediavault/templates/config.xml" OMV_PRODUCTINFO_FILE="/usr/share/openmediavault/productinfo.xml" ...

  1. installation was successful.
ryecoaaron commented 2 years ago

This would only help on a fresh install. Anyone upgrading from pre-6.0.9 would be broken. Luckily, Volker has added the needed code and I tested. It is working but not released yet - https://github.com/openmediavault/openmediavault/pull/1187

thizmo commented 2 years ago

Hi,

Your system is evidently not being detected as an RPi. What is the output of: cat /proc/device-tree/model

Raspberry Pi 4 Model B Rev 1.4 i am having the same trouble today. Yet with an Bananapi on freshly deployed armbian bullseye 21.08. Can/should i try to use the same variables above as @jae-hyup did?

Or just wait for the new version to be realesed. And then test it. ;-)

frozenBranch commented 2 years ago

FYI

I have managed installation following below step.

  1. Created /etc/default/openmediavault file with below contents before execute install script. OMV_WATCHDOG_SYSTEMD_RUNTIMEWATCHDOGSEC="14" OMV_WATCHDOG_DEFAULT_MODULE="bcm2835_wdt" OMV_WATCHDOG_CONF_WATCHDOG_TIMEOUT="14"
  2. execute sudo install -n ( ( if omit "-n" option, it broke eth0 interface. )
  3. at below step, type D to display difference and manually merge all other OMV_ configurations.

Configuration file '/etc/default/openmediavault' ==> File on system created by you or by a script. ==> File also in package provided by package maintainer. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version.

Add below contents in /etc/default/openmediavault and type "N" OMV_CONFIG_FILE="/etc/openmediavault/config.xml" OMV_CONFIG_TEMPLATE_FILE="/usr/share/openmediavault/templates/config.xml" OMV_PRODUCTINFO_FILE="/usr/share/openmediavault/productinfo.xml" ...

  1. installation was successful.

Hi, would you tell me which version of raspberry pi os did you install?

illmaren commented 2 years ago

The Workaround is not really needed as ryecoaaron only needs to remove the code which disabled the omv6 installation on Raspis.

openmediavault/openmediavault#1187 is merged

ryecoaaron commented 2 years ago

The script exit has been removed. The world can move on now.

ryecoaaron commented 2 years ago

Just being merged did not mean I could remove the code. I had to wait until the package was in the repos.

thizmo commented 2 years ago

Hi,

sorry to report that the fix does not work on an Banana Pi with Armbian Bullseye (freshly installed).

Tried it right now, it still seems to be hanging at this point:

Created symlink /etc/systemd/system/multi-user.target.wants/openmediavault-beep-up.service → /lib/systemd/system/openmediavault-beep-up.service.
Created symlink /etc/systemd/system/reboot.target.wants/openmediavault-beep-down.service → /lib/systemd/system/openmediavault-beep-down.service.
Created symlink /etc/systemd/system/halt.target.wants/openmediavault-beep-down.service → /lib/systemd/system/openmediavault-beep-down.service.
Created symlink /etc/systemd/system/poweroff.target.wants/openmediavault-beep-down.service → /lib/systemd/system/openmediavault-beep-down.service.
Created symlink /etc/systemd/system/multi-user.target.wants/openmediavault-cleanup-monit.service → /lib/systemd/system/openmediavault-cleanup-monit.service.
Created symlink /etc/systemd/system/multi-user.target.wants/openmediavault-cleanup-php.service → /lib/systemd/system/openmediavault-cleanup-php.service.
Created symlink /etc/systemd/system/multi-user.target.wants/openmediavault-engined.service → /lib/systemd/system/openmediavault-engined.service.
Created symlink /etc/systemd/system/getty.target.wants/openmediavault-issue.service → /lib/systemd/system/openmediavault-issue.service.
Creating configuration database ...
Setting up Salt environment ...
Setting up system ...
[WARNING ] /usr/lib/python3/dist-packages/salt/utils/files.py:396: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  f_handle = open(*args, **kwargs)  # pylint: disable=resource-leakage

[WARNING ] /usr/lib/python3/dist-packages/salt/utils/files.py:396: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  f_handle = open(*args, **kwargs)  # pylint: disable=resource-leakage

Deploying service configurations ...

` Even used the "-n option".

But there is no connection (eth0) any more left. All other SSH-sessions are gone / lost connection.

ryecoaaron commented 2 years ago

@thizmo The fix was only targeted at the RPi. We did not know it caused the same problem on other arm boards. I guess this will have to apply generically to all arm boards.

ryecoaaron commented 2 years ago

You should have no problem installing on Armbian Buster though.

thizmo commented 2 years ago

You should have no problem installing on Armbian Buster though.

OK, will try that for now. See it if it is working. Thx!

mihailovsboriss commented 2 years ago

Thanks guys for working on this script, great work. However I still run into problems while using it.

Installed on fresh copy of bullseye-arm64 on Raspberry Pi 4B. Using -n flag openmediavault finally installed. However after updating the system though OMV updater menu it went offline and is not accessible anymore.

Tried installing also without -n flag, then it goes offline during installation and never comes back.

clothlust commented 2 years ago

Thanks guys for working on this script, great work. However I still run into problems while using it.

Installed on fresh copy of bullseye-arm64 on Raspberry Pi 4B. Using -n flag openmediavault finally installed. However after updating the system though OMV updater menu it went offline and is not accessible anymore.

Tried installing also without -n flag, then it goes offline during installation and never comes back.

same, im experiencing the same issue, will be testing sudo omv-firstaid and see if it solves the issue. Do update me if it solves your problem