search

Screenly / Anthias

The world's most popular open source digital signage project.
https://anthias.screenly.io
Other
2.49k stars 618 forks source link

Pi 3B Bullseye: Anthias update hangs at "TASK [system : Remove libc6-dev]" #1962

Open traeu opened 3 months ago

traeu commented 3 months ago

Overview of the Issue

I use multiple Anthias-installations. Normally, I don't update them regularly when everything works. Recently, I tried to update two of my screens with latest anthias, but I experience problems. My Pis run on Raspberry Pi OS Bullseye, all apt updates are installed. I tried to start the update-process by running the install-script again, but the script freezes at "TASK [system : Remove libc6-dev]". While it freezes, I noticed that the system is lagging in general, for example, another ssh-session responds only delayed while the install-script runs in another ssh-session. A bit strange: Last thursday, I only got this problem on one of my Pis, not on both. Both run bullseye, both had all apt updates made. But one would run through this libc6-dev task quickly, while the other one hangs at it. I thought, I will try my luck again on monday. Now today, I try to run the install-scripts on both Pis again, and now on both Pis the install-script freezes at this libc6 remove task.

Reproduction Steps

Environment

nicomiguelino commented 3 months ago

@traeu, I'll look into this issue closely. Is your device still operational even after you halt the update. In the meantime, if it's still working, you can download backup files via the web UI (Settings > Get Backup).

nicomiguelino commented 3 months ago

@traeu I tried reproducing the issue, tweaked things so that I can install an older version of Anthias (48e427b to be specific). The installation went fine. Then I ran bash <(curl -sL https://install-anthias.srly.io/). Mine was not stuck at the libc6-dev part. It could be that yours was way older.

If it wouldn't be a hassle on your part, you can backup the data first, then re-install Anthias. Alternatively, you can comment out those steps (see https://github.com/Screenly/Anthias/blob/master/ansible/roles/system/tasks/main.yml#L180-L191 for details) in the Pi, then run the installer script again.

traeu commented 3 months ago

hello @nicomiguelino, thank you for your quick response! Anthias seems to be operational after the script hangs and I forced it to quit with ctrl-c. I also tried to run the update-container script after my failed upgrade-attempt, that also worked successfully. I also tried to install/re-install and also remove this libc6-dev package with apt before running the script. But that made no difference.

Is there a way how I can make a fresh Anthias installation and remove an existing installation, without reinstalling the whole Raspberry Pi OS? Wiping the SD card and doing a clean install would be my next option to try, I was hoping that I can avoid it... But no big deal, especially as this looks like a problem on my side and not like a common problem.

What would be the best way to modify the main.yml locally before executing the installer script? Can I somehow pause the installer script after it downloaded all files, then modify it, and then resume the install-script?

Thanks for your help!

nicomiguelino commented 3 months ago

@traeu I have only tried the following once yesterday:

https://github.com/Screenly/Anthias/blob/5bb412319f19a4814b3f472b182ba370362eb1ea/ansible/roles/system/tasks/main.yml#L180-L191

I hope this helps.

traeu commented 3 months ago

Today I choose the easiest solution and wiped all SD cards, re-installed Raspberry Pi OS (legacy/bullseye 64bit lite) and made fresh installs of Anthias. As expected, everything works perfectly. Before wiping, I tried upgrading one of my other Pis. Installed apt updates and re-run the anthias install script. This also worked without problems. That must mean, that two of my Pis had some other problems, and in general, upgrading works also for older installations. But while I was at it, I didn't want to risk anything and I made fresh installs on all Pis, even if upgrading may would have worked perfectly for all my Pis except for the two that made trouble with this libc6 package.

I guess I can close this issue now, it was clearly some problem on my side, even if I will never know why exactly it failed...

Thanks again for your quick help and your work on this amazing software!

Edit: I also tried to comment out the removal and installation of libc6-dev in main.yml. After that, installation/upgrading also worked, the script was able to run until it was done, no freezes. But if I enabled this removal and installation of libc6-dev again in main.yml after a first successful run while it was commented out, the same problems appeared again. Because I just didn't want to permanently use modified local files that are not in sync with your files on github, I choose to made fresh installs of everything.

kroam commented 2 weeks ago

Hey @traeu what file did you edit to comment out the libc6-dev lines? I have run the bash <(curl -sL https://install-anthias.srly.io/) on my Raspberry PI Zero 2 but it failed at ["TASK [system : Remove libc6-dev]. I then edited /home/pi/screenly/ansible/roles/system/tasks/main.yml and commented out the lines for removing and installing libc6-dev and saved the file. Then i ran ~/screenly/bin/install.sh but the installation still hangs at remove libc6-dev also after running the install.sh the file main.yml is back to its original state.

nicomiguelino commented 2 weeks ago

@kroam, you can comment out the following lines inside ansible/roles/system/tasks/main.yml:

https://github.com/Screenly/Anthias/blob/5bb412319f19a4814b3f472b182ba370362eb1ea/ansible/roles/system/tasks/main.yml#L180-L191

traeu commented 2 weeks ago

@kroam later, I found out by accident what the cause of my problem was. I used the option "force_turbo=1" in config.txt, because only with this I could get 4K output to work on my Raspi 3B. But this unofficial hack seems to come with unexpected downsides. Thats why after reinstalling (and using stock config.txt at first) the problem was gone and installation was successful. Later, after I tested 4K again and noticed that the problem reappeared, I found out that this turbo-setting was causing the installation process to hang at this remove libc6-dev.

nicomiguelino commented 2 weeks ago

@traeu, does removing force_turbo=1 fixes the issue where you're stuck at removing libc6-dev?

traeu commented 2 weeks ago

@nicomiguelino yes for me this worked. As soon as I commented out this line in the config file (and rebooted of course), the installation process did not hang anymore at this step of removing libc6-dev.

nicomiguelino commented 2 weeks ago

@traeu, @kroam, thanks. I'm still testing the upgrade process, given the details that you provided. Don't hesitate to fork Anthias and create a pull request if you feel like it and are comfortable with the changes.

kroam commented 1 week ago

On my Raspberry PI Zero 2 i don't use the "force_turbo=1" in config.txt. its a stock Raspberry Pi OS 64bit Bookworm

kroam commented 1 week ago

Maybe a dumb question but when i git clone anthias to my pi and change the main.yml to comment out the lines for removing and installing libc6-dev, then run the /bin/install.sh file, the installer still hangs at remove libc6-dev. Am i missing something?

nicomiguelino commented 1 week ago

@kroam, no, it's not a dumb question. Glad you asked. The script always pulls the latest changes from the master branch, causing your changes to be overwritten. Here's what you can do:

I hope it helps. If not, please let me know.