search

YunoHost-Apps / mastodon_ynh

Free, open-source social network for YunoHost
https://joinmastodon.org/
GNU Affero General Public License v3.0
84 stars 36 forks source link

One-click upgrade failure #363

Closed yelvington closed 1 year ago

yelvington commented 1 year ago

One-click upgrade from 4.0.2~ynh1 to 4.0.2~ynh2 proceeded as far as "Now upgrading Mastodon" and then errored out:

YunoHost encountered an unexpected error Really sorry about that. You should look for help on the forum or the chat to fix the situation, or report the bug on the bugtracker. The following information might be useful for the person helping you: Error: "504"

Action: "PUT" /yunohost/api/apps/mastodon/upgrade

Error message:

504 Gateway Time-out

nginx

It appears that "chmod -R -o-rwx /var/www/mastodon" ran for quite some time after the error, then disappeared from the process table, and I noticed "whiptail --inputbox Which user would you like to rename: 20 60" is running as root.

Hardware: Raspberry Pi 4, 4 gig memory + 4G swap. Plenty of disk space.

isAAAc commented 1 year ago

same here on a vps trying to upgrade in cli at this moment

isAAAc commented 1 year ago

for now, in cli mode, it fails here because of lack of free space for the backup :/

Info: DEBUG - 3447588 Not enough space at /home/yunohost.backup/archives (free: 103297626112 / needed: 140266659192)
Info: DEBUG - 3447636 The operation 'Create a backup archive' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20230201-133702-backup_create' to get help
Info: DEBUG - 3447660 action [1182869.1] executed in 3447.475s
Info: WARNING - 3447662 Not enough free space on '/home/yunohost.backup/archives'

i deleted all backups, and just restarted the upgrade, keep in touch

isAAAc commented 1 year ago

@yelvington , the upgrade is still on his way at this moment in cli mode, as explained in https://github.com/YunoHost-Apps/mastodon_ynh/issues/351 , upgrading mastodon can be very (very very) long, and perhaps gives a time-out in webadmin mode, i'll post again when it finished ;) EDIT: not enough free space again ... more than 130G needed just for this backup ...

yelvington commented 1 year ago

A running Mastodon instance uses an insane amount of disk space, and the backup-before-upgrade script doesn't do any sanity checking on availability.

In my case, though, I did not run out of disk. Not sure what killed the upgrade. I was able to manually go in and re-precompile Mastodon and get running again.

I am very tempted to chuck out the whole idea of this being integrated with Yunohost and just do a straight upgrade to the current release candidate. What would I be losing? I have no interest in the LDAP integration.

jcrabapple commented 1 year ago

A running Mastodon instance uses an insane amount of disk space, and the backup-before-upgrade script doesn't do any sanity checking on availability.

In my case, though, I did not run out of disk. Not sure what killed the upgrade. I was able to manually go in and re-precompile Mastodon and get running again.

I am very tempted to chuck out the whole idea of this being integrated with Yunohost and just do a straight upgrade to the current release candidate. What would I be losing? I have no interest in the LDAP integration.

Can a straight Mastodon upgrade be done to the Yunohost version without going through the Yunohost updater?

isAAAc commented 1 year ago

damn, i just realize mastodon disapeared from my server ... no backup in /home/yunohost.backup i just have this (and the folder is empty ... :( )

/home/yunohost.backup/tmp# ls -al
total 12
drwxr-x--- 3 admin root   4096 Feb 10 19:48 .
drwxrwx--- 5 root  admins 4096 Mar 29  2020 ..
drwxrwxrwx 2 root  root   4096 Feb  9 01:13 mastodon-pre-upgrade1

i'm affraid i lost everything :(

jcrabapple commented 1 year ago

How does one get Mastodon off of the Yunohost release channel and onto the mainline Mastodon release channel?

isAAAc commented 1 year ago

ok , i think the upgrade script removed my mastodon instead of reinstall it from backup after the last upgrade fail:

https://paste.yunohost.org/raw/epojihumol the remove, https://paste.yunohost.org/raw/inizomitew the failed upgrade (parent operation)

i don't understand why it removed everyting, and without any local backup, i think everything is lost :/

alexAubin commented 1 year ago

Hmkay, isn't there a "mastodon-pre-upgrade1" backup ? YunoHost was in the process of restoring the archive but your log stops at "Extracting needed files from the archive..."

alexAubin commented 1 year ago

no backup in /home/yunohost.backup i just have this (and the folder is empty ... :( ) /home/yunohost.backup/tmp

Where exactly did you look ... Your message mentions /home/yunohost.backup/tmp which is only a tmp location, not the actual place for backups, which is /home/yunohost.backup/archives/

isAAAc commented 1 year ago

no mastodon backup in the folder: 

root@krashboyz:/home/yunohost.backup/archives# ls -al
total 374064
drwxrwx--- 2 root admins      4096 Feb 10 19:32 .
drwxrwx--- 5 root admins      4096 Mar 29  2020 ..
-rw-r--r-- 1 root root         306 Feb 10 19:32 invidious-pre-upgrade2.info.json
lrwxrwxrwx 1 root root          60 Feb 10 19:32 invidious-pre-upgrade2.tar -> /home/yunohost.backup/archives/invidious-pre-upgrade2.tar.gz
-rw-r--r-- 1 root root    93486632 Feb 10 19:32 invidious-pre-upgrade2.tar.gz
-rw-r--r-- 1 root root         340 Feb 10 19:14 nextcloud-pre-upgrade1.info.json
lrwxrwxrwx 1 root root          60 Feb 10 19:14 nextcloud-pre-upgrade1.tar -> /home/yunohost.backup/archives/nextcloud-pre-upgrade1.tar.gz
-rw-r--r-- 1 root root   289518608 Feb 10 19:14 nextcloud-pre-upgrade1.tar.gz

how does it happen with fediverse if i reinstall mastodon with the same fqdn ?

alexAubin commented 1 year ago

The backup was clearly created during the upgrade as can be seen in the log, and the script double-check that it existed (cf ynh_backup_archive_exists mastodon-pre-upgrade1 in the log), so there's more to the story, I don't understand how it magically disappeared since then ...

isAAAc commented 1 year ago

neither me ^^ no mail recieved telling mastodon was removed ...

alexAubin commented 1 year ago

Are there more related operations available in the logs ? (i mean the webadmin > Tools > Logs)

isAAAc commented 1 year ago

I was very surprised because I was sleeping at that time (for once),

but I found the right log in the webadmin area: https://paste.yunohost.org/raw/yizewovori

operation started at 18:24 (and i wasn't sleeping ^^)

edit: we are migrating the server via drbd, I had to delete the backups to free up some space without realising that the application had not restored itself. This is the only logical explanation I can find.

isAAAc commented 1 year ago

i reinstalled from scratch. case closed on my side.