search

YunoHost-Apps / synapse_ynh

Matrix server (synapse) package for YunoHost
https://matrix.org/
GNU General Public License v3.0
79 stars 43 forks source link

Failed to upgrade to 1.92.3~ynh1 #414

Closed tio-trom closed 9 months ago

tio-trom commented 9 months ago 
ended_at: 2023-09-28 20:36:39.425090
env:
  BASH_XTRACEFD: '7'
  NO_BACKUP_UPGRADE: '1'
  YNH_APP_ACTION: upgrade
  YNH_APP_BASEDIR: /var/cache/yunohost/app_tmp_work_dirs/app_fzykotli
  YNH_APP_CURRENT_VERSION: 1.91.0~ynh1
  YNH_APP_ID: synapse
  YNH_APP_INSTANCE_NAME: synapse
  YNH_APP_INSTANCE_NUMBER: '1'
  YNH_APP_MANIFEST_VERSION: 1.92.3~ynh1
  YNH_APP_PACKAGING_FORMAT: '1.0'
  YNH_APP_UPGRADE_TYPE: UPGRADE_APP
  YNH_ARCH: amd64
  YNH_CWD: /var/cache/yunohost/app_tmp_work_dirs/app_fzykotli/scripts
  YNH_DEBIAN_VERSION: bullseye
  YNH_INTERFACE: cli
  YNH_STDRETURN: /tmp/tmp6gn89a9g/stdreturn
error: An error occurred inside the app upgrade script
interface: cli
operation: app_upgrade
parent: null
related_to:
- - app
  - synapse
started_at: 2023-09-28 20:36:10.319537
success: false
yunohost_version: 11.2.4

Full log here https://paste.yunohost.org/raw/satajenofu

hieronymousch commented 9 months ago

same here and AGAIN it removes everything after a failed upgrade and is unable to restore. I really don't understand why it deletes everything.

hieronymousch commented 9 months ago

Temporary fix, based on https://forum.yunohost.org/t/solved-synapse-update-failed-restore-not-possible/22888/5

cd /home/yunohost.backup/archives/
ls -l synapse-pre-upgrade* # This is to double check the name of the backup, adjust the file name accordingly later, and take the latest
cp synapse-pre-upgrade2.tar synapse-pre-upgrade2.tar.bak # Make a backup of the existing backup before modifying, just in case
tar --delete --file synapse-pre-upgrade2.tar apps/synapse/backup/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/setuptools_rust apps/synapse/backup/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/setuptools_rust-1.7.0.dist-info
yunohost backup restore synapse-pre-upgrade2

Not sure why this rust python became mandatory, as I understood things, this is not really necessary for Synapse to run.

I was able to successfully restore the backup after this.

hieronymousch commented 9 months ago

Upgrade to testing (1.93) worked fine sudo yunohost app upgrade synapse -u https://github.com/YunoHost-Apps/synapse_ynh/tree/testing --debug

tio-trom commented 9 months ago

Ok thank you very much for the info. So the testing works then? I cannot try it these days but as soon as I will I can close this issue.

hieronymousch commented 9 months ago

To be clear, did a restore as shown above and then upgraded to testing

MayeulC commented 9 months ago

Thanks a lot @hieronymousch, I had the same issue, your fix worked. I just added --app synapse to the last command: yunohost backup restore synapse-pre-upgrade2 --app synapse.

tio-trom commented 9 months ago

Why do you have to also add --app synapse at the end?

MayeulC commented 9 months ago

Oh, it's not needed, it's just because yunohost backups can contain multiple apps, and the restore command otherwise shows a scary warning.

Restoring the database took about 4 hours and 15 minutes for my 40 GiB backup, I think a lot of that time was spent on index creation. As I don't want to wait such a long time again in case of failure, I'll also upgrade to testing.

Edit: BTW, you can keep an eye on the restore process by running sudo -u postgres psql -c "SELECT pg_database_size('matrix_synapse') ;" or maybe SELECT pg_size_pretty( pg_database_size('matrix_synapse') ); if you want a prettier output. My 40 GiB backup expanded to about 70 GiB.

hieronymousch commented 9 months ago

Oh, it's not needed, it's just because yunohost backups can contain multiple apps, and the restore command otherwise shows a scary warning.

Restoring the database took about 4 hours and 15 minutes for my 40 GiB backup, I think a lot of that time was spent on index creation. As I don't want to wait such a long time again in case of failure, I'll also upgrade to testing.

Thats a huge DB. I hope it's not for a familiy server because some cleaning might then be in order ;) (I'm at 3.8 GB for small familiy server but 5+ years of use)

tio-trom commented 9 months ago

My database was 260GB after 2 years of use and 500 users. I managed to optimize it and bring it down to 100GB. Read more here as to why this is so big and how I managed to make it much smaller https://github.com/YunoHost-Apps/synapse_ynh/issues/354

So yeah for me the restore process even backup, is not feasible. I usually do a server snapshot before updating Synapse without backing it up, then if it fails I restore a snapshot for the entire server....else it is not manageable.

hieronymousch commented 9 months ago

I got pretty paranoid after the upgrade to bullseye and now I have a remote backup running with borg (iterative backup) and at the same time pgbackrest for incremental DB commits as well... but luckily not the volume you talk about... interesting stuff ;)

tio-trom commented 9 months ago

Ok so now I finally tested it. My database is actually 60GB and files around 15GB.

Restoring the backup worked thanks to the https://github.com/YunoHost-Apps/synapse_ynh/issues/414#issuecomment-1741831776 comment. Backing up now takes around 25-30 minutes. Restoring takes around 40-50 minutes.

Upgrading to testing also works with sudo yunohost app upgrade synapse -u https://github.com/YunoHost-Apps/synapse_ynh/tree/testing --debug and that took around 30 minutes with backup.

It is not terrible but it is something you have to do via the terminal. In case Matrix fails, then you will have to spend around 1h restoring it, if your instance is of a similar size.

I hope they will push the testing into the master soon to fix this terrible issue.