1.65 update failed

[xabirequejo]

Hello,

I just tried to update synapse from 1.59 to 1.65 and it has failed. Synapse is completely gone and I cannot restore the pre-upgrade backup.
My YunoHost server

Hardware: Raspberry Pi 4 at home YunoHost version: 11.0.9.12 I have access to my server: Through SSH and through the webadmin Are you in a special context or did you perform some particular tweaking on your YunoHost instance? Yes, I modified dphys-swapfile to use up to 8GB of swap.

Description of my issue

• I was running Synapse 1.59 and saw there was an update available. I clicked on it (webadmin) and it started the process but it failed (log).
• Synapse was completely gone — new version didn’t install and the backup wasn’t restored (log).
• I usually remove the webapp before updating synapse to avoid conflicts, but I forgot to do it this time (matrix’s identity is my main domain name, where the main website is hosted using my_webapp).
• I removed the my_webapp and tried to restore the backup but it didn’t work either (log).
• I realised that there was a new user on the server: synapse, that must've been created automatically when trying to upgrade to 1.65. I removed the user and tried restoring synapse 1.59 but it failed once again.

I have two synapse instances on two subdomains (I only tried to update one of them, the other one still runs 1.59 and is fine), I don't know if this is relevant, but I wonder if it is possible to have two users with the same name (synapse, one for each instance) or if this is/will be an issue.

Thanks
— Originally published on the forum

[xabirequejo]

I dared to try to upgrade my 2nd instance, but this time I made sure I removed my_webapp before the attempt, and everything went fine. 1.59 > 1.65 without issues.

This is what I keep getting when I try to restore the backup for the 1st instance:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/yunohost/backup.py", line 1453, in _restore_app
    permission_create(
  File "/usr/lib/python3/dist-packages/yunohost/log.py", line 419, in func_wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/yunohost/permission.py", line 410, in permission_create
    raise YunohostValidationError("permission_already_exist", permission=permission)
yunohost.utils.error.YunohostValidationError: 'synapse.main' permission already exists

[SiM]

To force the removal of these permissions, type the following commands:

sudo yunohost tools shell -c "from yunohost.permission import permission_delete; permission_delete('synapse.main', force=True)"
sudo yunohost tools shell -c "from yunohost.permission import permission_delete; permission_delete('synapse.server_api', force=True)"
sudo yunohost tools shell -c "from yunohost.permission import permission_delete; permission_delete('synapse.server_client_infos', force=True)"

Then you should be able to restore your synapse backup (if the webapp on same domain is still removed)

[xabirequejo]

Thanks for getting back to me.

1. I executed the commands and after that I was able to restore the backup (1.59).
2. When I tried to upgrade to 1.65 it failed because fail2ban wasn't running.
3. I restarted it manually and I attempted to upgrade synapse. This time it failed because user synapse existed, so I removed it and tried again.
4. The upgrade failed once more because fail2ban was down again. After restarting it I tried to upgrade synapse and this time it worked.

I'm sharing my fail2ban log here just in case it can be of use. Otherwise this ticket can be closed.

Thank you for your help

[shukon]

My instance also failed upgrading and restoring, restoring failed with 2022-09-07 17:18:44,951: DEBUG - Sep 07 17:18:44 python[29324]: AttributeError: module '\''lib'\'' has no attribute '\''X509_V_FLAG_CB_ISSUER_CHECK'\'' Has anyone observed that error yet?

[yajo]

Also happening here. These are my upgrade logs: https://paste.yunohost.org/raw/hoyajehase

Also these are the relevant fail2ban logs:

Oct 26 07:58:42 systemd[1]: Stopping Fail2Ban Service...
Oct 26 07:58:44 fail2ban-client[18885]: Shutdown successful
Oct 26 07:58:44 systemd[1]: fail2ban.service: Succeeded.
Oct 26 07:58:44 systemd[1]: Stopped Fail2Ban Service.
Oct 26 07:58:44 systemd[1]: Starting Fail2Ban Service...
Oct 26 07:58:44 systemd[1]: Started Fail2Ban Service.
Oct 26 07:58:46 fail2ban-server[18888]: 2022-10-26 07:58:46,203 fail2ban                [18888]: ERROR   Failed during configuration: Have not found any log file for synapse jail
Oct 26 07:58:46 fail2ban-server[18888]: 2022-10-26 07:58:46,207 fail2ban                [18888]: ERROR   Async configuration of server failed
Oct 26 07:58:46 systemd[1]: fail2ban.service: Main process exited, code=exited, status=255/EXCEPTION
Oct 26 07:58:46 systemd[1]: fail2ban.service: Failed with result 'exit-code'.
Oct 26 08:03:55 systemd[1]: fail2ban.service: Unit cannot be reloaded because it is inactive.
Oct 26 08:03:56 systemd[1]: fail2ban.service: Unit cannot be reloaded because it is inactive.
Oct 26 08:03:57 systemd[1]: fail2ban.service: Unit cannot be reloaded because it is inactive.

The process was:

1. fail2ban working
2. upgrading synapse, fail2ban fails
3. after synapse upgrade finishes its rollback, fail2ban can be started again

That makes me think synapse is doing something that breaks fail2ban, which in turn breaks synapse upgrade

I already repeated the process twice, and always happens the same.

Luckily, in my case the rollback worked fine. Unluckily, mautrix_telegram is dead now.

[Josue-T]

Also happening here. These are my upgrade logs: https://paste.yunohost.org/raw/hoyajehase

Also these are the relevant fail2ban logs:

Oct 26 07:58:42 systemd[1]: Stopping Fail2Ban Service...
Oct 26 07:58:44 fail2ban-client[18885]: Shutdown successful
Oct 26 07:58:44 systemd[1]: fail2ban.service: Succeeded.
Oct 26 07:58:44 systemd[1]: Stopped Fail2Ban Service.
Oct 26 07:58:44 systemd[1]: Starting Fail2Ban Service...
Oct 26 07:58:44 systemd[1]: Started Fail2Ban Service.
Oct 26 07:58:46 fail2ban-server[18888]: 2022-10-26 07:58:46,203 fail2ban                [18888]: ERROR   Failed during configuration: Have not found any log file for synapse jail
Oct 26 07:58:46 fail2ban-server[18888]: 2022-10-26 07:58:46,207 fail2ban                [18888]: ERROR   Async configuration of server failed
Oct 26 07:58:46 systemd[1]: fail2ban.service: Main process exited, code=exited, status=255/EXCEPTION
Oct 26 07:58:46 systemd[1]: fail2ban.service: Failed with result 'exit-code'.
Oct 26 08:03:55 systemd[1]: fail2ban.service: Unit cannot be reloaded because it is inactive.
Oct 26 08:03:56 systemd[1]: fail2ban.service: Unit cannot be reloaded because it is inactive.
Oct 26 08:03:57 systemd[1]: fail2ban.service: Unit cannot be reloaded because it is inactive.

The process was:

1. fail2ban working

2. upgrading synapse, fail2ban fails

3. after synapse upgrade finishes its rollback, fail2ban can be started again

That makes me think synapse is doing something that breaks fail2ban, which in turn breaks synapse upgrade

I already repeated the process twice, and always happens the same.

Luckily, in my case the rollback worked fine. Unluckily, mautrix_telegram is dead now.

No link to fail2ban. Did you already create a yunohost user synapse on your instance manually ? If yes you must remove it to fix your issue because the app need a user synapse.

[Josue-T]

My instance also failed upgrading and restoring, restoring failed with 2022-09-07 17:18:44,951: DEBUG - Sep 07 17:18:44 python[29324]: AttributeError: module '\''lib'\'' has no attribute '\''X509_V_FLAG_CB_ISSUER_CHECK'\'' Has anyone observed that error yet?

Please create a other issue with the full log because it could be not the same error.

[Josue-T]

Look like that the original issue was fixed here so closing this issue.

[yajo]

No link to fail2ban. Did you already create a yunohost user synapse on your instance manually ? If yes you must remove it to fix your issue because the app need a user synapse.

I didn't.

There's a matrix-synapse user created by yunohost, but no synapse user.

[Josue-T]

No link to fail2ban. Did you already create a yunohost user synapse on your instance manually ? If yes you must remove it to fix your issue because the app need a user synapse.

I didn't.

There's a matrix-synapse user created by yunohost, but no synapse user.

Ok but in yunohost user (in LDAP) not a system user

[yajo]

Thanks, indeed there was a synapse yunohost user. I don't know why it was there. Maybe remainders of an old broken update... Deleting it fixed the update!