Update breaks installation

[JMans15]

Describe the bug

I'm using version 25.0.2 on a vps alog with easy-openvpn-server for some time now, when the automatic refresh hit with version 25.0.3, my nextcoud instance stopped working I investigated the logs and saw that nextcloud.nextcloud-fixer was "Waiting for Nextcloud to be installed" nextcloud.nextcloud-cron also logged that "Nextcloud or one of the apps require upgrade", and suggested that I use occ to update I tried to do so but occ wasn't working either, see network configuration logs below I tried to disable the other snap to no avail Reverting the update did allow me to continue using nextcloud though

To Reproduce

Steps to reproduce the behavior:

1. Install this snap, version 25.0.2 (and easy-openvpn-server maybe?)
2. Enjoy your very own nexcloud instance
3. Refresh to version 25.0.3

Expected behavior

It should continue running as before

Screenshots

Message on my website page

OS/snapd/snap version

Running debian 11, snap & snapd version 2.58

$ snap list nextcloud
Name       Version      Rev    Tracking       Publisher   Notes
nextcloud  25.0.3snap2  33587  latest/stable  nextcloud✓  -

$ snap version
snap    2.58
snapd   2.58
series  16
debian  11
kernel  5.10.0-21-cloud-amd64

Logs

See attached file, sorry for the inconvenience but I seem to be limited to 65536 characters while the result of the script is 389746 characters long

[kyrofa]

See attached file, sorry for the inconvenience

Oh not at all-- I'm happy to fetch attachments. I appreciate you bothering to grab the logs at all!

This seems problematic:

Feb 04 20:56:52 vps-44f0858e nextcloud.apache[9450]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80

You have something claiming the port Apache needs, and it's not able to properly update Nextcloud. Any idea what that is?

[JMans15]

Oh yeah I remind seeing this but forgot about it, my mistake I only have an openvpn server running on the machine but I changed the ports for http and https while nextcloud runs on the default ones, disabling it (it's also a snap) didn't change anything So I don't know what it could be, netstat -pant tells that something is listening on ::80 but doesn't show a PID I'm wondering if this could be my ssh connection but the problem first appeared because of an auto-refresh so there wasn't an ssh connection, and version 25.0.2 still works

[kyrofa]

Huh. Can I see that netstat output?

[JMans15]

Nevermind that was just me being stupid, I reverted to 25.0.2 and this was simply httpd working as expected netstat now doesn't show anything using port 80 (my ssh connection takes port 443 though)

[kyrofa]

Can you run sudo snap disable nextcloud and reboot, and then run netstat and check port 80 again? Obviously give it a minute for all the services to fire up.

[kyrofa]

my ssh connection takes port 443 though

I assume you haven't enabled HTTPS in the Nextcloud snap, then?

[JMans15]

my ssh connection takes port 443 though

I assume you haven't enabled HTTPS in the Nextcloud snap, then?

I did actually, and it did work with the older version

[kyrofa]

I did actually, and it did work with the older version

Any chance you're visiting Nextcloud and SSHing in on different interfaces? Maybe you're SSHing over the VPN interface, while Nextcloud is listening on the public interface?

[JMans15]

I did actually, and it did work with the older version

Any chance you're visiting Nextcloud and SSHing in on different interfaces? Maybe you're SSHing over the VPN interface, while Nextcloud is listening on the public interface?

Sorry I'm not familiar with interfaces, what do you mean? My vpn is disabled though

Can you run sudo snap disable nextcloud and reboot, and then run netstat and check port 80 again? Obviously give it a minute for all the services to fire up.

Here is the output, apache seems to be running while nextcloud is still deactivated, this doesn't seem right

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:51368           0.0.0.0:*               LISTEN      693/sshd: /usr/sbin
tcp        0    372 SERVER_IP:51368      MY_IP:56348   ESTABLISHED 811/sshd: debian [p
tcp6       0      0 :::51368                :::*                    LISTEN      693/sshd: /usr/sbin
tcp6       0      0 :::80                   :::*                    LISTEN      696/apache2

[kyrofa]

That implies that you have apache installed from the system. Run dpkg -l | grep apache.

[JMans15]

Here's the output, I may have installed it along with certbot now that I see this, smh

ii  apache2                           2.4.54-1~deb11u1               amd64        Apache HTTP Server
ii  apache2-bin                       2.4.54-1~deb11u1               amd64        Apache HTTP Server (modules and other binary files)
ii  apache2-data                      2.4.54-1~deb11u1               all          Apache HTTP Server (common files)
ii  apache2-utils                     2.4.54-1~deb11u1               amd64        Apache HTTP Server (utility programs for web servers)
ii  libapache2-mod-php7.4             7.4.33-1+deb11u1               amd64        server-side, HTML-embedded scripting language (Apache 2 module)
ii  python3-certbot-apache            1.10.1-1                       all          Apache plugin for Certbot

(I don't even use certbot, I used the enable https script provided with nextcloud-snap)

[kyrofa]

Yeah, that'll be a problem. Ports are an exclusive resource per interface. You can't have both the system apache and nextcloud listening on port 80. You'll need to uninstall the former, or change ports for one of them so they don't clash.

To preempt your next question: this sometimes works for you because port claims are a race condition: whichever one fires up first gets it. Sometimes it will be apache, sometimes it will be the snap.

[JMans15]

I understand the problem, I don't use it anyway (and forgot about it) so I removed and purged everything related to it, rebooted and enabled nextcloud, but this didn't solve the problem :/

Feb 04 20:56:52 vps-44f0858e nextcloud.apache[9450]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80

I noted that this occurred a week ago but doesn't seem to have since, and I did try some things last week so it's possible that I ran into this error because of my shenanigans but then rolled back

httpd is now broadcasting on port 80 as expected though

[kyrofa]

Did you try disabling the snap and rebooting again? It's possible you have yet another thing wanting that port! Nginx for bonus points? :stuck_out_tongue:

[JMans15]

Yes I did, disabled everything (except core and core18), rebooted, watched top and netstat to see if anything suspicious was starting up, enabled nextcloud but ran into "Waiting for nextcloud to be installed" again

[JMans15]

I noted that it calls nextcloud.occ while starting up and that it fails with the same error as when I called it manually, any chance that something didn't update right?

[kyrofa]

Oh we've already established that is hasn't updated right! Ignore that for now, hopefully it will resolve itself once we get to the bottom of the rest of the issue.

This is also very interesting, from your PHP logs:

Feb 04 20:55:31 vps-44f0858e nextcloud.php-fpm[4184]: cannot locate base snap core18: No such file or directory

I'm not quite sure what to make of that. Can I see the output of snap list?

[kyrofa]

but this didn't solve the problem :/

Feb 04 20:56:52 vps-44f0858e nextcloud.apache[9450]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80

I misunderstood this, by the way: you weren't saying that the port was still claimed (this is a log message from days ago), you're just saying that Nextcloud is still busted. I'm with you now. Port problem is gone, at least.

[JMans15]

I tried to disable everything but the core last week and forgot that core18 was needed, so I re-enabled it right away and it shouldn't be an issue anymore

[JMans15]

I'm with you now. Port problem is gone, at least.

Sorry I wasn't clear, I wasn't sure the port wasn't a problem myself

[kyrofa]

Alright, can I see the output of snap list nextcloud --all, please?

[JMans15]

Here it is :

Name       Version      Rev    Tracking       Publisher   Notes
nextcloud  25.0.2snap1  33054  latest/stable  nextcloud✓  disabled
nextcloud  25.0.3snap2  33587  latest/stable  nextcloud✓  -

[kyrofa]

Alright I propose reverting back to 25.0.2snap1, and confirming it's still in good working order. Then blow away the broken one: sudo snap remove nextcloud --revision 33587. Once that is done, refresh again to start clean (i.e. with the state from 25.0.2snap1), and I suspect the upgrade will go through: sudo snap refresh nextcloud.

[JMans15]

I reverted and removed the latest revision, everything was still working, I refreshed from scratch and ran into "Waiting for nextcloud to be installed" again

Here are the new logs if it helps

[kyrofa]

Do you have collabora, richdocumentscode, those kind of apps enabled?

[JMans15]

If you mean application/plugins installed through nextcloud, here is the output of nextcloud.occ app:list with version 25.0.2 enabled :

Nextcloud is in maintenance mode, hence the database isn't accessible.
Cannot perform any command except 'maintenance:mode --off'

Enabled:
  - activity: 2.17.0
  - calendar: 4.2.3
  - cfg_share_links: 3.0.2
  - cloud_federation_api: 1.8.0
  - comments: 1.15.0
  - contactsinteraction: 1.6.0
  - dashboard: 7.5.0
  - dav: 1.24.0
  - deck: 1.8.3
  - drawio: 2.1.0
  - encryption: 2.13.0
  - external: 5.0.0
  - externalportal: 1.0.2
  - federatedfilesharing: 1.15.0
  - federation: 1.15.0
  - files: 1.20.1
  - files_external: 1.17.0
  - files_pdfviewer: 2.6.0
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - fileslibreofficeedit: 1.1.0
  - firstrunwizard: 2.14.0
  - integration_github: 1.0.15
  - integration_onedrive: 1.1.4
  - integration_twitter: 1.0.3
  - integration_youtube: 0.1.2
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - mail: 2.2.3
  - metadata: 0.17.0
  - nextcloud_announcements: 1.14.0
  - notes: 4.6.0
  - notifications: 2.13.1
  - oauth2: 1.13.0
  - onlyoffice: 7.6.8
  - password_policy: 1.15.0
  - photos: 2.0.1
  - privacy: 1.9.0
  - provisioning_api: 1.15.0
  - recommendations: 1.4.0
  - related_resources: 1.0.3
  - serverinfo: 1.15.0
  - settings: 1.7.0
  - sharebymail: 1.15.0
  - spreed: 15.0.3
  - support: 1.8.0
  - survey_client: 1.13.0
  - systemtags: 1.15.0
  - text: 3.6.0
  - theming: 2.0.1
  - twofactor_backupcodes: 1.14.0
  - twofactor_totp: 7.0.0
  - unsplash: 2.1.1
  - user_status: 1.5.0
  - viewer: 1.9.0
  - weather_status: 1.5.0
  - welcome: 1.0.8
  - workflowengine: 2.7.0
Disabled:
  - admin_audit
  - bruteforcesettings
  - circles: 25.0.0
  - end_to_end_encryption: 1.11.1
  - hibp: 0.1.0
  - integration_nuiteq: 1.0.3
  - officeonline: 1.1.4
  - richdocuments: 7.1.0
  - suspicious_login
  - user_ldap

[kyrofa]

Huh, a closer look at your error:

"file":"/var/snap/nextcloud/33587/nextcloud/extra-apps/files_external_gdrive/vendor/guzzlehttp/guzzle/src/Client.php"

That app is ancient and abandoned. I suggest reverting again, blowing away the new rev, then try moving that old app out of the way and trying the refresh once more.

[JMans15]

smh, I will never learn to read the warnings

Thanks for the help and (very) quick response! People like you are the reason why the open web will survive

[kyrofa]

Ah good, glad we got to the bottom of it!

Now that it's working again, I will take one quick opportunity to suggest that you reconsider your combination of technologies. You should think about using Ubuntu, or potentially migrating away from the snap. We're generally happy to help in any way we can, but we want to make sure you're aware that you're not in a great situation right now, security-wise.

[JMans15]

I actually thought about it already, I was looking for a way of moving to docker without having to setup everything again and laziness won against security considerations

[kyrofa]

Excellent. You're welcome here as long as you want to be here :) .