502 Bad Gateway - used to be able to access apps, now can't - just nginx warning

[chegeiser]

I used to be able to access Nextcloud and Notes at nextcloud.mydomain.com and notes.mydomain.com. And two days ago they were no longer available - I just get the 502 error. In the Syncloud dashboard the apps are shown as active apps. (I'm using an Odroid XU4)

When I go to the apps addresses (notes.mydomain.com) and check using the developer tools in the browser it shows the address is still a secure https connection - so nothing wrong on my domain end (and I made no changes at this time either).

I've tried re-activating at my domain - that worked in that the activation held, but I still have the 502 error when accessing the apps. I tried deleting apps and re-adding them, still the 502 error.

The nginx_internal_error.log found at /var/snap/platform/common/log/ contains nothing.

I did find this in /var/snap/platform/common/log/nginx_public_error.log: 2019/04/29 18:45:10 [error] 20499#0: *10558 connect() to unix:/var/snap/nextcloud/common/web.socket failed (111: Connection refused) while connecting to upstream, client: 174.222.27.138, server: ~^(?P<app>.*)\.cebaxo\.com$, request: "GET / HTTP/2.0", upstream: "http://unix:/var/snap/nextcloud/common/web.socket:/", host: "nextcloud.cebaxo.com"

I just sent logs to support - hopefully it went through.

Update: I ran sudo snap info nextcloud and got:

summary:   Nextcloud file sharing
publisher: syncloud
license:   unknown
description: |
  Nextcloud
commands:
  - nextcloud.cron
  - nextcloud.occ
  - nextcloud.php
  - nextcloud.psql
services:
  nextcloud.nginx:      simple, enabled, inactive
  nextcloud.php-fpm:    forking, enabled, inactive
  nextcloud.postgresql: forking, enabled, inactive
snap-id:   nextcloud.190211412
tracking:  stable
refreshed: 2019-02-11T20:29:58Z
installed: 190211412 (190211412) 99MB - 

Since it showed nextcloud as inactive, I ran sudo snap start nextcloud - that worked, it is now up and running! So I then tried restarting the server and Notes and Nextcloud and they are both running after the reboot - without me having to run 'sudo snap start ...'

Any idea what might have gone wrong here?

Thanks, Che

[cyberb]

You are running out of disk space on the root device.

2019/04/26 05:26:22 [crit] 1054#0: *24562 pwritev() "/var/snap/nextcloud/common/nginx/fastcgi_temp/0000000985" failed (28: No space left on device) while 

I guess you are running on sd card or emmc and it has 16Gb and is full.

df

Filesystem     1K-blocks     Used Available Use% Mounted on
udev              952168        0    952168   0% /dev
tmpfs             204240    22020    182220  11% /run
/dev/mmcblk0p2  14837196 14060592    127040 100% /
tmpfs            1021188        0   1021188   0% /dev/shm
tmpfs               5120        0      5120   0% /run/lock
tmpfs            1021188        0   1021188   0% /sys/fs/cgroup
/dev/loop1         42368    42368         0 100% /snap/platform/181109762
/dev/loop2         97408    97408         0 100% /snap/nextcloud/190211412
/dev/loop3         27520    27520         0 100% /snap/notes/19040763
/dev/loop5         27008    27008         0 100% /snap/notes/19040552
/dev/sda1      961272872 14717916 897702084   2% /opt/disk/external
tmpfs             204240        0    204240   0% /run/user/0

I would suggest checking what is eating space with this command:

du -h --max-depth=1 / 

If you installed apps and stored many files before activating external disk it would have saved them on your internal disk.

Did you extend boot disk from storage settings?

[chegeiser]

That seems to have done the trick. I had first added Nextcloud, then added the external drive. There were some files in /opt/disk/internal/nextcloud so I deleted that folder (and /opt/disk/internal/notes).

Thank you!