Backend Issue - Internal Server Error

[mreid-tt]

Hi @Diaoul @ymartin59, I hope you're well. Our back-end server has been consistently reporting "Internal Server Error" for the past day, affecting users. While I don't have access to the back-end server, I'm here to assist in any way I can. Your expertise in resolving this matter would be greatly appreciated.

[szyb]

Synopackage website has been also affected with Internal Server Error when trying to get packages from https://packages.synocommunity.com. First error was recorded at 2023-11-29 13:27:51.3559 CET

[hgy59]

Synopackage website has been also affected with Internal Server Error when trying to get packages from https://packages.synocommunity.com. First error was recorded at 2023-11-29 13:27:51.3559 CET

https://packages.synocommunity.com is not a website, this is the package server API and the resulting error in DSM Package Center (using the API) is "Invalid Location".

[szyb]

Yes, I know. By "synopackage website" I meant synopackage.com - the search engine. It is affected by this error. Sorry for the confusion

[gpion]

The error seems to depend on the internet provider (weird!) Free : KO Orange mobile: OK

[szyb]

@gpion I have similar feelings. My phisical device (DS718+ @DSM7.1.1) can add the repository to Package Center, while in ohter location and provider it is Internal Server Error.

[publicarray]

@hgy59 @mreid-tt @Diaoul The disk is full No space left on device. I reclamed some space with rm but it's still not working. I think the file system needs to be repaired.

[publicarray]

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           384M  384K  384M   1% /run
/dev/sda1        79G   76G     0 100% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           384M     0  384M   0% /run/user/1003

[Diaoul]

Ah shit, thanks for looking into it :pray: Keep us posted!

[mreid-tt]

@publicarray, I noticed the backend is online as of a few minutes ago. Really appreciate you jumping in on this. Could you please update me on whether the reported issue has been resolved, or if work is still in progress?

[publicarray]

@Diaoul It's up and running @hgy59 can you check if we need 4 homeassitant versions so we can cleanup some disk space? https://synocommunity.com/package/homeassistant @Diaoul we should think about moving the storage to something else if possable or increase the drive space.

$ du -h /etc/synocommunity/data | sort -rh | head -25
64G /etc/synocommunity/data
2.8G    /etc/synocommunity/data/homeassistant
2.6G    /etc/synocommunity/data/mono
2.1G    /etc/synocommunity/data/synocli-devel
1.9G    /etc/synocommunity/data/python311
1.9G    /etc/synocommunity/data/owncloud
1.8G    /etc/synocommunity/data/erlang
1.8G    /etc/synocommunity/data/domoticz
1.8G    /etc/synocommunity/data/bazarr
1.7G    /etc/synocommunity/data/java-11-openjdk/1
1.7G    /etc/synocommunity/data/java-11-openjdk
1.6G    /etc/synocommunity/data/sickchill
1.4G    /etc/synocommunity/data/radarr
1.4G    /etc/synocommunity/data/python310
1.2G    /etc/synocommunity/data/sonarr
1.1G    /etc/synocommunity/data/tvheadend
1.1G    /etc/synocommunity/data/synocli-disk
1.1G    /etc/synocommunity/data/synocli-devel/3
1.1G    /etc/synocommunity/data/ombi
1.1G    /etc/synocommunity/data/mono/20
1.1G    /etc/synocommunity/data/imagemagick
984M    /etc/synocommunity/data/flexget
[...]

[publicarray]

To get the server running again I removed some older versions for jellyfin, synocli-files and bazzar

[mreid-tt]

@publicarray, if you need to clear additional space, I would volunteer every previous version of ownCloud. The current version resolves a number of issues with previous builds and the oldest versions are not upgradeable to the newer builds.

[hgy59]

To get the server running again I removed some older versions for jellyfin, synocli-files and bazzar

Thanks mate Additionally I removed all the python3 packages (Python 3.7 and older).

[hgy59]

@mreid-tt, @th0ma7 shall I delete the ffsync packages now?

• the single version is v1.5.2-3 BETA for DSM 5.x and published long ago (2015-09-16).

This will allow us to start from scratch with mozilla sync server (https://github.com/SynoCommunity/spksrc/pull/5942) by keeping the package name "ffsync".

[th0ma7]

I don't mind for ffsync. But can't we just buy more storage space?

[th0ma7]

Adding to my previous comment, there are multiple updates to ffmpeg + all that serie coming up. More storage space is really something to look for. @Diaoul have we got a few spare coins from sponsors that could help?

[Diaoul]

I don't know if I can upgrade storage on this very entry-level hosting :grimacing: I'd rather move away it and go for something more robust but that means ops work and potentially some dev as well for which I don't really have the time...

[Diaoul]

Double the storage space for an additional price of 8.80 €/month, i.e. a total of 21.30 €/month

:exploding_head: that seems excessive nowadays for 80GB...

[th0ma7]

There may be other options ... cheaper 1tb hdd vps. Or looking into getting a free 5tb one drive that gets map on our existing vps.

[th0ma7]

interesting thread... https://www.reddit.com/r/selfhosted/comments/16ijs7j/cheap_vps_with_1tb_of_space/

[mreid-tt]

@publicarray, I've published a new version of ownCloud, can you remove the oldest v10.13.1-11?

[publicarray]

Done! FYI I just noticed Jellyfin has a security update as well https://github.com/jellyfin/jellyfin/releases/tag/v10.8.13 Edit: and It's already published! good work!

[mreid-tt]

@publicarray, thanks for that. Regarding this ticket, is it now considered resolved? Also, if low disk space was the only issue, what mitigations can we put in place to avoid in the future? Perhaps a simple bot to alert on the #repo-dev discord when disk space is running low?