Fetching JSON or any other API hits data returns 500 Internal Server Error

[D00Movenok]

Hi there! I've been using your awesome project for quite a long time.

A few days ago I noticed that my "visitors" badge is not working. After taking a closer look I discovered that all the API endpoints (https://hits.dwyl.com/D00Movenok/D00Movenok, https://hits.dwyl.com/D00Movenok/D00Movenok.json, https://hits.dwyl.com/D00Movenok/D00Movenok.svg) return an Internal Server Error (500).

Seems that something was broken on the backend.

[FurryR]

I have the same issue. It affected to my projects; is there anything I can do to help?

[bearsyankees]

Same!

[nelsonic]

Sadly, the PostgreSQL on Fly.io is not reliable. Zero warning when running out of disk space. Requires some time to recover ... 🙃

[LuchoTurtle]

@nelsonic the only way to fix this is to increase disk space? Don't have access to the logs (for some reason)

[nelsonic]

The volume doesn't appear to be full ... the only one that is used is 4,045 MB of 10GB ... https://fly.io/apps/hits-db/volumes

@LuchoTurtle you are admin. Feel free to poke around non-destructively and see what can be done. 🙏

[LuchoTurtle]

The logs clearly state that the problem is with DB connection. It is currently suspended.

Can I just start it? @nelsonic

[nelsonic]

If it's just a case of booting the DB instance, go for it and monitor closely. But I suspect it might need a boost in RAM. Which will tip us over into the next cost bracket... let's try and see what happens.

[LuchoTurtle]

Just started it. By looking at the logs, the DB seems to be working properly as well.

https://hits.dwyl.com/dwyl/hits.svg?style=flat-square

The hits are looking good as well. If it stops again, we should be able to receive an e-mail of memory issues. It's running on only 564mb of RAM, after all.

Tho it's worth noting that one of the checks is in "critical condition"

vm | critical | 500 Internal Server Error [✓] checkDisk: 5.98 GB (61.3%!)(MISSING) free space on /data/ (389.19µs) [✓] checkLoad: load averages: 0.01 0.02 0.08 (75.37µs) [✓] memory: system spent 132ms of the last 60s waiting on memory (338.17µs) [✓] cpu: system spent 750ms of the last 60s waiting on cpu (34.77µs) [✗] io: system spent 3.28s of the last 10 seconds waiting on io (28.62µs) | 2024-03-02 00:57:43

[LuchoTurtle]

It's been roughly an hour since deploying and it seems to be stable.

https://fly.io/apps/hits-db/activity

I'm closing this issue. If anything happens, it can be reopened.

[nelsonic]

Thanks @LuchoTurtle 🙏