search

ryansch / docker-unifi-rpi

Unifi Controller for Raspberry Pi 3+ and x86
https://hub.docker.com/r/ryansch/unifi-rpi/
347 stars 42 forks source link

High Disk IO Requests after MongoDB Upgrade #97

Closed marcogracklauer closed 8 months ago

marcogracklauer commented 2 years ago

I have been running the controller for years on a 32 bit RPI4 with docker-compose. After reinstalling to 64 bit and the new version I have a lot of problems with disk IO requests.

The Raspberry Pi gets stuck and only a reboot helps. I tried to increase the memory option to 2 GB but see the same problem.

I have two access points, two switches and the USG in my controller. So not a really big environment.

controller-io
glichten1 commented 2 years ago

Same Problem here

ryansch commented 2 years ago

@marcogracklauer What do the mongodb logs look like?

Edit: How much memory do you have on your pi? Is this swap usage?

mmims commented 1 year ago

I came to report the same. I had not tracked it down to mongodb, but I have noticed high memory usage which I believe would lead to swap and disk IO.

My Pi 4 only has 1GB of RAM. Is there a reason 1024M was chosen for max heap allocation? I'm going to reduce it to 768M to see if that helps.

marcogracklauer commented 1 year ago

@marcogracklauer What do the mongodb logs look like?

Edit: How much memory do you have on your pi? Is this swap usage?

sorry for my delayed answer. I've deactived the swap to reduce any impact caused by the ssd. My pi has 4 GB.

Is there a way to save the mongodb logs to the host disk?
After the latest pull and start my system gets stucked immediately (maybe because of an migration...?)

Edit: I've increased the Swap to 1 GB. Now it seems to be better. But I need to run it for a while to check if its stable. Where do I find the mongo-Logs? I can't find it in the container...

marcogracklauer commented 1 year ago

@marcogracklauer What do the mongodb logs look like? Edit: I've increased the Swap to 1 GB. Now it seems to be better. But I need to run it for a while to check if its stable. Where do I find the mongo-Logs? I can't find it in the container...

20:54:15 up 23 days, 22:33, 1 user, load average: 0,37, 0,32, 0,31

It seems to be very important that you have a big swap. No restart in more than 23 days.

ryansch commented 1 year ago

That would make sense. In the move to 64 bit stuff, we're going to use more memory and the 1GB model is pretty small.

marcogracklauer commented 1 year ago
ram

Didn't think RAM would be the problem, since the workload was always about that even before the SWAP increase.

Maybe you can add a note to the how to.

johnboiles commented 1 year ago

Just updated to the 64bit version after several years running fine on the 32 bit version. I'm on a Pi3 w/ 1GB ram. Runs fine initially, then dies within 12-24hrs of boot. It acts to me like an OOM (system gets really slow in ssh, then ultimately becomes unresponsive (though it will respond to pings just fine!). Maybe I'll set up memory logging so i can watch memory increasing. The closest I've gotten is in top, I can see java using 579MB RAM before things die.

Nothing else is running on that pi. Previously I ran several applications on the Pi and didn't have an issue. Is it possible the recent version of the controller just has a good old fashioned memory leak?

Edit: Ah I guess the default after 7.0 is to configure java to use 1024MB RAM (instead of 512MB). I don't think this will work with a Pi 3B. I opened issue #100 to discuss.

ryansch commented 8 months ago

Honestly I don't see a way to support older pis at this point.