search

stashapp / stash

An organizer for your porn, written in Go. Documentation: https://docs.stashapp.cc
https://stashapp.cc/
GNU Affero General Public License v3.0
8.8k stars 778 forks source link

Log messages can consume an excessive amount of system memory #3194

Open cj12312021 opened 1 year ago

cj12312021 commented 1 year ago

Describe the bug Some users, including myself, have reported that Stash consumes much of their system memory when a log file is not specified. Without a specified log file, Stash will write the messages to stdout, which can be the system memory in the docker container.

Expected behavior We need to either stop logging to stdout in docker containers or document that the default behavior can lead to excessive memory usage.

PhuriousGeorge commented 1 year ago

Removing all my comments, as my observations aren't an issue

WithoutPants commented 1 year ago

This looks related to #2195

cj12312021 commented 1 year ago

It's possible his issue was due to the log messages being written to system memory. However, what throws me off is that he mentioned that he was able to free up ram by restarting his container. The behavior may be different on a Synology system, but from my experience in my Unraid system, restarting the container doesn't clear the log messages. I had to reinstall the container to free up the memory consumed by its log files. Either way, since both issues are related to excessive memory usage, this issue could be merged into his, and we could include the info I have here about the log problem.

Flashy78 commented 1 year ago

I'm on Unraid and have never had an issue with memory leaks in the container. I haven't changed any of the Stash logging settings from default, so I'm not using a log file. My current container has a 3 week uptime, with around 500 scenes added in that time, and it just hovers between 500-600mb memory. I do my phash/preview generation as part of the scan task, so I don't use generate, which sounds like the issue may be in the other bug report.

When you click the Container Size button in Unraid, is Stash your only container with a runaway log size? Mine is currently at 2.6mb. While I have another container that uses Python basic logging config, which I believe outputs to StdErr, which is only at 0.75mb despite also running for 3 weeks with tons more logging than Stash. Looks like Stash is also outputting to StdErr https://github.com/stashapp/stash/blob/develop/internal/log/logger.go#L49 , afaik this isn't a known "never do this in a Docker container" thing.

In Stash what about changing your logging level to warn or error? How about trying to run your docker with --log-opt max-size=50m? Are you running the latest Unraid? My in my docker settings, my Docker is v20.10.18 and Docker LOG rotation is Enabled.

WithoutPants commented 1 year ago

Have we verified that the problem is eliminated when stash is configured to log to a file (or /dev/null) instead? If that is the case, I think we should set to log to /root/.stash/stash.log by default in docker environments. We can document why as well.

cj12312021 commented 1 year ago

Yes, specifying a log file did resolve the issue. Setting the log file by default sounds like a good solution here.

@Flashy78, my custom Stash scrapper, is the reason I am getting so many log messages. The scrapper outputs a lot for debugging purposes(I do plan on trimming this down). So the drastic growth I'm seeing others wouldn't see unless they are doing something similar with a scrapper.

Flashy78 commented 1 year ago

Back in the day I definitely had issues with container logs filling up my docker.img, so I must have enabled log rotation whenever it became available in Unraid Docker (since I can see it enabled now), which explains why I never have issues anymore with containers with runaway logging.

I much prefer ephemeral logging in my containers vs writing to my SSD, but I understand that's a personal preference.

As long as it also still outputs to the container log so I don't have to tail a log file to view the logs, then that works for me.. although I guess it would have to ONLY log to the file in order to actually solve your issue. And I would just have to update my config once to disable log files?

cj12312021 commented 1 year ago

It's possible that docker log rotation on Unriad has been fixed since I initially ran into this issue. I recall reading that it was not working as intended.

I guess it would have to ONLY log to the file in order to actually solve your issue. And I would just have to update my config once to disable log files?

Yes, it would have to only log to the file to address the issue. With the update, you should just be able to disable the log file.