Closed bdmac closed 10 years ago
FWIW here is my initializer:
PumaAutoTune.config do |config|
config.ram = Integer(ENV['AVAILABLE_MEMORY'] || 768) * 0.98 # memory * percent utilization
# Set our max_workers to what is in the ENV var + 1 because
# PumaAutoTune uses new worker count < max_workers. That means
# w/out adding 1 with a PUMA_MAX_WORKERS of 5 we would only
# ever actually run 4... seems counter-intuitive so +1.
config.max_workers = Integer(ENV['PUMA_MAX_WORKERS'] || 3) + 1
end
PumaAutoTune.start
Relevant Heroku config:
AVAILABLE_MEMORY: 512
PUMA_MAX_WORKERS: 2
Further digging implicates the get_process_mem gem even more so I will open an issue there and reference this one. Running a bash shell on Heroku and trying to run what it looks like get_process_mem would run from there results in the exact output I see in my web dyno logs above:
$ heroku run bash --remote staging
Running `bash` attached to terminal... up, run.7561
~ $ ps
PID TTY TIME CMD
2 ? 00:00:00 bash
3 ? 00:00:00 ps
~ $ ps -o pss= -p 2
ERROR: Unknown user-defined format specifier "pss".
********* simple selection ********* ********* selection by list *********
-A all processes -C by command name
-N negate selection -G by real group ID (supports names)
-a all w/ tty except session leaders -U by real user ID (supports names)
-d all except session leaders -g by session OR by effective group name
-e all processes -p by process ID
T all processes on this terminal -s processes in the sessions given
a all w/ tty, including other users -t by tty
g OBSOLETE -- DO NOT USE -u by effective user ID (supports names)
r only running processes U processes for specified users
x processes w/o controlling ttys t by tty
*********** output format ********** *********** long options ***********
-o,o user-defined -f full --Group --User --pid --cols --ppid
-j,j job control s signal --group --user --sid --rows --info
-O,O preloaded -o v virtual memory --cumulative --format --deselect
-l,l long u user-oriented --sort --tty --forest --version
-F extra full X registers --heading --no-heading --context
********* misc options *********
-V,V show version L list format codes f ASCII art forest
-m,m,-L,-T,H threads S children in sum -y change -l format
-M,Z security data c true command name -c scheduling class
-w,w wide output n numeric WCHAN,UID -H process hierarchy
I've updated get_process_mem
you can bundle update get_process_mem
to fix this. Thanks for reporting! Sorry for the trouble :(
This looks like it could be related to #2 #3 and #6 but I'm not positive.
It also seems like it may be related to the v1.0 updates in your get_process_mem gem?
Below is the log from Heroku for one of my web dynos shortly after updating from puma_worker_killer to puma_auto_tune (which included an update to get_process_mem along the way). Things started out OK until I hit a heavy page. As you can see below, I hit a few R14 memory errors (padding added around them so they stand out). The logs also include the output of log-runtime-metrics.
The memory usage reported by PumaAutoTune does not seem to match what is reported by log-runtime-metrics. The first time the R14 comes up, lrm reports 554M in use whereas the PAT stats around there report < 500M in use.
I have PAT configured with
config.ram = Integer(ENV['AVAILABLE_MEMORY'] || 768) * 0.98
andENV['AVAILABLE_MEMORY']
is set to 512.Eventually, after 2x R14 errors, PAT detects the over memory condition. At that point it freaks the hell out dumping a ton of log entries stating that it's resizing to remove a worker and current_cluster_size=2. This goes on for quite a bit during which period we see two additional R14 errors.
Eventually towards the bottom is where things get REALLY interesting as we see
2014-04-03T22:57:03.602261+00:00 app[web.2]: ERROR: Unknown user-defined format specifier "pss".
followed by what looks like help output for a linux command. That's what made me think of the changes to get_process_mem...