For each runnable job we have several estimates of its future WSS:
a) the project-supplied value rsc_memory_bound
b) APP_VERSION::max_working_set_size:
the max measured WSS of jobs using this app version
since the client started (not saved to state file)
c) if the job has already run, ACTIVE_TASK::working_set_size_smoothed:
recent average (on the order of 1 min) of its WSS
Current policy:
In job scheduling, for job WSS we use
if job has run: c)
else if b) is nonzero: b)
else a)
Problem:
CPDN jobs run for a few minutes with small WSS (say, 1MB).
Then they grow to full WSS (say, 6 GB).
There are various scenarios in which this leads to problems.
E.g. suppose the host has 16GB RAM and 8 cores.
It gets 8 CPDN jobs.
It starts 2 of them.
Their WSS is measured as 1MB.
On the next reschedule the client starts 2 more CPDN jobs.
Eventually all 4 jobs expand to 6GB WSS.
This is bigger than RAM and maybe bigger than swap space.
Some of the jobs fail with memory allocation failure.
Solution:
Change the WSS policy to:
If job has run: max(a, c)
else max(b, c)
Note:
What happens if project's rsc_memory_bound is wildly wrong?
if too large:
client may run fewer jobs for that project
if too small:
same as current: client may run too many jobs,
and they may fail (and possibly cause jobs of other projects to fail)
with malloc failure
(This problem was reported by Glenn from CPDN)
For each runnable job we have several estimates of its future WSS:
a) the project-supplied value rsc_memory_bound b) APP_VERSION::max_working_set_size: the max measured WSS of jobs using this app version since the client started (not saved to state file) c) if the job has already run, ACTIVE_TASK::working_set_size_smoothed: recent average (on the order of 1 min) of its WSS
Current policy: In job scheduling, for job WSS we use if job has run: c) else if b) is nonzero: b) else a)
Problem: CPDN jobs run for a few minutes with small WSS (say, 1MB). Then they grow to full WSS (say, 6 GB). There are various scenarios in which this leads to problems. E.g. suppose the host has 16GB RAM and 8 cores. It gets 8 CPDN jobs. It starts 2 of them. Their WSS is measured as 1MB. On the next reschedule the client starts 2 more CPDN jobs. Eventually all 4 jobs expand to 6GB WSS. This is bigger than RAM and maybe bigger than swap space. Some of the jobs fail with memory allocation failure.
Solution: Change the WSS policy to: If job has run: max(a, c) else max(b, c)
Note: What happens if project's rsc_memory_bound is wildly wrong? if too large: client may run fewer jobs for that project if too small: same as current: client may run too many jobs, and they may fail (and possibly cause jobs of other projects to fail) with malloc failure