prevent the use of too small jobs (execution time)

[npf]

Add start time penalty in the schduler to bad users

Add support to scheduler to postpone new next jobs of a user if his last jobs where too short in execution time.

[npf]

In order to compute the list of "bad" users, the following SQL query would be executed at the beginning of every scheduler run (once per run).

Say we want to detect users with at least one job that lasted less than 5 minutes: MINDURATION=300

Then the SQL query would be:

select job_user,count(job_id) from jobs where start_time > $(($(date +%s) - $MINDURATION)) and state in ('Error','Terminated') group by job_user;

Jobs of every listed users could then be postponed a bit.

@bzizou, @pmorillon, could you time such a query while having an heavy load ?

[bzizou]

On 05/23/15 18:04, Pierre Neyron wrote:

In order to compute the list of "bad" users, the following SQL query would be executed at the beginning of every scheduler run (once per run).

Say we want to detect users with at least one job that lasted less than 5 minutes: MINDURATION=300

Then the SQL query would be:

select job_user,count(job_id) from jobs where start_time > $(($(date +%s) - $MINDURATION)) and state in ('error','Terminated') group by job_user;

Jobs of every listed users could then be postponed a bit.

@bzizou https://github.com/bzizou, @pmorillon https://github.com/pmorillon, could you time such a query while having an heavy load ?

— Reply to this email directly or view it on GitHub https://github.com/oar-team/oar/issues/36#issuecomment-104917457.

4,5s en moyenne J'ai soumis 1000 jobs de 60s pour augmenter la charge, mais elle était déja assez importante (cf les jobs de bcaller)

-bash-4.1$ time psql oar -c "select job_user,count(job_id) from jobs where start_time > $(($(date +%s) - 300)) and state in ('error','Terminated') group by job_user;" job_user | count ----------+------- bcaller | 40 pinardh | 1 luyang | 1 bzizou | 110 (4 rows)

real 0m4.377s user 0m0.000s sys 0m0.000s -bash-4.1$ time psql oar -c "select job_user,count(job_id) from jobs where start_time > $(($(date +%s) - 300)) and state in ('error','Terminated') group by job_user;" job_user | count ----------+------- bcaller | 40 pinardh | 1 bzizou | 80 (3 rows)

real 0m4.314s user 0m0.000s sys 0m0.000s -bash-4.1$ time psql oar -c "select job_user,count(job_id) from jobs where start_time > $(($(date +%s) - 300)) and state in ('error','Terminated') group by job_user;" job_user | count ----------+------- bcaller | 50 pinardh | 2 bzizou | 110 (3 rows)

real 0m4.753s user 0m0.000s sys 0m0.000s

                 Bruno.Bzeznik@imag.fr
                     +33642763280

CIMENT project OAR Team http://ciment.ujf-grenoble.fr http://oar.imag.fr

[npf]

Ok, thx. 5s is quite significant with regard to the default scheduler timeout (30s per queue). However, Bruno has it set to 90s. Note that the query would be executed once per meta-scheduler call (not per queue).

@capitn, @pmorillon , what do you think ?

[capitn]

Indeed it is quite long. We have to cache this value and update it every 5 minutes, for example.

I don't think we have to put this cache value into the database, a temporary file can be enough because it is only used by the scheduler, isn't it?

[capitn]

After reflexion, maybe it is more suitable to create a new table in the database with this kind of data. For example:

CREATE TABLE accounting_recent_usage (
    accounting_recent_usage_user varchar(255) NOT NULL,
    accounting_recent_usage_nbjobs integer NOT NULL default '0',
PRIMARY KEY  (accounting_recent_usage_user)
);

The scheduler can update the usage every 5 minutes for example. Then if the last update has happened in the past 5 minutes, it just get the value from this cache table.