Test the job grouping capability of CiGri

[jgaida]

CiGri provides two options for grouping task into OAR jobs:

• dimensional_grouping: allow to execute several jobs in parallel in a single submission if possible
• temporal_grouping: allow to execute several jobs one after the other in a single submission. The number of jobs is computed automatically by Cigri

Before advising users to use this feature, we should check if these options work as expected.

[jgaida]

I investigated this issue and here is what I found.

• both dimensional_grouping and temporal_grouping are documented in doc_jdl.rst.
• dimensional_grouping is not implemented (cf. comments in lib/cigri-joblib.rb)
• temporal_grouping seems to be implemented by submit_batch_job (in lib/cigri-joblib.rb) but I was unable to use it.

For temporal_grouping, my campaign script looks like this:

  {
    "name": "Some campaign",
    "nb_jobs": 40,

    "resources": "nodes=1",
    "properties": "",
    "exec_file": "$HOME/script.sh",
    "exec_directory": "$HOME",
    "temporal_grouping": "yes",

    "clusters": {
      "nancy": {}
    }

The temporal_grouping property is properly inserted into the database:

cigri=# select * from campaign_properties where campaign_id=90;
  id  | cluster_id | campaign_id |          name           | value
------+------------+-------------+-------------------------+-----------------

 5743 |          5 |          90 | checkpointing_type      | None
 5744 |          5 |          90 | dimensional_grouping    | false
 5745 |          5 |          90 | exec_file               |
$HOME/script.sh
 5746 |          5 |          90 | output_gathering_method | None
 5747 |          5 |          90 | properties              |
 5748 |          5 |          90 | resources               | nodes=1
 5749 |          5 |          90 | temporal_grouping       | yes
 5750 |          5 |          90 | walltime                | 03:00:00
 5751 |          5 |          90 | type                    | best-effort
 5752 |          5 |          90 | test_mode               | false
 5753 |          5 |          90 | project                 |
 5754 |          5 |          90 | exec_directory          | $HOME
(12 rows)

After that, the submit_batch_job function is supposed to do the actual grouping. This function is called by submit2if temporal_grouping is within the option list. Inside the loop by_options_jobs.each do |runner_options,jobs| of submit2, I only get the following options:

{"besteffort"=>true, "batch_id"=>2}. 

I thinks those options are related to the default configuration of the runner and are not related to the campaign properties (?) but bottom line, the function submit_batch_job is not called (as temporal_grouping is absent from the option list).