OverSubscribe in ParallelCluster 3.8.0 does not work

[michaelmayer2]

When trying to use OverSubscribe parameter via CustomSlurmSettings at the Queue level, the pcluster command produces a validation error

  "configurationValidationErrors": [
    {
      "level": "ERROR",
      "type": "CustomSlurmSettingsValidator",
      "message": "Using the following custom Slurm settings at Queue level is not allowed: OverSubscribe"
    },

If setting the same at ComputeResources level, the validator is ok with it but evidently this then leads to a broken SLURM configuration given that the OVERSUBSCRIBE parameter is only allowed at Queue/ Partition level.

In the pcluster source code we are associating OverSubscribe with the Queue level but put it on the DENY_LIST https://github.com/aws/aws-parallelcluster/blob/develop/cli/src/pcluster/validators/slurm_settings_validator.py#L50

If removing oversubscribe from the DENY_LIST, pcluster will create a valid SLURM configuration, e.g. using this partial YAML code

....
  SlurmQueues: 
    - Name: interactive 
      ComputeResources:
        - Name: interactive 
          InstanceType: t2.xlarge 
          MaxCount: 20
          MinCount: 1
          Efa:
            Enabled: FALSE
      CustomSlurmSettings:
        OverSubscribe: FORCE
...

[hgreebe]

The reason we haveOverSubscribe in the deny list is because it is mutually exclusive with the JobExclusiveAllocation parameter.

If you really want to use OverSubscribe, make sure you don't have JobExclusiveAllocation and suppress the validator when creating the cluster.

[michaelmayer2]

Fair enough - happy to use the --validation-failure-level WARNING workaround for the time being.

It still would be great to add OverSubscribe on the Queue/Partition level though - adding OverSubscribe there will provide a non-functional SLURM cluster, even with validators as default.