Open ruspaul013 opened 1 year ago
@ruspaul013 I'm surprised that job spec doesn't simply return a validation error, but the constraint
block belongs under job
, group
, or task
, not under resources.device
. Can you verify this is not working after that's corrected?
Hello @tgross, thank you for the suggestion. I tried it, but now I get an error even if I do not use constraint
block for ids. I can not place a simple job using a GPU.
resources {
cpu = 3200*4
memory = 8192
device "nvidia/gpu" {
count = 1
}
}
constraint {
attribute = "${device.model}"
value = "NVIDIA GeForce RTX 4090"
}
and I get this warning
Scheduler dry-run:
- WARNING: Failed to place all allocations.
Task Group "paulr_gpu_test" (failed to place 1 allocation):
* Constraint "${device.model} = NVIDIA GeForce RTX 4090": 2 nodes excluded by filter
but the
constraint
block belongs underjob
,group
, ortask
, not underresources.device
I know that constraint block does not belong there, but the example from device
block says otherwise.
Yeah, that's something I often forget but you can set constraint
and affinity
at the device
level 😅
https://developer.hashicorp.com/nomad/docs/job-specification/device#constraint
But setting the constraint only filters out nodes from scheduling. Taking a look at the code I think the problem is that AllocsFit
doesn't take current device usage into account.
Devices are not considered comparable resources, so they're not filtered here: https://github.com/hashicorp/nomad/blob/e3c8700ded891702b7f94636033109efd8f71c3a/nomad/structs/funcs.go#L172-L178
And this part of the code only looks for device oversubscription among the allocs being scheduled, so it ignores the ones already running in the node: https://github.com/hashicorp/nomad/blob/e3c8700ded891702b7f94636033109efd8f71c3a/nomad/structs/funcs.go#L201-L207
Nomad version
nomad server:
Nomad v1.6.1
+ patch with #18141 nomad client:Nomad v1.6.1
+ patch with #18141Operating system and Environment details
Plugin "nomad-driver-podman" v0.5.1 Plugin "nomad-device-nvidia" v1.0.0
Issue
I created a patch with the solution provides in #18141 to test on our cluster. While testing I discovered that if a GPU or multiple GPU are already used in jobs, nomad will not give a warning about this and will place the job without using those GPUs.
Reproduction steps
Expected Result
Throw an warning like
WARNING: Failed to place all allocations.
Actual Result
Place the job on the client.
Job file (if appropriate)
Job file 1:
Job file 2: