[Feature Request] Parameterized Job submit constraints at dispatch

[erhlee-bird]

A very useful potential enhancement to the Parameterized Job dispatch API would be to allow submittable constraints at dispatch time.

The current /v1/job/:job_id/dispatch endpoint takes the following parameters:

{
  "Payload": "A28C3==",
  "Meta": {
    "key": "Value"
  }
}

This issue proposes the following addition:

{
  "Payload": "A28C3==",
  "Meta": {
    "key": "Value"
  }, 
  "Constraints": [
    {
      "LTarget": "${attr.kernel.name}",
      "RTarget": "linux",
      "Operand": "="
    }
  ]
}

An example use-case my team has is as subsequently described:

# Job-1 writes a file to a shared mounted directory on one client
alloc-1 = dispatch(Job-1) 

# Job-2 would need to be scheduled to the exact client that alloc-1 
# was scheduled to in order to process the written artifact
dispatch(Job-2, client of alloc-1) 

We explicitly would like to adopt this pattern to take advantage of client locality and avoid incurring additional heavy network costs to uploading and redownloading large artifacts.

[diegs]

I tried to accomplish this use case via metadata interpolation in the constraint.

job "my_parameterized_job" {
  type = "batch"
  parameterized {
    meta_required = ["host_machine"]
  }
  constraint {
    attribute = "${attr.unique.hostname}"
    value = meta.host_machine # or any other variant of "${meta.host_machine}", "${NOMAD_META_host_machine}"
  }

However when I nomad job dispatch it seems that the metadata is not interpolated:

      * Constraint "${attr.unique.hostname} = ${meta.host_machine}": XX nodes excluded by filter

Per the constraints docs (https://www.nomadproject.io/docs/job-specification/constraint#value) and other issues I think this should work, and metadata is obviously interpolated correctly elsewhere in the job spec, so this seems inconsistent and possibly broken.

(This is with nomad v1.3.5)