Add tasks Meta and tags available for templates

[dani]

For now, nomad-vector-logger exposes the following task's attributes, which are then available in templates

• Namespace
• Job
• Group
• Task
• ID
• Node

It'd be great to also have access to Tags and Meta. I use meta fields for my prometheus metrics already, to differentiate multiple instances of the same task

    meta {
        metrics-port = "${NOMAD_HOST_PORT_metrics}"
        alloc        = "${NOMAD_ALLOC_INDEX}"
      }

Then some prometheus remap config so I have automatically the instance label set as $TaskName-$AllocMeta (eg : redis-0)

It'd be great to have access to those here as well, so vector could attach Metadata, like

[transforms.transform_{{$value.Key}}]
type = "remap"
inputs = ["source_{{$value.Key}}"]
source = '''
# Store Nomad metadata.
.nomad.namespace = "{{$value.Namespace}}"
.nomad.node_name = "{{$value.Node}}"
.nomad.job_name = "{{$value.Job}}"
.nomad.group_name = "{{$value.Group}}"
.nomad.task_name = "{{$value.Task}}"
.nomad.alloc_id = "{{$value.ID}}"
.nomad.instance = "{{$value.Task}}-{{index $value.Meta "alloc"}}"
'''

[mr-karan]

This one's a bit tricky because meta can be specified at job, group, task level. Quoting the docs:

Metadata is merged up the job specification, so metadata defined at the job level applies to all groups and tasks within that job. Metadata defined at the group layer applies to all tasks within that group.

So, which meta key should be exposed isn't very straightforward. Do you have any suggestions?

[dani]

Mmmm, indeed my bad. I though the API would return the merged Meta, but it's not. Maybe just adding the allocation index would already help. This info can be extracted from the Name, eg

[...]
    "EvalID": "121d6a59-c045-6974-5881-f2d21e920e26",
    "Name": "zookeeper.server[2]",
    "Namespace": "default",
[...]

=> Task is in allocation index 2, of in the server TaskGroup of the zookeeper Job.

[mr-karan]

I believe this parsing of zookeeper.server[2] -> [2] can be done using VRL. To keep it generic, I've added Name field

source = '''
# Store Nomad metadata.
.nomad.namespace = "default"
.nomad.node_name = "pop-os"
.nomad.job_name = "doggo"
.nomad.group_name = "app"
.nomad.task_name = "web"
.nomad.alloc_id = "e45a64fa-23c5-51ad-f5c6-2149dfaa5ce7"
.nomad.alloc_name = "doggo.app[0]"
'''

[JesseStorms]

I'm unsure if this is unrelated to this issue. Sorry if it isn't!

I'm trying to include Nomad Meta defined at job level, adapting dani's example for a bit more context:

[transforms.transform_{{$value.Key}}]
type = "remap"
inputs = ["source_{{$value.Key}}"]
source = '''
# Store Nomad metadata.
.nomad.namespace = "{{$value.Namespace}}"
.nomad.node_name = "{{$value.Node}}"
.nomad.job_name = "{{$value.Job}}"
.nomad.group_name = "{{$value.Group}}"
.nomad.task_name = "{{$value.Task}}"
.nomad.alloc_id = "{{$value.ID}}"
.nomad.metrics_port = "{{index $value.Meta "metrics-port"}}"
'''

Essentially, job.Meta.metrics-port is defined in the jobspec but I'm unsure how to actually get it in the vector template. I'm completely new to Golang, so the code goes way over my head. I could attempt to help myself if you can give me some pointers or insight how the templating works. Thanks!

[revverse]

For our Nomad configuration, we use a lot of tags that delineate the service sending logs (env, internal name, priority, etc.) So we consolidated all these tags within the Meta stanza and implemented modifications in nomad-vector-logger - commit All Meta tags will be stored as JSON fields: .nomad.task_meta = {{$value.TaskMeta}} .nomad.job_meta = {{$value.JobMeta}} .nomad.group_meta = {{$value.GroupMeta}}

The code is not optimally designed, so I refrained from creating a merge request in the main repository