Add support for Nomad - Githubissues

jsoriano commented 4 years ago

Nomad is an orchestrator for heterogeneous workloads maintained by Hashicorp.

This issue keeps track of efforts to support Nomad in Beats.

Nomad support could follow a similar approach to the available support for Kubernetes, for that the following components should be implemented:

[x] nomad autodiscover provider.
[x] add_nomad_metadata processor.
[ ] Try to use the events API for watching https://www.nomadproject.io/api-docs/events (it is polling now).
[ ] nomad metricbeat module.
[ ] nomad filebeat module for nomad agents.
[x] ~~Filebeat input if current ones are not enough.~~
[ ] Documentation and reference job files to deploy beats in nomad.

The code for these features should be placed under x-pack (licensed as Basic).

At the moment of creating this issue there are some open community PRs that provide some of the required components, we have to review and continue with them:

elastic/beats#14500 - Adds an add_nomad_metadata processor.
elastic/beats#14954 - Adds an add_nomad_metadata processor too, and a nomad autodiscover provider.

There are also some known issues regarding Nomad support (though they can be general issues):

elastic/beats#12988 - Filebeat fails to collect logs from containers started by nomad
elastic/beats#12178 - Problems reaching non-published ports with Docker autodiscover

// cc: @jorgelbg and @jorgemarey as authors of current open PRs

elasticmachine commented 4 years ago

Pinging @elastic/integrations-platforms (Team:Platforms)

jsoriano commented 4 years ago

Hey @jorgelbg and @jorgemarey,

I have been taking a look to your pull requests and the truth is that both look great, I would love to see Nomad support in Beats. Thanks both for your efforts!

Both PRs contain an add_nomad_metadata processor, and they both work pretty well, but #14954 is more complete, it has support for indexers and matchers, what I think will help to support more Beats and nomad runtimes in the future, and it also includes an autodiscover provider. So I think that it'd be better to go on with this PR only. In any case, @jorgemarey, if you have some time, it'd be great if you could also take a look to #14954, I am sure you can give some ideas there! For example I like that your processor in #14500 automatically gets the node ID of the local agent.

Thanks again!

jsoriano commented 3 years ago

Initial support merged, what includes an autodiscover provider and an add_nomad_metadata processor. It will be released in principle as experimental feature in 7.12.

Cobular commented 3 years ago

Thank you! Really looking forward to this being fully released.

botelastic[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

botelastic[bot] commented 1 year ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

joaquinjsb commented 1 year ago

👍🏼

botelastic[bot] commented 3 months ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

matya commented 3 months ago

The hope dies last 😅

elastic / beats

Add support for Nomad #16853