Open andrewkroh opened 5 years ago
@webmat Would it be appropriate to populate observer.*
in any of these above use cases? If so which host's data would go there?
Here are cases that should populate observer
:
observer.hostname
& so on with other observer fieldshost.hostname
& so on with other host fields.observer.hostname
& other fields should be populated with that host's detailhost.*
host.*
cannot be populated.Here are cases that should not populate observer
:
In both of these cases, if the data stream contains the source host's detail, it should to go to host.*
.
The case of monitoring containers may require a chat, I see a few cases, there may be more:
I'm not sure the current semantics as defined by ECS are clear nor sufficient to fully capture this. Maybe I'm wrong. But I'd be more than happy to have a chat with folks specialized with monitoring Docker, and hash out ideas here. Let me know if that would be useful.
I just installed filebeat 7.4 and I am using the netflow module. I have the same problem. agent.hostname is the agent.hostname for the hostname of netflow input server instead of the sender. I had to update some visualization for Kibana and replace agent.hostname with observer.ip.
It is not only the processors, but also libbeat directly adding some fields. See: https://github.com/elastic/beats/blob/master/libbeat/publisher/processing/default.go#L78
I'm in favor of not automatically modifying any event once it hits libbeat. All modification should be opt-in via processors.
I think we need way for inputs and modules to be able to "designate" that
host.*
should not be set by default. The output pipeline and also theadd_host_metadata
processor will need to honor this "designation".
I totally agree, we have this same problem with cloud metadata and cloud monitoring modules (AWS, Azure, etc). Something we have with add_cloud_metadata
is that it won't override any info sent by the input/module itself:
In this case, cloud metadata for the agent is not sent, which may not be ideal (it could make sense to have it under observer.cloud
?)
Something like this could make sense for add_host_metadata
, where it could decide to put the metadata somewhere else (as you said, maybe observer
)
The case of monitoring containers may require a chat, I see a few cases, there may be more:
An agent can be installed on the host running Docker directly (not using Docker)
- It can then monitor containers
- It can monitor the host itself
- An agent can be installed as a sidecar container, with the purpose of monitoring other containers local to the host
- An agent can be installed in a container, with the purpose of monitoring the host
I'm not sure the current semantics as defined by ECS are clear nor sufficient to fully capture this. Maybe I'm wrong. But I'd be more than happy to have a chat with folks specialized with monitoring Docker, and hash out ideas here. Let me know if that would be useful.
Happy to participate in this conversation @webmat!
To me it feels like the overall problem is that we have a limited set of namespaces, but yet there is potentially a trail of subsystems an event might have been passed through. Ultimately one might want an array of system descriptors the event has passed.
For now I want to solve the issue at hand, as this has come up a few times already. The host
and agent
fields are always overwritten. Currently host
and other fields are enforced, potentially overwriting fields no matter where values come from. This is not really ECS problem itself, but a general beats one, as we also mess with users already using ECS for their own data.
Overall ECS discussions are maybe better handled in the github.com/elastic/ecs repository.
For example in filebeat we introduced a kafka input, to allow architectures like Beats->Kafka->Filebeat->Elasticsearch
. The problem becomes even more apparent in this situation. In the simplest case Beats
would be the host, and the second Filebeat should be the collector
. We also pass some fields via @metadata
from the inputs like pipeline name, document id, or index name. Once the event reaches the 'collector' filebeat, it should be treated by default as if Beats->Elasticsearch has been configured directly. And this is currently not the case.
We also do not want to introduce "heavy" breaking changes in 7.x. So for 7.x I'm planning these changes (independent of ECS):
ecs.version
, if it's already presenthost
field or any of its subfields if already presentagent
fields, if agent
is already presentobserver
fields, if observer
is already presentoverwrite
setting to selected list of processors that affect cloud
, host
, observer
fieldsfalse
, the namespaces are protected. e.g. if host
is already available in the event, the processor will not add any host.X fields (as these might represent the wrong host)target
or namespace
setting, so users can overwrite the the field names (e.g. set target to 'collector' when add_host_metadata is used)add_cloud_metadata
, add_host_metadata
, add_observer_metadata
add_locale
: do not overwrite event.timezone if present, but allow user to configure alternative target fieldFor 8.x I would like to remove setting host
, agent
or observer
from within libbeat. Libbeat should not enforce fields, but allow solutions/users to opt-in. All these fields are already available via processors, and I'd prefer to provide default configs with these being enabled. Moving more functionality to processors and removing default behavior will also simplify the setup in libbeat itself.
I will create issues for individual tasks, if we agree on the plan.
This plan makes a lot of sense. Thanks for putting this together, @urso!
FWIW this also concerns APM, we are setting observer
information.
@urso, regarding an overwrite
option in the processors, you wrote:
If false, the namespaces are protected. e.g. if host is already available in the event, the processor will not add any host.X fields.
This sounds good but I don't think it cannot work as long as libbeat continues to set host.name
because then add_host_metadata
would never get added because host
would always exist. We could move setting the host.name
field into the add_host_metadata
processor. I think that would address this problem, but would probably cause some level of breaking change (haven't thought through all the consequences yet). WDYT?
For reference this is the code that adds host.name
followed by where it runs the global add_host_metadata
:
Pinging @elastic/integrations-services (Team:Services)
Good point. Maybe it would be better to apply the 'builtin' fields after the local and global processors have been run. When applying builtins, the builtins must not overwrite existing fields, but can add missing fields to namespaces. I think this is a change we can do in 7.x. Although it is a change in behavior, we might consider the current behavior a bug, and the change a fix :)
I'm afraid that if we move the builtins after the globals that we will break existing pipelines. There are users with drop_fields in their pipelines to remove builtins.
My proposal is that we implement your plan above in master targeting 8.0. Then in order to avoid breaking changes in 7.x, we use a bit of event metadata to influence the later stages of the pipeline's handling of host metadata. See https://github.com/elastic/beats/pull/17919 which would address the pressing need of omitting host
when forwarding.
We should implement the changes detailed above by @urso for 8.0. In an effort to minimize breaking changes to 7.x I am making some changes to address the issue without affecting the default behavior users are currently expecting. This is mostly accomplished through updating the example configurations to demonstrate using tags to disable add_host_metadata
and by adding config options where necessary to disable host.name
in libbeat.
publisher_pipeline.disable_host: true/false
option to disable host.name
from an input. https://github.com/elastic/beats/pull/18159Hi all, just want to mention, it is also a problem with all the cisco modules: https://github.com/elastic/beats/issues/14933
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)
There are several use cases in Beats where the data reported by a Beat did not originate on that Beat host. Some examples are syslog, windows forwarded events, router netflow data, and cloud watch logs. In these cases it would be appropriate to set the
host.*
field to information about the originating machine.From ECS:
Some issues related to this:
13777
13706
13589
10698
I think we need way for inputs and modules to be able to "designate" that
host.*
should not be set by default. The output pipeline and also theadd_host_metadata
processor will need to honor this "designation".