search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.74k stars 8.14k forks source link

[Fleet] Outputs > Specify where agents will send data #150111

Open joepa37 opened 1 year ago

joepa37 commented 1 year ago

Kibana version: kibana:8.5.3

Elasticsearch version: elasticsearch:8.5.3

Original install method (e.g. download page, yum, from source, etc.): Elastic ECK 2.6

Describe the bug: Is posible to send data to an external elasticsearch deployment with fleet server?

I have tried with the Kibana Fleet UI settings but there is no username, password field for connection, if I specify those on Advanced YAML configuration give me these error: cannot set both api_key and username/password accessing 'elasticsearch'

I can see the Kibana Fleet Settings xpack.fleet.outputs > config described as Extra config for that output to set this manually but there is no example to set this config variable.

Screenshots (if relevant): image (1)

elasticmachine commented 1 year ago

Pinging @elastic/fleet (Team:Fleet)

kpollich commented 1 year ago

Hi @joepa37 - it looks like you might be trying to configure a remote Elasticsearch output here, which is not supported today. However, supporting remote Elasticsearch clusters is on our roadmap, and there's a tracking issue here with some context: https://github.com/elastic/kibana/issues/104986.

If this isn't a remote cluster, then you shouldn't need to provide a username/password as Fleet's "enrollment tokens" resource will provide the API key access each Agent needs to authenticate requests and ship data.

For some additional technical context, Fleet and Agent use Elasticsearch API keys for authentication. So, in order to support output to a remote Elasticsearch cluster, you'd need to have the same API key be valid for two separate clusters - which is not something plain API keys support. In order to fulfill this authentication need, remote clusters will need to provision a service account token that can be given to your Fleet Server instances.

cc @nimarezainia

joepa37 commented 1 year ago

Hello @kpollich - it's possible to achieve this using two fleet servers? I can see some done tasks related with "Proxies" feature here:

nimarezainia commented 1 year ago

Hello @kpollich - it's possible to achieve this using two fleet servers? I can see some done tasks related with "Proxies" feature here:

* [[REQUEST]: Support multiple Fleet Servers in Fleet UI observability-docs#2343](https://github.com/elastic/observability-docs/issues/2343)

* [[META]Allow specifying different and multiple fleet servers in agent policy fleet-server#903](https://github.com/elastic/fleet-server/issues/903)

* [[REQUEST]: Support multiple Fleet Servers in Fleet UI observability-docs#2343](https://github.com/elastic/observability-docs/issues/2343)

i believe the way you may want to achieve this is by writing to Logstash and then route from there to multiple ES clusters. As mentioned above agents don't support connections to a re mote S cluster at the moment.