Best Practices and Known unsupported flows with Elastic Agent assets.

lucabelluccini commented 1 year ago

Observing the user activity with Elastic Agents & Fleet, we should clearly state the conditions of use of package/integration assets.

Clearly state Elastic Agent Standalone requirements

Example When using the Elastic Agent Standalone, we must also clearly state the integration assets associated to the integrations added to the EA policy must be installed on the destination Elasticsearch cluster.

If Kibana is available, it's possible to do it via Fleet Kibana UI.
If Kibana is not available (imagine a remote cluster with no Kibana connected to it), then those operations should be done manually.

Using Elastic Agent & Fleet integration assets without Elastic Agent is not supported.

Example We do not test nor support installing the Redis Integration assets and then use Logstash to collect the Redis logs from a 3rd party source (e.g. Kafka or AWS S3) to the Redis integration Data Streams. Example We do not test nor support installing the Redis Integration assets and then use Filebeat to collect the Redis logs from a 3rd party source (e.g. Kafka or AWS S3) to the Redis integration Data Streams.

Reasons:

We do not expose any spec on the event format prior hitting the ingest pipeline on Elasticsearch and we do not test/validate this kind of flows.
Things might break on upgrade
We do not test it

Using Elastic Agent & Fleet integration assets in Custom Integrations can be done but it will be considered a custom setup and we should never reference assets of other integrations directly.

Example Let's suppose we want to ingest Redis logs from Kafka. We have the Redis integration, but it allows to do it only from files at a given path. We have observed users with Redis logs coming from a Kafka queue trying to build a Custom ingest pipeline using the Custom Kafka Logs integration, referencing the ingest pipeline of Redis into the index templates of the Custom Kafka Logs integration data streams.

Reasons:

If one references the ingest pipeline redis.info-1.10.0 in the index templates, on the next upgrade (which can happen automatically) of the Redis integration, the pipeline will be deleted (this is the default behavior of Fleet - see https://github.com/elastic/kibana/issues/141347). The use of such ingest pipelines might actually prevent the correct upgrade of the Redis integration as the ingest pipeline cannot be deleted if it is referenced by one data stream.

We must document how to properly create a copy of the integration assets (index templates, ingest pipelines) so that they can be considered customizations and they become "standalone" and therefore users are:

aware they might not work out of the box because they are, in fact, custom ingestion flows which are "inspired" by Elastic integrations
there is no risk of them breaking on upgrade because they get deleted, as the asset was managed by another integration

We can help users to set this up but it must be considered as a custom flow as there is no test/validation done on using Fleet assets as "foundations" for other custom integrations.

FYI @111andre111 @jlind23

lucabelluccini commented 2 weeks ago

Bumping

jlind23 commented 2 weeks ago

@kilfoyle you should probably take a look at this issue.

kilfoyle commented 2 weeks ago

Thanks @lucabelluccini. I'll try to get to this one next week or sooner. Sorry that it got missed.

kilfoyle commented 6 days ago

@lucabelluccini I've opened up this PR with the new "Best practices for integrations assets" page. Please let me know if it looks okay.

lucabelluccini commented 6 days ago

That's great @kilfoyle ! Thank you

kilfoyle commented 6 days ago

Here's the new Best practices for integrations assets page. Thanks for providing the nice draft, Luca!

Closing.

elastic / ingest-docs