[docs] Update docs guidelines to include supported inputs for each integration

[colleenmcginnis]

In #3433, @jamiehynds suggested:

It may also be worth calling out which inputs are supported by the integration. This come up occasionally with users - they have to go to the integration and manually determine which inputs are support. e.g. for some integrations users can ingest logs via API, S3 or log file.

Input types can be found in the manifest.yml for each integration. However, it's still not clear to me what the one-to-one comparison of the inputs you can find in the manifest.yml align with the inputs listed in the Filebeat docs referenced in https://github.com/elastic/integrations/pull/3433#discussion_r908215821.

I don't know if I'm overthinking this, but (for example) I see four inputs for the System integration:

Based on Jamie's comment from the PR I would guess...

• logfile = Log File
• httpjson = REST API

...but I don't know how to reference winlog and system/metrics.

As next steps, maybe we can:

• Work together separately to develop a one-to-one comparison of inputs to the relevant reference docs
• Update the documentation guidelines to explain how to document inputs in READMEs
• Update all docs that have already been reworked to follow the new guidelines

cc @elastic/obs-docs

[dedemorton]

I don't think we should point to the Beats documentation because the syntax for Elastic Agent inputs is different from Beats (and we don't want users to think they need to modify the Beats config).

What we need to do is get the settings documented for Elastic Agent. I have some big (old) PRs that I updated recently, but until I get some input from development, I can't move the PRs forward. Here's the issue I created to track where we're at: https://github.com/elastic/ingest-docs/issues/131.

@jlind23 Who is the best person to help us move forward with https://github.com/elastic/ingest-docs/issues/131? I don't think we can fix the problem that @colleenmcginnis brings up until we've gotten the settings documented. Everytime I ask about this in slack, the response is crickets.

UPDATED: I have a little brain fog and missed Colleen's comment about "next steps." So documenting the settings isn't a blocker for updating the guidelines, but IMO we do need to get the inputs documented if we want the list of inputs to make sense to users.

[dedemorton]

Sorry, added the wrong link in my comment. Updated it to https://github.com/elastic/ingest-docs/issues/131. elastic/ingest-docs#131 points to all the related PRs that are currently open and blocked.

[bmorelli25]

++ on not linking these inputs to the Beats documentation.

I also think this ties into the automation meta issue https://github.com/elastic/observability-docs/issues/1848. IMO this should be automated as well.

[bmorelli25]

I played around with this for a bit tonight. I created a script to determine all of the available input types and the number of integrations that use each type. There are a lot of different input types (see the table at bottom of this comment).

In the package manifest, each integration describes a type, title, and description per input. For example:

https://github.com/elastic/integrations/blob/f1a33c30dc0d92defe9e33996c9ab873708510bc/packages/apache/manifest.yml#L32-L34

In almost every integration, the title and description are very similar. What if we updated the title and description of each input to be more descriptive? We could then use these titles and descriptions to programmatically add input documentation to each readme file. I created a proof of concept for this in https://github.com/elastic/wordlake/pull/32. Right now it looks at the policy_templates.inputs array and adds the type and title to a new Inputs documentation section. The result looks like this:

## Inputs

Some descriptive text about inputs...

| Type | Description |
| ---- | ---- |
| tcp | Collect logs via syslog over TCP|
| udp | Collect logs via syslog over UDP|
| logfile | Collect logs via log file|

---

Input type	Count of type
logfile	64
tcp	38
udp	38
httpjson	36
aws/metrics	19
aws-s3	13
azure/metrics	13
aws-cloudwatch	9
filestream	9
kubernetes/metrics	8
http_endpoint	7
azure-eventhub	6
winlog	5
prometheus/metrics	3
gcp-pubsub	2
journald	2
system/metrics	2
activemq/metrics	1
apache/metrics	1
apm	1
audit/auditd	1
audit/file_integrity	1
awsfargate/metrics	1
cloudbeat	1
docker/metrics	1
elasticsearch/metrics	1
fleet-server	1
haproxy/metrics	1
iis/metrics	1
jolokia/metrics	1
kafka/metrics	1
kibana/metrics	1
linux/metrics	1
logstash/metrics	1
mongodb/metrics	1
mysql/metrics	1
nats/metrics	1
netflow	1
nginx/metrics	1
o365audit	1
osquery	1
packet	1
postgresql/metrics	1
rabbitmq/metrics	1
redis	1
redis/metrics	1
sql/metrics	1
stan/metrics	1
synthetics/browser	1
synthetics/http	1
synthetics/icmp	1
synthetics/tcp	1
syslog	1
traefik/metrics	1
vsphere/metrics	1
windows/metrics	1
zookeeper/metrics	1

[jlind23]

@dedemorton the best ones to help you out with the standalone config doc https://github.com/elastic/ingest-docs/issues/131 will be @pierrehilbert and @michalpristas

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!