enhancement: diagnostics subcommand

[lukesteensen]

One of the biggest challenges when trying to help Vector users debug issues is collecting all of the relevant data from their environment. This can involve a lot of back and forth and it's not always clear which commands to run to get Vector-specific info like file checkpoints.

To address this, we should add a diagnostics subcommand (similar to homebrew's doctor command) that loads a user's config and gathers relevant information. That information can then be formatted nicely and output to the terminal or a file for the user to pass along to us.

Each component can independently implement its own set of checks via another build-style method on the config traits. For example, the file source checks could return information like the following:

1. Count and sizes of files in the configured directories
2. Checkpointed positions compared to the current file size (i.e. lag)
3. Latency of operations like globbing, fingerprinting, etc

[binarylogic]

Please also see #4670 for more details.

[neuronull]

Jotting down some thoughts that came to mind when this was brought up recently. There was a lot of discussion around having a command similar to what the Agent "flare" command does, for Vector.

For some of the investigations I've looked into, here are some things that would be helpful to capture:

• full vector configuration. Not just the parsed, but also what parts are in different files, if multi file
  • this is often the first thing we need from a customer and isn't always available so we have to ask for it, and there can be some back and forth, and even user error in providing the full / correct config
• vector version (obvious)
• some of the similar output provided to vector top , basically total counts for some of our more important internal telemetry such as bytes/events sent/received, component errors, events dropped. And per component.
• I like the idea from the initial comment about each component being able to provide specific data. This could be modularized and default to none, so there isn't a requirement, but we could add it to critical components, or problem centric ones, and have a precedent to add it to new components.
  • For example the datadog metrics sink could provide data on what types of metrics it has processed, batch sizes
    • the sink infrastructure layer could report retry attempt data
  • we could even have a parameter to the diagnostic sub command to collect profiling data. The profiling would be all optional but if the code path of the running instance exercised it, and profiling was enabled, perhaps through RAII techniques we could have an understanding of how much time is spent at various levels of the code path from initialization, to data flow through the components.