Make it easier for new users to configure their first (example) OTC pipeline

[calebhailey]

Related: #586

The Sumo Logic OTC Distro Configuration documentation is pretty good, but I was able to spend over 30 minutes reading through the docs without being instructed to actually start the collector with a functional configuration. We should offer users a complete example configuration that moves them one step closer to configuring real workloads using Sumo Logic with the Sumo Logic OTC Distro.

If I could suggest a minor revision, I'd make the following changes to the Configuration documentation:

1. Add a new "Example configuration" heading (above the "Basic configuration" heading, or in place of it).

   This section should provide a complete config.yaml example that users can copy and paste without modifications, and this configuration should provide a completely functional pipeline so that users can then start the collector and observe data is being delivered to their Sumo Logic account.

   I would suggest that this example configuration use a filelog receiver that is configured to read log data from a non-standard log file (e.g. /tmp/sumologic-otc-example.log).

   Why use a non-standard log file? Two reasons: 1) users might not be comfortable sending actual log data (yet), perhaps for fear of leaking sensitive data, or for other reasons. 2) choosing the right log file is tricky – we would want to avoid a system log file that could potentially be inactive at the moment the user is testing OTC (otherwise they might think something isn't working), and we might want to avoid a noisy log file like /var/log/messages because you don't necessarily want new users to drink directly from the fire hose. A known fake log file sets good expectations for the user – they wouldn't expect anything to happen without their intervention, and they get to control exactly when data is written to the file and thus get to see exactly how fast the collector sends their data to Sumo Logic!

2. Use environment variables!

   When I saw that the Configuration documentation provided an example config.yaml file with a placeholder for my installation token (e.g. install_token: <token>), I started wondering if the OpenTelemetry Collector config file supported environment variables. It turns out that it does!

   "The use and expansion of environment variables is supported in the Collector configuration."

   Source: https://opentelemetry.io/docs/collector/configuration/#configuration-environment-variables

   If we would provide example config.yaml files with configuration parameters that reference environment variables, it's one less chance for users to misconfigure their collector – they can just copy and paste and set one or more environment variables:

   extensions:
    sumologic:
      install_token: ${SUMOLOGIC_INSTALL_TOKEN}

3. Instruct users to verify the OTC pipeline by visiting their Sumo Logic account and starting a LiveTail, then run a command to append data to the configured log file. For example:

   echo $(date) ${hostname} INFO: Hello, Sumo Logic OpenTelemetry Collector Distro! >> /tmp/sumologic-otc-example.log

   Do they see the message in LiveTail? SUCCESS!! 🎉

From here there are a number of ways the Configuration documentation could guide users to take very productive next steps:

• Configure service management for OTC (e.g. systemd unit files)
• Configure log monitoring (e.g. get them to hook up OTC pipelines for actual log files)
• Configure metrics collection (e.g. provide a simple example to enable host monitoring)
• Configure tracing

/cc @pmm-sumo

[andrzej-stencel]

Thanks @calebhailey for these remarks, they are very very valueable for us. 🙇 Keep them coming!