Improve logging while running under Newrelic infra agent

[nmoseman]

Description

It is difficult to configure the elasticsearch integration because if you make changes the only indication that if something is working or not is by checking whether data is making it into Newrelic.

What I would like to see is some logging output that would tell me if the nri-elasticsearch plugin is actually pulling data or if not then why not. Like a verbose option to say if it is getting no data, or timing out, or seeing a SSL error, and if it is getting data then optionally telling how much data it is pulling in bytes or number of characters or something like that.

I don't know if the nri-elasticsearch is responsible for this while running under newrelic infra or if this is the correct place to request it. But I figure it is a good idea to put it out there because it does cause a lot of people to trip up. While talking to other people in my org they all experienced mysterious "no data" results that were confusing at different times.

Acceptance Criteria

Enabling a optional verbose mode while running under newrelic infra or, if such a option exists, documenting how to use it.

Describe Alternatives

Executing the nri-elasticsearch on the command line with the options you would put in /etc/newrelic-infra/integrations.d/elasticsearch-config.yml is useful at troubleshooting issues. But this is sub-optimal because with modern cloud-based infrastructure it is not always easy to get on the host running the integration to test things manually.

Dependencies

Probably something to do with newrelic infra agent itself. I don't know how logging works for plugins like this.

Additional context

Estimates

I am hoping that it is a "S". Just adding a env option that can be used in elasticsearch-config.yml or in a container that would give us better logging.

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5): Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration[bot]]

https://new-relic.atlassian.net/browse/NR-173730

[paologallinaharbur]

Hello! Currently, you need to enable the verbose mode of the infrastructure agent to see all the logs generated by integrations. On the other hand, error logs generated by the integration should be still visible without verbose mode, however, due to a compatibility issue in the infra agent they are currently not shown.

We have an issue already to fix that behavior. Related to https://github.com/newrelic/nri-mssql/issues/137

[josemore]

Also consider enabling "smart" verbose mode by default in the agent, it should print errors + "n" configurable lines before that error when issues are detected in the agent: https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent/configuration/infrastructure-agent-configuration-settings/#level

[workato-integration[bot]]

The requested feature already exists.