[http metricbeat module] Enhance the http metricbeat module to support HTTP Digest Authentication

kush-elastic commented 2 years ago

Describe the enhancement: The current http metricbeat module does not have support for HTTP Digest Authentication. HTTP Digest Authentication is designed to be more secure than traditional digest authentication schemes. Many products expose their metrics on endpoints that use digest authentication, which can't be accessed using the current http metricbeat module.

Describe a specific use case for the enhancement or feature: To create an integration package for MongoDB Atlas we need to access endpoints that require HTTP Digest Authentication. For example if we try to access following endpoint GET https://cloud.mongodb.com/api/atlas/v1.0/groups/{GROUP-ID}/processes/{HOST}:{PORT}/measurements we get following error.

{ "error": 401, "reason": "Unauthorized", "detail": "You are not authorized for this resource." }

To accesses this endpoint we can perform following curl command.

curl --user "{PUBLIC-KEY}:{PRIVATE-KEY}" --digest \ --header "Content-Type: application/json" \ --include \ --request GET "https://cloud.mongodb.com/api/atlas/v1.0/groups/{GROUP-ID}/processes/{HOST}:{PORT}/measurements?granularity=PT1M&period=PT1M&pretty=true"

yug-rajani commented 2 years ago

On digging deeper into the issue, we came across a similar issue (https://github.com/elastic/beats/issues/15837) for Heartbeat. The issue seems to be blocked because of its dependency on this enhancement request (https://github.com/golang/go/issues/29409) on the net/http package which is open. http metricbeat module uses net/http package for the implementation of the json metricset which we intend to leverage.

There are some Golang third-party libraries using which digest authentication is supported. However, usage of third party libraries as an extension of http module in beats just to add support for digest authentication does not seem to be a good idea.

As far as Python is concerned, this can be implemented directly using a simple snippet which uses the requests library which has the support for digest authentication. However, the http metricbeat module has been built using Golang and integrating a Python snippet there does not seem to be a feasible solution.

elastic / beats

[http metricbeat module] Enhance the http metricbeat module to support HTTP Digest Authentication #32849