[Meta] MongoDB Atlas

[akshay-saraswat]

This is the Meta issue to track the development of MongoDB Atlas Integration

MongoDB Atlas

It is an integrated suite of data services centered around a cloud database designed to accelerate and simplify how you build with data.

Decisions

1. Endpoints:

   • It pushes metrics through both REST and Prometheus endpoints.
   • Prometheus Endpoint collects far lesser metrics than REST endpoint. Hence REST endpoint is being preferred.
   • REST endpoints require digest authentication.
   • It supports both API 1.0 and API 2.0 for the mongo db API calls. The difference between these should be analyzed

2. Input Type :

   • Since we are going ahead with the REST endpoints. We have 2 inputs under consideration:
   • CEL Input:
   • HTTP JSON Input:
3. Default Time Duration of collection
4. Response format conversion to ES recognized JSON format
5. Pagination

Datastreams:

MongoDb Atlas supports both Logs and Metrics.

Metrics Datastreams:

1. Disk Metrics

2. Database Metrics

3. Host Metrics

4. Memory Metrics

5. Operations Metrics

6. System Metrics

7. Process Metrics

Logs Datastreams:

TBD

[yug-rajani]

The following issue seems to be a blocker for moving ahead with the development of MongoDB Atlas integration package: https://github.com/elastic/beats/issues/32849

httpjson can also be used here, but it also lacks the support for digest authentication. In addition to that, it even lacks the support of multi-host i.e. an array of hosts which is present in http metricbeat module. Hence, for now, as far as MongoDB Atlas integration is concerned, enhancing http metricbeat module seems to be a better option as compared to extending httpjson.

Alternative option for data collection: MongoDB also exposes metrics on Prometheus endpoints for which users will need to configure MongoDB's Prometheus integration, which is only available on M10+ clusters. These M10+ clusters are only available on the paid version of MongoDB. In addition to that, based on the documentation, the number of metrics exposed on Prometheus seems to be less than the metrics available on the REST endpoints. For example, following is the list of some of the metrics that are available using the REST endpoints but not using Prometheus:

• CACHE_USED_BYTES
• DATABASE_AVERAGE_OBJECT_SIZE
• DB_STORAGE_TOTAL
• DISK_PARTITION_IOPS_READ
• DISK_PARTITION_IOPS_WRITE
• DISK_PARTITION_UTILIZATION
• DISK_PARTITION_LATENCY_READ

CC: @lalit-satapathy @rameshelastic

[SubhrataK]

Update: Narrowing down the scope of this integration to support M10+ clusters and above.

@andresrc and @rameshelastic Please help with the license for M10+ clusters.

[rameshelastic]

Still following up with IT. Not much luck so far.

Update on Jun 23: We have set this up and are unlocked on this issue.

[milan-elastic]

Started exploring CEL input to build the MongoDB Atlas, but I found HTTP Digest Authentication support is not available in CEL input hence we won't be able to use it. I have reached out to @efd6 and created an issue for it, He is currently working upon to provide support of Digest Authentication and MD5 lib support in CEL. cc: @rameshelastic @lalit-satapathy

[milan-elastic]

As discussed with @SubhrataK, Currently downloading the logs from the MongoDB Atlas cloud is not the viable option for the client. So we'll make a pause on logs data streams development, and start working on the Metrics data stream after having the discussion on OTEL and Prometheus approach.

[ishleenk17]

Current Status :

1. Analysis done for Prometheus endpoint vs REST based endpoints. Prometheus supports much lesser metrics. (https://docs.google.com/document/d/1aBpGN2pMMDgDqLyeC4UjOBrHpP28KooGmbP4xa1nCC4/edit).
2. For REST endpoint, there are 2 options : HTTPJSON or CEL Input. Mongodb Atlas requires digest authentication, which is available only in CEL Input from 8.12 onwards.
3. The details of the different metrics we would support is being worked upon here.
4. CEL Input has been checked to verify that the digest authentication is indeed working.

Next Steps:

1. Shortlist the type of metrics we would like to support and divide them into datastreams
2. Verify that CEL Input fulfills all requirements for MongoDB Atlas
3. MongoDb clusters can have single node/multiple nodes. Figure out how we would be handling metrics from multiple nodes. Aggregation or individual node wise metrics ?

cc: @milan-elastic

[ishleenk17]

We are prioritizing only metrics datastreams as of now as discussed with PM @SubhrataK

[samcsmith]

I'm trying to use this integration for organization logs however it isn't working. Are there any steps to debug or is it a known issue?

[milan-elastic]

@samcsmith can you try to check following curl command to see if there is any data in your organization monogdbatlas env?

curl --location 'https://<PUBLIC_KEY>:<PRIVATE_KEY>@[cloud.mongodb.com/api/atlas/v2/orgs/](http://cloud.mongodb.com/api/atlas/v2/orgs/)<ORGANIZATION ID>/events/?includeRaw=true&itemsPerPage=100&pageNum=1&minDate=2024-06-11T11%3A00%3A10Z&maxDate=2024-06-11T12%3A00%3A10Z' \ --digest \ --header 'Content-Type: application/json' \ --header 'Accept: application/vnd.atlas.2023-01-01+json'

Please replace your pubilic key, private key, orgid and set minDate, maxDate based on the time you have configured the integration!

There is possiblity that there is not changes happen in your org recently, but if there are any events comming in curl then feel free to create an issue! FYI , Integration collects only last 30 min data when its installed first time, if it's not present then you wont see any events flowing!

[samcsmith]

I figured it out, we have our mongodb atlas setup to use API Access Lists meaning I had to specify the IP addresses of our agents natgw in the api key config on mongo.

[samcsmith]

I have found in some cases audit logs error with

        "Processor json with tag json_decoding in pipeline logs-mongodb_atlas.mongod_audit-0.0.5 failed with message: field [original] not present as part of path [event.original]",
        "field [json] not present as part of path [json.ts.$date]" ```

[milan-elastic]

@samcsmith

We need to check the logs of Audit corresponding to these error messages. Can you share the audit log file ( by downloading it from these steps ) of last one day? Also, if you can find and highlight the exact logs in which these errors appear. Please mask the sensitive information if any in the logs. Also, please cofirm this error is faced for some of the logs not for every logs. To track this please create a new issue where we can discuss on it.