[MongoDB Atlas] Identify a working solution for the MongoDB Atlas integration

[milan-elastic]

Identify a way to implement MongoDB Atlas integration using available beats inputs/modules and analysis of what kind of data will be generated/collected using it. for more information please find meta here.

[milan-elastic]

For the MongoDB Atlas integration, we have a total of 3 approaches: 1 . MongoDB Atlas metrics can be collected using Prometheus, But we need M10+ cluster license for this approach.

2. MongoDB Atlas data can be collected using REST API, but Digest Authentication is not supported in Http metricbeat module hence we won't be able to make REST API calls using it.
3. We explored CEL input in order to collect MongoDB metrics from the REST API, but in CEL also Digest Authentication is not supported. We have raised issue related to it. So currently we can't use CEL as well

Note: If CEL/Http metricbeat module will support Digest Authentication then we won't need an M10+ cluster license.

[milan-elastic]

Following data can be collected using the prometheus input and filestream input, please find the details below:

No. | Data stream name | Type of data streams (logs/metrics) | Input Type | Description -- | -- | -- | -- | -- 1 | connection | Metrics | Prometheus(Metricbeat module) | A datastream that reports on the status of the connections. Use these values to assess the current load and capacity requirements of the server. 2 | network | Metrics | Prometheus(Metricbeat module) | A datastream that reports data on MongoDB's network use. 4 | operation | Metrics | Prometheus(Metricbeat module) | A datastream that reports on database operations and database replication operations by type since the mongod instance last started. 5 | global_lock | Metrics | Prometheus(Metricbeat module) | A document that reports on the database's lock state. 6 | replication | Metrics | Prometheus(Metricbeat module) | A datastream that reports metrics related to the replication process. 7 | assert | Metrics | Prometheus(Metricbeat module) | A datastream that reports on the number of assertions raised since the MongoDB process started. 8 | wired_tiger | Metrics | Prometheus(Metricbeat module) | wiredTiger information only appears if using the WiredTiger storage engine. Some of the statistics roll up for the server. 9 | hardware | Metrics | Prometheus(Metricbeat module) | This datastream collects metrics related to hardware such as disk process etc. 10 | activity_log | Logs | Filestream(Filebeat Input) | This datastream collects activity logs of MongoDB atlas. 11 | audit_log | Logs | Filestream(Filebeat Input) | Database auditing lets administrators track system activity for deployments with multiple users. Atlas administrators can select the actions, database users, Atlas roles, and LDAP groups that they want to audit. 12 | miscellaneous | Metrics | Prometheus(Metricbeat module) | This datastream collects data of the query execution system, memory, document, command.