search

ovh / public-cloud-roadmap

Agile roadmap for OVHcloud Public Cloud services. Discover the features our product teams are working on, comment and influence our backlog.
https://www.ovhcloud.com/en/public-cloud/
185 stars 5 forks source link

Let users configure their MongoDB database's Operation Log size #581

Open auguste-elax opened 3 months ago

auguste-elax commented 3 months ago

As an OVHcloud customer using a Public Cloud MongoDB instance, I want to be able to manually configure my database's operation log size (oplog) so that I can successfully stream this database's data into our internal applications. Currently it seems the oplog is set as 6 hours but fluctuates and can go lower (we observed ~3 hours recently). I'd like to set it to 24hours which is considered as best practice.

Streaming data into other applications is essential to us as it'll enable our reporting platform, as well as machine learning feature computation that need to run on the data outside of the database in services adapted to our intended workloads.

A workaround will be to switch cloud provider for the database or self host.

thild42 commented 3 months ago

Hello @auguste-elax, I am interested to know more about your use case, could you tell me more on how you are streaming your MongoDB data into your applications ? Maybe are you implementing CDC with a MongoDB Kafka sink ?

auguste-elax commented 3 months ago

hi @thild42 ! I'm indeed leveraging log-based cdc with debezium through an Airbyte instance deployed in our cluster. Here's their documentation on the matter if your are interested! Thanks