Open auguste-elax opened 3 months ago
Hello @auguste-elax, I am interested to know more about your use case, could you tell me more on how you are streaming your MongoDB data into your applications ? Maybe are you implementing CDC with a MongoDB Kafka sink ?
hi @thild42 ! I'm indeed leveraging log-based cdc with debezium through an Airbyte instance deployed in our cluster. Here's their documentation on the matter if your are interested! Thanks
As an OVHcloud customer using a Public Cloud MongoDB instance, I want to be able to manually configure my database's operation log size (oplog) so that I can successfully stream this database's data into our internal applications. Currently it seems the oplog is set as 6 hours but fluctuates and can go lower (we observed ~3 hours recently). I'd like to set it to 24hours which is considered as best practice.
Streaming data into other applications is essential to us as it'll enable our reporting platform, as well as machine learning feature computation that need to run on the data outside of the database in services adapted to our intended workloads.
A workaround will be to switch cloud provider for the database or self host.