estuary / connectors

Connectors for capturing data from external data sources
Other
48 stars 13 forks source link

source-mongodb: add exclusive collection filter advanced option #1649

Closed williamhbaker closed 3 months ago

williamhbaker commented 3 months ago

Description:

If only a small number of relatively low-volume bindings are enabled for a database with a large number of other potentially very high-volume collections, it is desirable to only receive documents for the collections that have enabled bindings. This adds an advanced configuration option to build such a filter for the MongoDB change stream pipeline.

Also some logging tweaks for tracking change stream lag times on a per-database level.

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

(anything that might help someone review this PR)


This change is Reviewable