search

yougov / mongo-connector

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Apache License 2.0
1.88k stars 478 forks source link

Dump no tail #387

Open marcinkubica opened 8 years ago

marcinkubica commented 8 years ago

Hi chaps

Would it be possible to add a no_tail conf option for this new iteration?

Have two kinds of dbs I have to index in elastic. One are being updated and modified where tailing oplog works good but the other one are daily static collections which wont change.

Option to no tail would allow me to just generate config with injected collection date and dump the latter ones and forget.

Cheers

ShaneHarvey commented 7 years ago

@marcinkubica Can explain in more detail why this is useful? It sounds like one long running mongo-connector instance can already handle syncing theses daily collections. Is the problem that the new collections are large and the connector is too slow to sync the oplog changes in a reasonable amount of time? If so, can you post some stats for reference?

marcinkubica commented 7 years ago

Hi @ShaneHarvey Sorry for responding now. My usecase was: collections are being generated daily without further changes (ie. a day worth of log ingest) I was looking for a functionality, where I can instruct mongoconnector to ingest such collection and disconnect (since no further changes to the collection would have been expected).

something like mongo-connector --collection XYZ --index XYZ --dump-only