search

yougov / mongo-connector

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Apache License 2.0
1.88k stars 479 forks source link

Arbitrary function that receives original DOC and returns expected DOC? #851

Open RafaPolit opened 5 years ago

RafaPolit commented 5 years ago

Good morning!

I am trying to get a grasp of the philosophy behind this connector. I have read that it is difficult to implement a filter based on values. For instance, having a "published" key that can hold 'true' or 'false', and determine if that particular operation should go forward or not. (#304) Still, this has been requested in 2015 and it really seems that many scenarios (mine included) would require such a functionality, at least in its simpler form.

In absence of that sort of functionality... could it be possible to have an arbitrary function that gets the oplog event with its doc and returns a transformed doc to be passed forward to the chain?

In that way, we may find a way to include any value-related logic into that function? Or at least append 'flag' properties based on value to pass along. For instance, something that 'if value of key X is Y, append property "Z" with value false'.

Something that could allow manipulation of data based on value rather than structure.

Thanks for any feedback, best regards, Rafa.

jaraco commented 5 years ago

Seems like a reasonable request. Would you be willing to draft and submit a pull request?