Open make-github-pseudonymous-again opened 1 month ago
One problem with point 3. is that watch
does not seem to handle all possible queries that find
handles. A possible solution is to resort to polling the query using find
whenever watch
triggers an event for a reasonable superset of the input query.
For ordered callbacks (addedBefore
, removedBefore
), the different query responses can be compared using a longest increasing subsequence algorithm in order to trigger the corresponding callbacks:
changed
callbacksremovedBefore
from their previous position and addedBefore
to their new positionremovedBefore
addedBefore
This is implemented by Meteor's diff-sequence
package:
For unordered callbacks (added
, removed
), the solution is simpler: at most changed
for what is in both the old and the new response, removed
for what is not in the new response, and added
for what is in the old response.
This is implemented by Meteor's diff-sequence
package:
Note that Meteor's current implementation also fails to implement all possible queries supported by find
: it resorts to polling if minimongo
cannot create a matcher for it
Note also that Meteor's oplog observe driver becomes quite complicated once limit
is involved:
Unless watch
does not suffer from the same inherent complexity, it would be wiser to start implementing a simple polling solution first.
If resorting to pure change stream updates instead of polling, the following ChangeStream
change
event handler may make sense:
const handleChangeStreamEvents = <TSchema extends { _id: string, [key: string]: any }>(
self: Subscription,
collectionName: string,
doc: ChangeStreamDocument<TSchema>
) => {
switch (doc.operationType) {
case "replace":
if (doc.fullDocument) {
self.changed(collectionName, doc.documentKey._id, doc.fullDocument);
}
break;
case "insert":
if (doc.fullDocument) {
self.added(collectionName, doc.documentKey._id, doc.fullDocument);
}
break;
case "delete":
self.removed(collectionName, doc.documentKey._id);
break;
case "update":
const {removedFields, updatedFields} = doc.updateDescription;
const fields = Object.fromEntries([
...(removedFields?.map((key: keyof TSchema) => [key, undefined] as const) ?? []),
...Object.entries(updatedFields ?? {}),
]) as Partial<TSchema>;
self.changed(collectionName, doc.documentKey._id, fields);
break;
case "drop":
case "dropDatabase":
case "rename":
case "invalidate":
self.stop();
break;
default:
break;
}
};
See also: https://forums.meteor.com/t/pub-sub-with-mongodb-change-stream/57495.
observeChangesAsync
can have extreme lag when combined with raw collection operations (e.g. transactions). We should try to exploit Change Streams instead. Might also help with migrating away from Meteor.See: