search

cjb / codex-blackboard

Meteor app for coordinating solving for our MIT Mystery Hunt team
GNU Affero General Public License v3.0
25 stars 17 forks source link

Investigate SmartCollections / $limit support #104

Closed cscott closed 10 years ago

cscott commented 10 years ago

As noted at https://github.com/meteor/meteor/wiki/Oplog-Observe-Driver the new Meteor oplog tailing implementation doesn't apply if your queries use limit -- which our chat message queries do.

Add the facts collection as described at the bottom of that wiki page (issue #58) and see how much this hurts us. Perhaps isssue #100 has already fixed our major scaling issues. If not, then we might need to use SmartCollections (http://meteorhacks.com/introducing-smart-collections.html) after all.

cscott commented 10 years ago

Another option is to split the Messages collection (again) so that the most recent > MESSAGE_PAGE messages (not counting PMs) are stored in a separate RecentMessages collection which can be published without a $limit (and then hard-limited to MESSAGE_PAGE on the client side). We run a periodic update task every minute or so which moves old messages from RecentMessages to Messages (for every possible room, yuck). Querying with an explicit timestamp merges RecentMessages and Messages -- but those queries should be rare.

Perhaps we could use beta code for $limit support if dglasser has it?

cjb commented 10 years ago

SmartCollections seems like a better idea than that refactor, if oplog limit isn't forthcoming.

cscott commented 10 years ago

Refactored in the page-refactor branch to avoid limit queries. Not quite the refactor described above -- what I did was create a new Pages collection which stored the min/max timestamp for every page, maintained by the server. Seems to work, though.

cscott commented 10 years ago

Closing this bug, the page refactor should be sufficient.