cscott
commented
8 years ago Additional idea: make the prune-db script "the new normal" by adding a process to serialize messages to a on-disk log file after a certain time as past (messages more than N minutes old, for example). Then somehow hack the client to do a fetch of those static files to fill in the back history. Perhaps as a start just piggy back on the paging feature, and fall back to two different paging mechanisms?
To do: figure out how to integrate read-file to a static file w/ the existing db mechanism.
Torgen
JeffJakubowski
So this hunt, we had to periodically purge history from ringhunters chat in order to maintain website responsiveness. That sucked! But recording the script I used (
prune.mdb):Then run this with
meteor mongo < prune.mdb.Also, startup connection storms were also The Worst. We managed these by:
I think this worked in part because meteor handles a dropped connection slightly differently that a hot code swap. For a dropped connection some sort of randomized backoff & retry was used, which spaced out the reconnection enough to allow meteor to recover from the restart.
Ideas for next year: 1) test hunt in early december to figure out if we still have a load issue after the annual meteor rebase. 2) fill up the db with messages for testing, since part of the issue seems to be related to size of ringhunters chat 3) separate ringhunters chat db from puzzle chat db, so that ringhunters doesn't bring down puzzles. (this would be annoying to code, sigh) 4) check to see if the followup processing is slowing things down. Perhaps we need to throttle that explicitly -- like, add a
followup dirty: truefield whenever we insert a new message, then once every throttle period start by doing a search for the earliest and latest timestamp withfollowupDirty=true, and replaying that range (or, technically, between the messages immediately preceding and following that range) when updating followups. 5) throttle should take into account how long it takes to synchronously execute the throttled computation. The underscore _.throttle appeared to run every minute, regardless of how long it tool to do its update task. We might need to write our own version of throttle, which sleeps X minutes after the synchronous task completed, before reruning the update. 6) Perhaps the$orin themessages-in-range-nicksubscription is confusing minimongo into doing expensive searches through all messages. Try separating that out into two separate subscriptions (one for PMs sent by me, and some for PMs sent to me). 7) Use a real database, instead of mongo?