search

percolatestudio / publish-counts

Meteor package to help you publish the count of a cursor, in real time
https://atmospherejs.com/tmeasday/publish-counts
MIT License
200 stars 46 forks source link

publish-counts causes mongodb connection timeout with large datasets #86

Closed tgoldenberg closed 8 years ago

tgoldenberg commented 8 years ago

When I use this package locally with meteor run I have no errors and it works great. When I use meteor build and deploy the app in production, my connection to mongo crashes. Here are the logs. I hope there is an answer to this problem. 

   at /home/will/portal/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/mongodb/cursor.js:835:31
    at [object Object].<anonymous> (/home/will/portal/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/mongodb/db.js:1905:9)
    at [object Object].g (events.js:180:16)
    at [object Object].emit (events.js:98:17)
    at [object Object].Base.__executeAllServerSpecificErrorCallbacks (/home/will/portal/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/mongodb/connection/base.js:373:29)
    at Server.<anonymous> (/home/will/portal/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:469:10)
    at Server.g (events.js:180:16)
    at Server.emit (events.js:98:17)
    at [object Object].<anonymous> (/home/will/portal/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/mongodb/connection/server.js:521:88)
Exception while polling query {"collectionName":"articles","selector":{},"options":{"transform":null,"fields":{"_id":true}}}: MongoError: connection to [finance-production-mongodb-0.aqcqtyqs0tre1dl3mqmpmo4ogd.cx.internal.cloudapp.net:27000] timed out
    at Object.Future.wait (/home/will/portal/programs/server/node_modules/fibers/future.js:420:15)
    at SynchronousCursor._nextObject (packages/mongo/mongo_driver.js:986:47)
    at SynchronousCursor.forEach (packages/mongo/mongo_driver.js:1020:22)
    at SynchronousCursor.getRawObjects (packages/mongo/mongo_driver.js:1069:12)
    at PollingObserveDriver._pollMongo (packages/mongo/polling_observe_driver.js:152:48)
    at Object.task (packages/mongo/polling_observe_driver.js:90:12)
    at [object Object]._.extend._run (packages/meteor.js:800:18)
    at packages/meteor.js:778:14
    - - - - -
    at Object.toError (/home/will/portal/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/mongodb/utils.js:114:11)
    at /home/will/portal/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/mongodb/cursor.js:835:31
    at [object Object].<anonymous> (/home/will/portal/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/mongodb/db.js:1905:9)
    at [object Object].g (events.js:180:16)
    at [object Object].emit (events.js:98:17)
boxofrox commented 8 years ago 
Exception while polling query {"collectionName":"articles","selector":{},"options":{"transform":null,"fields":{"_id":true}}}:
  MongoError: connection to [finance-production-mongodb-0.aqcqtyqs0tre1dl3mqmpmo4ogd.cx.internal.cloudapp.net:27000] timed out

Indicates that your app lost its connection to the mongodb service. At first glance, this doesn't appear to be an issue specifically with publish-counts. You may have better luck on the meteor forums.

If you're adamant that publish-counts is causing the problem, then my recommendation is to set up a test with publish-counts removed and verify that the db connection doesn't time-out. If it doesn't time-out, try creating your own observer with Mongo.Cursor.observe [1] and verify the db connection doesn't time-out with that. If it does, then either Meteor or your MongoDB server is where the issue lies.

If the time-out doesn't occur, then I'm at a loss as to why publish-counts would instigate a time-out. At this point, I'd need a demo app that reproduces the problem as I have no experience deploying meteor apps.

tgoldenberg commented 8 years ago

@boxofrox pretty sure this is because the size of the collection was way to big, according to the spec for this package. On dev, we had maybe 1000 docs, but on staging, it would be in the 10s of 1000s. 90% sure that is what caused the remote DB connection to crash. As soon as I removed the package, everything worked fine. Just something to know for people who try to use it for way too large sets ;)

boxofrox commented 8 years ago

Thanks for the clarification, @tgoldenberg. This makes sense and is mentioned indirectly in the Notes section of the README [1]. Tmeasday clarified in #76 that this package is for counting small datasets only [2].

If you need to count large datasets, there's a link in the Notes section to arunoda's timer-based counting method.

I'll update the notes section so it's more obvious.

Cheers