mongo high CPU utilization

[catbarph]

When running Grow-IoT (meteor) on my macOS machine, process mongod always consumes 99% CPU

pls advise!

[JakeHartnell]

hmmm... more info would be helpful... we need tools for measuring these kind of things. @aruntk, @mitar any thoughts or suggestions?

Here's things I found in a brief search: https://github.com/meteorhacks/kadira-profiler https://www.gethappyboards.com/2017/07/rolling-out-your-own-instance-of-kadira/ https://projectricochet.com/blog/top-10-meteor-performance-problems https://forums.meteor.com/t/meteor-up-with-97-cpu-usage-on-production-im-in-panic-mode/29230/30

[catbarph]

Activity Monitor app says process mongod consistently has: ~12x collections files open (Grow-IoT/.meteor/local/db/collection-xx--...) ~8x index files open (Grow-IoT/.meteor/local/db/index-xx--...) ~9x ports open from redwood-broker->localhost

Does that seem right?

May investigate further with kadira if needed.

[aruntk]

@catbarph @JakeHartnell Found the issue. https://github.com/CommonGarden/Grow-IoT/blob/development/imports/api/events/eventBus.js is the culprit. I'll try to fix this.

[aruntk]

@catbarph This should solve the issue https://github.com/CommonGarden/Grow-IoT/commit/839b39d0f75c36041d662825cb25b3863ff9b5bb . To reduce size of the db delete all documents from jobs collection. db.getCollection('jobs').remove({}) .

@JakeHartnell We need to find a better way to integrate elastic search. I'll start working on that.

[mitar]

Why don't just use full-text indexes from mongo?

[aruntk]

@mitar Afaik mongo text search is very limited. Whereas elastic search gives a more powerful and extensive text search functionality. But for simple use cases like indexing events mongo indexing would suffice. Elastic connection was more of an experiment. @JakeHartnell Thoughts?

[mitar]

Yes, but which capabilities from full-text search do you need?

[aruntk]

@mitar My concern with mongodb in grow-iot context is more about aggregation performance rather than basic text search. Elasticsearch's aggregation seems to be more performant when in comes to large amounts of data (iot device events for example. analytics). In my experience even if it is indexed and sharded mongo still lags behind elastic-search/solr in terms of performance. Also with elastic search you get tools like kibana to visualise the data. What do you think?

[mitar]

I am just saying: what features you need -> map them to tools. Also, do not optimize things prematurely.

But yes, if plugging elasticsearch is easy, then do it. Do you need reactivity?

[aruntk]

Do you need reactivity?

In the context of search? No. Being said that I think the dbs (mongo and elasticsearch) should be in sync. To achieve this there is no need to use meteor's observeChanges(which caused this issue). We can poll the mongodb at long intervals and update elasticsearch db (since the use case here is analytics rather than search). Also there are tools like https://github.com/mongoosastic to sync the two dbs (even though it is mongoose dependant). If the question is about text search results changing on the front end reactively I'm not sure if that is a requirement. @JakeHartnell.

[JakeHartnell]

I am just saying: what features you need -> map them to tools. Also, do not optimize things prematurely.

Agreed, elasticsearch is not incredibly important right now, it's more than good enough if the two dbs sync. Frontend reactivity for search results is not a requirement.

Thanks @catbarph, @mitar, and @aruntk!

[JakeHartnell]

Talked with @catbarph, he's pulled the latest code with https://github.com/CommonGarden/Grow-IoT/commit/839b39d0f75c36041d662825cb25b3863ff9b5bb... still high cpu usage...

[aruntk]

@JakeHartnell Mongo cpu usage seems to be fixed in my system after those changes. If you are using meteor's mongo then try meteor reset

[catbarph]

meteor reset after pull worked! Thanks all!