Multiple instances

[jm4r7in]

When I reboot my server, htop shows a high CPU% and many instances. Do you have any idea how to fix this ?

Here is a screenshot of htop:

Thanks a lot!

[MasterJames]

Maybe try things from this doc http://docs.mongodb.org/manual/administration/monitoring/

[jm4r7in]

Thanks for your answer @MasterJames , but what exactly should I look for ? There is no user using the app, and for me it seems that for some reason something is starting many instances of node & mongo. I didn't have this situtation before, and it happened with the last deploy or restart of the server.

[MasterJames]

What happens if you put netstat -ln | grep -E '27017|28017'

[MasterJames]

How about trying from ssh on the instance... meteor mongo show logs show log global

Or maybe meteor reset Again from the deployed instance.

[jm4r7in]

@MasterJames Here is the commands I did via ssh :

netstat -ln | grep -E '27017|28017'

tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN
unix  2      [ ACC ]     STREAM     LISTENING     10992    /tmp/mongodb-27017.sock

mongo appName show log global

2015-08-18T22:31:37.457+0200 [initandlisten] connection accepted from 127.0.0.1:48357 #72 (2 connections now open)
2015-08-18T22:31:37.466+0200 [initandlisten] connection accepted from 127.0.0.1:48358 #73 (2 connections now open)
2015-08-18T22:31:37.466+0200 [initandlisten] connection accepted from 127.0.0.1:48359 #74 (3 connections now open)
2015-08-18T22:31:37.467+0200 [initandlisten] connection accepted from 127.0.0.1:48360 #75 (4 connections now open)
2015-08-18T22:31:37.467+0200 [initandlisten] connection accepted from 127.0.0.1:48361 #76 (5 connections now open)
2015-08-18T22:31:37.467+0200 [initandlisten] connection accepted from 127.0.0.1:48362 #77 (6 connections now open)
2015-08-18T22:31:46.331+0200 [clientcursormon] mem (MB) res:98 virt:951
2015-08-18T22:31:46.331+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T22:31:46.331+0200 [clientcursormon]  connections:6
2015-08-18T22:32:46.334+0200 [clientcursormon] mem (MB) res:118 virt:951
2015-08-18T22:32:46.334+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T22:32:46.334+0200 [clientcursormon]  connections:6
2015-08-18T22:37:46.347+0200 [clientcursormon] mem (MB) res:110 virt:951
2015-08-18T22:37:46.347+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T22:37:46.347+0200 [clientcursormon]  connections:6
2015-08-18T22:42:46.361+0200 [clientcursormon] mem (MB) res:116 virt:951
2015-08-18T22:42:46.361+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T22:42:46.361+0200 [clientcursormon]  connections:6
2015-08-18T22:47:46.374+0200 [clientcursormon] mem (MB) res:105 virt:951
2015-08-18T22:47:46.374+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T22:47:46.374+0200 [clientcursormon]  connections:6
2015-08-18T22:52:46.387+0200 [clientcursormon] mem (MB) res:106 virt:951
2015-08-18T22:52:46.387+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T22:52:46.387+0200 [clientcursormon]  connections:6
2015-08-18T22:57:46.401+0200 [clientcursormon] mem (MB) res:106 virt:951
2015-08-18T22:57:46.401+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T22:57:46.401+0200 [clientcursormon]  connections:6
2015-08-18T23:02:46.416+0200 [clientcursormon] mem (MB) res:114 virt:951
2015-08-18T23:02:46.416+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T23:02:46.416+0200 [clientcursormon]  connections:6
2015-08-18T23:07:46.431+0200 [clientcursormon] mem (MB) res:112 virt:951
2015-08-18T23:07:46.431+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T23:07:46.431+0200 [clientcursormon]  connections:6
2015-08-18T23:12:46.445+0200 [clientcursormon] mem (MB) res:111 virt:951
2015-08-18T23:12:46.445+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T23:12:46.445+0200 [clientcursormon]  connections:6
2015-08-18T23:17:46.458+0200 [clientcursormon] mem (MB) res:108 virt:951
2015-08-18T23:17:46.458+0200 [clientcursormon]  mapped (incl journal view):736
2015-08-18T23:17:46.458+0200 [clientcursormon]  connections:6
2015-08-18T23:18:23.481+0200 [conn71] end connection 127.0.0.1:48348 (5 connections now open)
2015-08-18T23:20:13.821+0200 [initandlisten] connection accepted from 127.0.0.1:48530 #78 (6 connections now open)

I hope this can help, because I have no idea how to solve this problem and I need to run my app...

[MasterJames]

I confess I'm not sure you're actually having a problem at all. when ssh'ed in you enter meteor mongo and then db.serverStatus() the output might say somewhere basically... connections: current:14 for a normal inactive DB so I think your just seeing connections waiting or something. Does your Processor Usage stabilize after a little while or go to 100%? Maybe block access or run it on a different port to see if any external connections are hitting it.

Anyway maybe you have some cluster setup or something? If you look into the meteor folder .meteor\local\db METEOR-PORT should have 3001 or what have you. if you kill the other meteor processes do they return? Anyway the port number might be different then the one configured but I doubt it.

Here is another thought. Shutdown the meteor or meteor mongo maybe instead of mongod --shutdown go into meteor mongo and then type use admin db.adminCommand({shutdown : 1, force : true})

exit that shell and check if mongod is still running

also you could try something like this to shutdown meteor. kill -9 ps ax | grep node | grep meteor | awk '{print $1}'

basically stop meteor from launching and see if there is still a mongod running.

I would think if you've used a clean image there isn't any problem. I hope this helps you debug things further on your end.

[MasterJames]

There is this link on Sharding http://docs.mongodb.org/v3.0/tutorial/view-sharded-cluster-configuration/ try db.printShardingStatus() or sh.status() to see if sharding is enabled

[MasterJames]

NOTE: Sharding isn't the same as Replica Set Also consider rs.status ()

When connected to the meteor:PRIMARY?> prompt via meteor mongo

[jm4r7in]

I don't use any Sharding, cluster or Replica Set. I use just a very basic/default meteor app deployed with mup (mongodb 2.6) and a basic cron to backup the DB.

Btw, when you tell me to do:

when ssh'ed in you enter
meteor mongo

I don't know exactly what you mean, because as fas as I know, the way to access the DB with an app deployed with mup is to:

ssh ...
mongo appName

The processor usage varies from 0 to 100%, with an average of 50% (according to kadira), even with no user/visitor. And before this problem happened (and yes, it's a problem), the processor usage averaged around 5% max. The host is a 4-core @3.4 Ghz 4 GB RAM VPS, so I don't think an idle meteor app would take 50% or CPU and 50% of RAM of it.

When I kill the processes, they come back.

Maybe it would be easier to backup the DB and reset the server from scratch ?

[jm4r7in]

I did:

mongo appName
use admin
db.adminCommand({shutdown : 1, force : true})

And the result is:

2015-08-25T17:42:35.530+0200 DBClientCursor::init call() failed
2015-08-25T17:42:35.531+0200 Error: error doing query: failed at src/mongo/shell/query.js:81
2015-08-25T17:42:35.532+0200 trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2015-08-25T17:42:35.532+0200 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2015-08-25T17:42:35.532+0200 reconnect 127.0.0.1:27017 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed

And also there are no more mongod instances. The CPU usage is much lower (< 1%).

Then I did:

kill -9 ps ax | grep node | grep meteor | awk '{print $1}'

I got:

-bash: kill: ps: arguments must be process or job IDs
-bash: kill: ax: arguments must be process or job IDs

[MasterJames]

As to differing syntax, it must be exclusive to mupx (which seems to currently help with more problems then it causes and worth trying). You have my oppologies as to your exact syntax it may need some massaging. Anyway if you deployed Mongo through mup you should be able to do "meteor mongo" (while it seems I can't type "mongo appname"?.)

Well somethings gone wrong. Your right to backup first. Only Then Try... "meteor reset" It's been known to help in such situations.

Still it seems like mongo is already running before meteor (try to massage the kill routine or google something to stop Meteor and it's Mongo and check for other instances.) I would also double check your backup/cron or disable it for now to make sure it's not going off here.

[jm4r7in]

I disabled the backup/cron and it didn't change anything.

Where do you suggest to try "meteor reset" ? If I do it on the server, it says "meteor: command not found". If I do it locally, I don't see how it can change the deployed server...

[jm4r7in]

If I kill one mongod instance, the CPU usage becomes normal and I get this:

[MasterJames]

I believe, it will change the meteor mongo db the next time you redeploy (to a fresh compatible instance ideally?). So after stopping Mongo and the system went nominal, did you look to see other versions of mongo still running and/or system working? Also noting maybe mupx stop mupx start mupx restart could be of help. Maybe you should SSH into the instance before you deploy with mup(x) and make sure it isn't running Mongo already? It looks like you have plenty of RAM in the instance. Is it 4GB?

[jm4r7in]

Before trying to fix something, I create an empty app (with only meteor create myApp) on a brand new server.

I followed exactly those steps : http://jm4r7in.github.io/deploy-a-meteor-js-application-on-ubuntu/

And here is what I got with htop:

Is it normal to have so many instances of mongod and node ?

I will try with mupx, because mup seems f*ed...

[jm4r7in]

I followed the migration guide to mupx, and now I've got:

After a restart (shutdown -r now):

I notice that there is only one instance of mongod left :) but after some time more instances appear :(

And is it normal to have so many instances of docker ?