Suddenly broke uploading packages (mupx)

[artpolikarpov]

I use a mupx for a long time without complaints, and suddenly in one of my projects uploading just stopped working.

All commands work — mupx logs, mupx reconfig. But mupx deploy doesn’t! I also can connect to this servers with my ssh-key. And I can deploy sibling-project to sibling-servers (in the same data-center, with the same ssh-key).

I have no idea what to check, because of poor error message. Please help?

[MasterJames]

This could also be caused by the mongod.lock file. Remove /var/lib/mongodb/mongod.lock and see if that helps. In mupx the Mongo container gets jammed by the volume mounted folders lock file. It's meant to preserve your database but this triggers a reboot loop of the docker images for mongodb and that becomes unconnectable so the meteor container also fails. I made a PR for Mongo Dump and Restore that includes tools to remove the locks. Also note you're suppose to do a manual repair when manually purging locks.

[artpolikarpov]

@MasterJames I do not install Mongo on that servers...

[arunoda]

Did you see the preloaded bar? If not try changing the network. On 2015 නොවැ 18, බදාදා at ප.ව. 4.37 Artem Polikarpov < notifications@github.com> wrote:

@MasterJames https://github.com/MasterJames I do not install Mongo on that servers...

— Reply to this email directly or view it on GitHub https://github.com/arunoda/meteor-up/issues/762#issuecomment-157680369.

[artpolikarpov]

Did you see the preloaded bar? If not try changing the network.

No! But in sibling-config (same ssh-key, same data-center for servers) everything works...

[arunoda]

Expose DEBUG=* to the console and run the deployment. It will give more info. On 2015 නොවැ 18, බදාදා at ප.ව. 5.03 Artem Polikarpov < notifications@github.com> wrote:

Did you see the preloaded bar? If not try changing the network.

No! But in sibling-config (same ssh-key, same data-center for servers) everything works...

— Reply to this email directly or view it on GitHub https://github.com/arunoda/meteor-up/issues/762#issuecomment-157684823.

[artpolikarpov]

Not much: http://d.pr/i/19T8P

It was working a few day ago. Config is the same.

[artpolikarpov]

Tried to mupx setup again on this servers, and got this:

Started TaskList: Setup (linux)
[104.197.83.173] - Installing Docker
[104.197.83.173] - Installing Docker: SUCCESS
[104.197.83.173] - Setting up Environment
[104.197.83.173] x Setting up Environment: FAILED

    -----------------------------------STDERR-----------------------------------
    usermod: /etc/passwd.3071: No space left on device
    usermod: cannot lock /etc/passwd; try again later.
    -----------------------------------STDOUT-----------------------------------
    ----------------------------------------------------------------------------

How to fix that? Clean up the logs or what?

[artpolikarpov]

The sudo df -h shows this:

[artpolikarpov]

It looks like we need to somehow clean up the /var/lib/docker/aufs/mnt folder...

[artpolikarpov]

After manual cleaning that folder and redeploying the app disk usage lowered to 24%:

[artpolikarpov]

I am prettry sure it happens because of growing logs:

How can we change this or config?

[artpolikarpov]

We can easily fix that by adding log rollover (see https://github.com/docker/docker/pull/11485). As I see it is all about few flags:

docker run --logging-driver rollover --log-driver-opts max_size=1k --log-driver-opts file_count=10

And also we have to add some new options to mup.json to make it configurable. What you think?

/@arunoda

[MasterJames]

Sounds important thanks for the input and discovery efforts. It's changed slightly in that discussion so in 1.8 it's --log-opt https://docs.docker.com/v1.8/reference/logging/overview/

[artpolikarpov]

I’m going to make a PR.

[arunoda]

Yeah! Send me that.

On Sun, Nov 22, 2015 at 6:55 PM Artem Polikarpov notifications@github.com wrote:

I’m going to make a PR.

— Reply to this email directly or view it on GitHub https://github.com/arunoda/meteor-up/issues/762#issuecomment-158759303.

[artpolikarpov]

Wrote and tested simple wrapper over --log-driver and --log-opt flags for configuring logging drivers.

Options in mup.json looks like this:

So now even possilbe to send logs to remote syslog/gelf servers natively!

If you’re ok with that API, I’ll send it to you soon :)

[arunoda]

Thanks. In the case of first one, we need to get logs via mup logs command.

Does it work ?

I assume this log-driver features (specially) comes with a later version of docker. So, we need to check in the deployment stage and warm and abort the process. On 2015 නොවැ 23, සඳුදා at පෙ.ව. 4.04 Artem Polikarpov < notifications@github.com> wrote:

Wrote and tested simple wrapper over --log-driver and --log-opt flags for configuring logging drivers.

Options in mup.json looks like this:

https://camo.githubusercontent.com/d5e407ca8507baa51ff227dd61265064a1dae4ec/687474703a2f2f642e70722f692f4931386c2b

https://camo.githubusercontent.com/81a8ee94e7b694c506e41706231f20cb1446c31d/687474703a2f2f642e70722f692f31664446612b

So now even possilbe to send logs to remote syslog/gelf servers natively!

If you’re ok with that API, I’ll send it to you soon :)

— Reply to this email directly or view it on GitHub https://github.com/arunoda/meteor-up/issues/762#issuecomment-158812333.

[artpolikarpov]

Well, mup logs command is available only for the json-file logging driver (it is default). It works if you specify max-size and max-file options. And it do not return anything if you change log.driver to syslog or whatever. This is the behavior of the Docker.

These features works since Docker v1.8 version, on servers we have 1.9.

[artpolikarpov]

I think it is ok to make a remark in docs that mup logs will be silent in case of alternative logging drivers. So it will be up to deployers.

[arunoda]

Then it's great. We don't need to get it when we are using alternate log drivers. Yeah! We can do a note. There are some older mup deployments may contains Docker versions other than 0.1.8. Then these options may not work, so in this case we need to warn the user.

Also, make sure to set of sensible defaults.

On Mon, Nov 23, 2015 at 4:36 AM Artem Polikarpov notifications@github.com wrote:

Well, mup logs command is available only for the json-file logging driver (it is default). It works if you specify max-size and max-file options. And it do not return anything if you change log.driver to syslog or whatever. This is the behavior of the Docker.

These features works since Docker v1.8 version, on servers we have 1.9.

— Reply to this email directly or view it on GitHub https://github.com/arunoda/meteor-up/issues/762#issuecomment-158817087.

[artpolikarpov]

I’ll leave defaults as is. If you don’t specify any of log options in the object, they will not be honored, and docker will work as now. I think it is no need to show all of these options in example/mup.json because for different drivers you have to specify different options and you can’t mix it.

I’m going to add a short section to a README.md about new possibilities.

About waring user in case of old Docker... We don’t have to do anything because Docker will do it for us: if you specify some unsupported arguments Docker doesn’t start and shows you why.

[artpolikarpov]

We also can add an another note in readme that these options works with fresh Docker.

P. S. I’m trying to solve this (not critical) issue as a designer and not like as a coder ;)

[arunoda]

Okay. That's okay about the docker client.

But we need to add some reasonable defaults to avoid the issues like you've faced. So, adding max-size=100m and max-files=10 seems good for me.

On Mon, Nov 23, 2015 at 4:52 AM Artem Polikarpov notifications@github.com wrote:

I’ll leave defaults as is. If you don’t specify any of log options in the object, they will not be honored, and docker will work as now. I think it is no need to show all of these options in example/mup.json because for different drivers you have to specify different options and you can’t mix it.

I’m going to add a short section to a README.md about new possibilities.

About waring user in case of old Docker... We don’t have to do anything because Docker will do it for us: if you specify some unsupported arguments Docker doesn’t start and shows you why.

— Reply to this email directly or view it on GitHub https://github.com/arunoda/meteor-up/issues/762#issuecomment-158817811.

[artpolikarpov]

Aha, I understand now. Ok, I’ll add it.

[arunoda]

Great.

On Mon, Nov 23, 2015 at 4:57 AM Artem Polikarpov notifications@github.com wrote:

Aha, I understand now. Ok, I’ll add it.

— Reply to this email directly or view it on GitHub https://github.com/arunoda/meteor-up/issues/762#issuecomment-158818066.