invoker mounts /var/lib/docker/containers - prevents docker rm from working correctly

[domdom82]

When invoker is launched, it mounts a volume as

/var/lib/docker/containers:/containers

which is the folder where docker keeps filesystems of all its containers.

I presume this is used for the invoker to access the filesystems of launched actions.

However, when trying to remove other containers (like registrator, consul etc.) you get the following error messages:

Docker API Error: Unable to remove filesystem for 57f62f43c6280934c44be469f33709498ac82dd6c960ea5a2bf02ac924db502e: remove /var/lib/docker/containers/57f62f43c6280934c44be469f33709498ac82dd6c960ea5a2bf02ac924db502e/mqueue: device or resource busy"

[domdom82]

Recent discussion has brought up the problems with fetching logs from user containers making this a non-trivial issue.

Current ideas for mitigation would be:

• docker logs --since=nnn containerid to fetch logs instead. needs to be evaluated on performance impact.
• customizing log driver to write into a different file. no apparent option available at least for the json driver. we could only limit log size with regard to security concerns.

[dfederschmidt]

Some thoughts on this from a discussion with @domdom82.

• We should get rid of mounting /var/lib/docker/containers/ in order to get user container logs if possible
• We could introduce a daemon which runs on the host pipes user container logs from /var/lib/docker/containers to a separate directory which is then mounted as a volume by the invoker. This daemon could potentially leverage docker events.

This should prevent the error observed. As a step forward on this issue, I will try to reproduce the error in a minimal working example as a playground for experimenting with different approaches to make fast log extraction possible without mounting /var/lib/docker/containers.

[perryibm]

Some thoughts on this:

1. Why doesn't the error occur for the many other action containers which are routinely removed from the system?
2. An external agent can overcome the problem but will complicate our container-only principle and make deployment potentially complicated. The agent will itself need to be restarted and coordinated with the invoker as essentially part of the logic is being moved into another component.
3. If we separate out blackbox containers, it is feasible to avoid logging through docker by having the action container push out its logs by some other means. This is of course a lot more work but might give us much more control and splitting off blackbox means we are committed to multiple implementation strategies.

[dfederschmidt]

I was able to reproduce the issue using the following sequence of actions:

• Checkout openwhisk's master branch
• Create a new VM using the Vagrantfile in /tools/vagrant/custom and a cloudant db
• Deploy openwhisk via ant clean build deploy
• Execute docker stop for the registrator container
• Execute docker rm for the registrator container

Our issue seems to be related to https://github.com/docker/docker/issues/17823 and https://github.com/docker/docker/issues/17902 where host's /var/lib/docker/containers is also added as a data volume to a container and 1.9 is also used.

Next steps:

• Check whether issue occurs with docker 1.10 / other kernel versions
• Find a solution on how to go forward

[csantanapr]

@domdom82 Do you know if this still is an issue with recent changes and docker 1.12 ?

[markusthoemmes]

@domdom82 is this still applicable?

[domdom82]

@markusthoemmes nope working now.