kata-deploy container cannot run well when using kata as runtime

[TerrenceXu]

Hi, All I am following https://github.com/kata-containers/packaging/tree/master/kata-deploy to build kata-deploy environment in my environment (my kata version is 1.4.0).

When I run the katadocker/kata-deploy container by below command:

docker run -v /opt/kata:/opt/kata -v /var/run/dbus:/var/run/dbus -v /run/systemd:/run/systemd -v /etc/docker:/etc/docker -it katadocker/kata-deploy kata-deploy-docker install

It will return below error information. docker: Error response from daemon: OCI runtime create failed: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/docker/volumes/f7cf15b71e20d41bd428f26d491eaa18351e76eb25fc0ebffc239ed189fb24d9/_data\\\" to rootfs \\\"/run/kata-containers/shared/containers/fc64e3f4039ae555536e4a1cd09989e0e039d4f6924d241aafcc4b00691852dd/rootfs\\\" at \\\"/sys/fs/cgroup\\\" caused \\\"stat /var/lib/docker/volumes/f7cf15b71e20d41bd428f26d491eaa18351e76eb25fc0ebffc239ed189fb24d9/_data: no such file or directory\\\"\"": unknown.

Any one know the solution? Thanks!

[egernst]

Hey @TerrenceXu sorry for delay in responding.

I don't see any issues on my side. A couple of queries:

• what is your host OS?
• You already have Kata 1.4 installed on the system? What's your goal of running this -- to install an additional version of Kata on your system and configure docker to use both?
• What's your current default runtime before running this?

[egernst]

@TerrenceXu from doing a quick test, I'm going to assume you have the default-runtime currently to kata-runtime, which is 1.4 version.

When I run this workload with Kata as the runtime, I do see a failure. As a workaround, please try running with --runtime=runc instead. Ex:

docker run --runtime=runc -v /opt/kata:/opt/kata -v /var/run/dbus:/var/run/dbus -v /run/systemd:/run/systemd -v /etc/docker:/etc/docker -it katadocker/kata-deploy kata-deploy-docker install

[egernst]

@amshinde is this 9pfs bind mount issue, similar to https://github.com/kata-containers/runtime/issues/1299 ?

[amshinde]

@egernst No this is not a 9pfs bind-mount issue. Just took a look at the base image used by katadocker/kata-deploy and looks like this causes docker volume mount for /sys/fs/cgroups like so:

 {
            "destination": "/sys/fs/cgroup",
            "options": [
                "rbind"
            ],
            "source": "/var/lib/docker/volumes/afc891c78bb7c1965976cb844a2a474bf1e7e5a0ea8fda2011375086cf59ec0e/_data",
            "type": "bind"
        }

We handle the system mounts, passing them as is to the kata-agent ( so that the agent bind-mounts the guest system mount(/proc, sys) to the container namespace). This is a bug and we should not do so in case of docker volumes.

This wip patch should fix this.

diff --git a/virtcontainers/container.go b/virtcontainers/container.go
index 5e1cb96..5ec2bb9 100644
--- a/virtcontainers/container.go
+++ b/virtcontainers/container.go
@@ -487,7 +487,7 @@ func (c *Container) mountSharedDirMounts(hostSharedDir, guestSharedDir string) (
        var sharedDirMounts []Mount
        var ignoredMounts []Mount
        for idx, m := range c.mounts {
-               if isSystemMount(m.Destination) || m.Type != "bind" {
+               if (isSystemMount(m.Destination) && filepath.Base(m.Source) != "_data") || m.Type != "bind" {
                        continue
                }

On applying this patch, you will no longer see the mount error, but then I see this error: Failed to get D-Bus connection: Connection refused, since the container tries to access the host dbus.

The workload is clearly not suitable for Kata and we should document this.

[amshinde]

cc @sboeuf

[egernst]

Agreed. I'd suggest we mark as "will not fix" -- we don't want Kata to restart system services from inside the guest.

@amshinde is https://github.com/kata-containers/packaging/issues/364#issuecomment-469917872 this part actually a bug though? Should we open a seperate issue for this?

@TerrenceXu are you okay if I close as "will not fix?"

[TerrenceXu]

@egernst, hey, sorry for the late reply and thank you for your kindly reply. I am okay to close it as "will not fix".