Unnamed volumes in OCI images are mounted with noexec

[carrete]

I'm using colima to test a deployment of mattermost with incus. Mattermost exports five volumes, https://github.com/mattermost/mattermost/blob/master/server/build/Dockerfile#L53, which are mounted inside the running mattermost container with the noexec option set when run with incus. This does not happen with the docker backend. Please see below.

This means plugins are not usable with Mattermost since they install executables which need to be run from under /mattermost/plugins.

This is on macOS ventura intel. The choice of filesystems doesn't appear to matter. Both btrfs and zfs behave the same, and if I remember correctly so does the dir backend. Ditto for qemu and vz.

Attempting to mount a volume at /mattermost/plugins results in an empty /mattermost/plugins directory.

I understand there are a lot of layers to this and this may not be related to incus, but my internet searches have turned up nothing, and the colima community doesn't think it's them. So here I am.

The output below was produced by exec'ing mount inside the running docker and mattermost containers.

With the docker backend:

/dev/root on /mattermost/client/plugins type ext4 (rw,relatime,discard,errors=remount-ro,commit=30)
/dev/root on /mattermost/config type ext4 (rw,relatime,discard,errors=remount-ro,commit=30)
/dev/root on /mattermost/data type ext4 (rw,relatime,discard,errors=remount-ro,commit=30)
/dev/root on /mattermost/logs type ext4 (rw,relatime,discard,errors=remount-ro,commit=30)
/dev/root on /mattermost/plugins type ext4 (rw,relatime,discard,errors=remount-ro,commit=30)

With the incus backend:

none on /mattermost/client/plugins type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000,inode64)
none on /mattermost/config type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000,inode64)
none on /mattermost/data type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000,inode64)
none on /mattermost/logs type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000,inode64)
none on /mattermost/plugins type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000,inode64)

[stgraber]

I don't have your particular image here so can't check for sure, but taking the mattermost/platform image from the DockerHub, we can see one such mount as defined (in the OCI config.json) as:

    {
      "destination": "/mm/mattermost-data",
      "type": "tmpfs",
      "source": "none",
      "options": [
        "rw",
        "nosuid",
        "nodev",
        "noexec",
        "relatime"
      ]
    }

So Incus is doing exactly what the OCI image definition tells it to do. The reason why it's different under Docker is that under Docker you have attached a persistent disk to each of those paths. I don't know if Docker automatically did this for you or if you did through the -V option (or equivalent in docker-compose.yml) but that's why the tmpfs isn't visible there.

You'll get the same behavior on Incus if you now do:

incus storage volume create default mattermost-data
incus config device add NAME client-plugins disk pool=default source=mattermost-data/client-plugins path=/mattermost/client/plugins
incus config device add NAME config disk pool=default source=mattermost-data/config path=/mattermost/config
incus config device add NAME data disk pool=default source=mattermost-data/data path=/mattermost/data
incus config device add NAME logs disk pool=default source=mattermost-data/logs path=/mattermost/logs
incus config device add NAME plugins disk pool=default source=mattermost-data/plugins path=/mattermost/plugins
incus restart NAME

That will allocate a persistent volume within Incus and then assign a sub-directory of it to each of those defined mount points.

[stgraber]

Closing as we're respecting the config.json from the OCI image.

[carrete]

Closing as we're respecting the config.json from the OCI image.

Interesting. Thanks for looking into this.

FYI, I started the containers with docker run --rm -it --name mattermost mattermost/mattermost-team-edition and incus launch docker:mattermost/mattermost-team-edition mattermost

[stgraber]

Interesting, so yeah, Docker must be automatically doing something when it sees that OCI data instead of what the OCI data actually says to do.

stgraber@dakara:~$ incus launch docker:mattermost/mattermost-team-edition mattermost
Launching mattermost
stgraber@dakara:~$ sudo cat /var/lib/incus/containers/mattermost/config.json 
{
    "ociVersion": "1.0.0",
    "process": {
        "terminal": true,
        "user": {
            "uid": 2000,
            "gid": 2000
        },
        "args": [
            "/entrypoint.sh",
            "mattermost"
        ],
        "env": [
            "PATH=/mattermost/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "TERM=xterm",
            "HOME=/mattermost"
        ],
        "cwd": "/mattermost",
        "capabilities": {
            "bounding": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ],
            "effective": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ],
            "inheritable": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ],
            "permitted": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ],
            "ambient": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ]
        },
        "rlimits": [
            {
                "type": "RLIMIT_NOFILE",
                "hard": 1024,
                "soft": 1024
            }
        ],
        "noNewPrivileges": true
    },
    "root": {
        "path": "rootfs"
    },
    "hostname": "umoci-default",
    "mounts": [
        {
            "destination": "/proc",
            "type": "proc",
            "source": "proc"
        },
        {
            "destination": "/dev",
            "type": "tmpfs",
            "source": "tmpfs",
            "options": [
                "nosuid",
                "strictatime",
                "mode=755",
                "size=65536k"
            ]
        },
        {
            "destination": "/dev/pts",
            "type": "devpts",
            "source": "devpts",
            "options": [
                "nosuid",
                "noexec",
                "newinstance",
                "ptmxmode=0666",
                "mode=0620",
                "gid=5"
            ]
        },
        {
            "destination": "/dev/shm",
            "type": "tmpfs",
            "source": "shm",
            "options": [
                "nosuid",
                "noexec",
                "nodev",
                "mode=1777",
                "size=65536k"
            ]
        },
        {
            "destination": "/dev/mqueue",
            "type": "mqueue",
            "source": "mqueue",
            "options": [
                "nosuid",
                "noexec",
                "nodev"
            ]
        },
        {
            "destination": "/sys",
            "type": "sysfs",
            "source": "sysfs",
            "options": [
                "nosuid",
                "noexec",
                "nodev",
                "ro"
            ]
        },
        {
            "destination": "/sys/fs/cgroup",
            "type": "cgroup",
            "source": "cgroup",
            "options": [
                "nosuid",
                "noexec",
                "nodev",
                "relatime",
                "ro"
            ]
        },
        {
            "destination": "/mattermost/logs",
            "type": "tmpfs",
            "source": "none",
            "options": [
                "rw",
                "nosuid",
                "nodev",
                "noexec",
                "relatime"
            ]
        },
        {
            "destination": "/mattermost/plugins",
            "type": "tmpfs",
            "source": "none",
            "options": [
                "rw",
                "nosuid",
                "nodev",
                "noexec",
                "relatime"
            ]
        },
        {
            "destination": "/mattermost/client/plugins",
            "type": "tmpfs",
            "source": "none",
            "options": [
                "rw",
                "nosuid",
                "nodev",
                "noexec",
                "relatime"
            ]
        },
        {
            "destination": "/mattermost/config",
            "type": "tmpfs",
            "source": "none",
            "options": [
                "rw",
                "nosuid",
                "nodev",
                "noexec",
                "relatime"
            ]
        },
        {
            "destination": "/mattermost/data",
            "type": "tmpfs",
            "source": "none",
            "options": [
                "rw",
                "nosuid",
                "nodev",
                "noexec",
                "relatime"
            ]
        }
    ],
    "annotations": {
        "org.opencontainers.image.architecture": "amd64",
        "org.opencontainers.image.author": "",
        "org.opencontainers.image.created": "2024-10-04T14:20:20.464588359Z",
        "org.opencontainers.image.exposedPorts": "8065/tcp,8067/tcp,8074/tcp,8075/tcp",
        "org.opencontainers.image.os": "linux",
        "org.opencontainers.image.ref.name": "ubuntu",
        "org.opencontainers.image.stopSignal": "",
        "org.opencontainers.image.version": "24.04"
    },
    "linux": {
        "resources": {
            "devices": [
                {
                    "allow": false,
                    "access": "rwm"
                }
            ]
        },
        "namespaces": [
            {
                "type": "pid"
            },
            {
                "type": "network"
            },
            {
                "type": "ipc"
            },
            {
                "type": "uts"
            },
            {
                "type": "mount"
            }
        ],
        "maskedPaths": [
            "/proc/kcore",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/sys/firmware",
            "/proc/scsi"
        ],
        "readonlyPaths": [
            "/proc/asound",
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ]
    }
}

So in this case we end up with:

        {
            "destination": "/mattermost/logs",
            "type": "tmpfs",
            "source": "none",
            "options": [
                "rw",
                "nosuid",
                "nodev",
                "noexec",
                "relatime"
            ]
        },
        {
            "destination": "/mattermost/plugins",
            "type": "tmpfs",
            "source": "none",
            "options": [
                "rw",
                "nosuid",
                "nodev",
                "noexec",
                "relatime"
            ]
        },
        {
            "destination": "/mattermost/client/plugins",
            "type": "tmpfs",
            "source": "none",
            "options": [
                "rw",
                "nosuid",
                "nodev",
                "noexec",
                "relatime"
            ]
        },
        {
            "destination": "/mattermost/config",
            "type": "tmpfs",
            "source": "none",
            "options": [
                "rw",
                "nosuid",
                "nodev",
                "noexec",
                "relatime"
            ]
        },
        {
            "destination": "/mattermost/data",
            "type": "tmpfs",
            "source": "none",
            "options": [
                "rw",
                "nosuid",
                "nodev",
                "noexec",
                "relatime"
            ]
        }

[stgraber]

So yeah, the OCI runtime config.json we're getting does instruct us to set up those tmpfs.

It's quite possible that the registry data tells Docker that those are volumes and so it automatically sets something up on there to persist those paths, but this may be pretty dependent on implementation so something like Kubernetes may behave differently.