Shiftfs module not present in Ubuntu Cloud Images

[stahnma]

I'm unsure what I'm doing wrong here, but missing a shiftfs module.

root@system:~# docker run --runtime=sysbox-runc --rm -it --hostname my_cont debian:latest
docker: Error response from daemon: OCI runtime create failed: container requires user-ID shifting but error was found: shiftfs module is not loaded in the kernel. Update your kernel to include shiftfs module or enable Docker with userns-remap. Refer to the Sysbox troubleshooting guide for more info: unknown.
ERRO[0000] error waiting for container: context canceled```

[ctalledo]

Hi Michael,

This error means the Ubuntu kernel is a bit older than the one required by Sysbox, and therefore does not have the "shiftfs" module.

Two options to work-around this:

1) Upgrade your kernel to the latest Ubuntu version.

or

2) Configure the Docker daemon in userns-remap mode (via the /etc/docker/daemon.json file).

The first option is best if you want to run Nestybox system containers along side regular Docker containers. The second options is best as a quick workaround.

The Sysbox troubleshooting guide has all the info on how to overcome this problem:

https://github.com/nestybox/sysbox-external/blob/master/docs/troubleshoot.md#ubuntu-shiftfs-module-not-present

Let us know if you need more help please.

Regards,

• Cesar

On Tue, Dec 17, 2019 at 12:27 PM Michael Stahnke notifications@github.com wrote:

I'm unsure what I'm doing wrong here, but missing a shiftfs module.

root@system:~# docker run --runtime=sysbox-runc --rm -it --hostname my_cont debian:latest docker: Error response from daemon: OCI runtime create failed: container requires user-ID shifting but error was found: shiftfs module is not loaded in the kernel. Update your kernel to include shiftfs module or enable Docker with userns-remap. Refer to the Sysbox troubleshooting guide for more info: unknown. ERRO[0000] error waiting for container: context canceled```

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nestybox/sysbox-external/issues/29?email_source=notifications&email_token=ADCIOCFO3ENUICAWWKWISR3QZEYZRA5CNFSM4J4BCWEKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IBE6N7A, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADCIOCDMVB7E2BYBW6JJFTTQZEYZRANCNFSM4J4BCWEA .

-- Cesar Talledo | Founder & CEO | Nestybox Inc. | (408) 691-5060

[stahnma]

I'm on Ubuntu 19.04. 5.0.0-1026-gcp

  architecture => "amd64",
  distro => {
    codename => "disco",
    description => "Ubuntu 19.04",
    id => "Ubuntu",
    release => {
      full => "19.04",
      major => "19.04"
    }
  },
  family => "Debian",
  hardware => "x86_64",
  name => "Ubuntu",
  release => {
    full => "19.04",
    major => "19.04"
  },
  selinux => {
    enabled => false
  }
}```

[ctalledo]

I'm on Ubuntu 19.04. 5.0.0-1026-gcp

Thanks; looks like this is a very recent Ubuntu image for GCP (per the info in this site), but for some reason it does not include the Ubuntu shiftfs module (which is present in their latest server and desktop kernel images). We will dig around a bit to see what's going on.

[ctalledo]

As mentioned earlier, you can try upgrading the kernel with:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get dist-upgrade
$ reboot

Or alternatively you can configure the Docker daemon in userns-remap mode. To do this:

1) After installing sysbox, edit the /etc/docker/daemon.json file to add the "userns-remap" entry:

{
    "runtimes": {
        "sysbox-runc": {
            "path": "/usr/local/sbin/sysbox-runc"
        }
    },
    "userns-remap": "sysbox"
}

2) Restart Docker:

$ sudo systemctl restart docker

[ctalledo]

Michael,

Hopefully you were able to get past the error you were seeing; let us know if you need any help with this please. Otherwise we can close this issue.

Thanks! -Cesar

[ctalledo]

Closing.

[ikropotov]

@ctalledo I was poking the same issue, and it turned out that ubuntu-2004-focal-* images and even ubuntu-2010-groovy-v20201022a does not include the shiftfs module (I also tested AWS focal images, they have not got it as well)

However the ubuntu-1804-bionic-* ones has got it and works charmly, after sudo modprobe shiftfs

[rubenanapu]

I am testing Amazon's Ubuntu 20.04 and this problem is still happening:

$ uname -rs

Linux 5.8.0-1041-aws

The error is

ERROR: for CONTAINERNAME Cannot start service CONTAINERNAME: OCI runtime create failed: error in the container spec: this container requires user-ID shifting but the kernel does not support it. Upgrade your kernel to include the shiftfs module, or alternatively enable Linux user-namespace support in the the container manager (e.g., Docker userns-remap, CRI-O userns annotation, etc). Refer to the Sysbox troubleshooting guide for more info.: unknown ERROR: Encountered errors while bringing up the project.

And if I run sudo modprobe shiftfs I have:

modprobe: FATAL: Module shiftfs not found in directory /lib/modules/5.8.0-1041-aws

[ctalledo]

Hi @rubenanapu ,

Thanks for trying Sysbox.

The AWS Ubuntu 20.04 images do not carry the shiftfs module required by Sysbox (the Ubuntu desktop and server images do).

However, it's pretty easy to build & install the module as described here:

https://github.com/toby63/shiftfs-dkms/tree/k5.8#howto

For example:

git clone -b k5.8 https://github.com/toby63/shiftfs-dkms.git shiftfs-k58
cd shiftfs-k58
./update1
sudo make -f Makefile.dkms
modinfo shiftfs

Please follow these steps and let me know if you hit any issues.

Once shiftfs is installed, you can try docker run --runtime=sysbox-runc -it nestybox/ubuntu-focal-systemd-docker and it should work without problem.

[rubenanapu]

Hi @ctalledo,

thank you very much for replying. The commands you suggested did the trick:

git clone -b k5.8 https://github.com/toby63/shiftfs-dkms.git shiftfs-k58
cd shiftfs-k58
./update1
sudo make -f Makefile.dkms
modinfo shiftfs

Thank you very much for this great sysbox project.

[ctalledo]

Hi @rubenanapu ... happy that it worked! We will update our docs to include this info in the Sysbox installation instructions.