rpc faults && permission denied issues

[theriverman]

Hi,

First of all thanks for the image, it is indeed a very good idea containerizing this service.

However, I'm struggling to make it run on my VPS.

Below is my console log, when trying to make it come alive.

root@ourkid /opt/nfs # pwd && ls -lh && cat exports.txt && cat startContainer.sh && ./startContainer.sh
/opt/nfs
total 12K
drw-rw---- 6 root root 4.0K Jun 15 17:11 codeDevelopment
-rw-rw---- 1 root root   48 Jun 16 13:32 exports.txt
-rwxr-xr-x 1 root root  190 Jun 16 13:34 startContainer.sh
/opt/nfs/codeDevelopment *(rw,no_subtree_check)
#!/bin/bash

docker run                                      \
  -v /opt/nfs/exports.txt:/etc/exports:ro       \
  -v /opt/nfs/codeDevelopment:/nfs              \
  --cap-add SYS_ADMIN                           \
  -p 2049:2049                                  \
  erichough/nfs-server:latest

==================================================================
      SETTING UP
==================================================================
----> /etc/exports already exists in the container
----> checking for presence of kernel module: nfs
----> checking for presence of kernel module: nfsd
----> setup complete

==================================================================
      STARTING SERVICES
==================================================================
----> mounting rpc_pipefs onto /var/lib/nfs/rpc_pipefs
mount: rpc_pipefs is write-protected, mounting read-only
mount: cannot mount rpc_pipefs read-only
----> unable to mount rpc_pipefs onto /var/lib/nfs/rpc_pipefs

==================================================================
      TERMINATING ...
==================================================================
----> rpc.svcgssd was not running
----> stopping nfsd
----> WARNING: unable to stop nfsd. if it had started already, check Docker host for lingering [nfsd] processes
----> rpc.idmapd was not running
----> rpc.statd was not running
----> rpc.mountd was not running
----> un-exporting filesystems
exportfs: could not open /proc/fs/nfs/exports for locking: errno 13 (Permission denied)
----> rpcbind was not running
----> nfsd was not mounted on /proc/fs/nfsd
----> rpc_pipefs was not mounted on /var/lib/nfs/rpc_pipefs

==================================================================
      TERMINATED
==================================================================
root@ourkid /opt/nfs # dr ps -a
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                      PORTS               NAMES
db982c254a00        erichough/nfs-server:latest   "/usr/local/bin/entr…"   11 seconds ago      Exited (0) 11 seconds ago                       adoring_jennings
root@ourkid /opt/nfs #

am I missing something totally obvious? I had to execute modprobe nfs && modprobe nfsd before getting stuck at this point.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04 LTS
Release:        18.04
Codename:       bionic

Thanks a lot for your help in advance!

[ehough]

Thanks for the report. You're definitely not missing anything obvious; I'm also a little stumped.

mount: rpc_pipefs is write-protected, mounting read-only mount: cannot mount rpc_pipefs read-only could not open /proc/fs/nfs/exports for locking: errno 13 (Permission denied)

These filesystem permissions issues are clearly the cause. It seems like the mounts (required by nfsd) aren't allowed, even though --cap-add SYS_ADMIN should permit them.

I'll spin up an Ubuntu vm to try to reproduce this. In the meantime, would you:

1. Send along the output of docker info so I can ensure we have the same environment and
2. Try using --privileged instead of --cap-add SYS_ADMIN, just to see if it's a Docker permissions issue.

We should be able to figure this out.

[theriverman]

Tried --privileged, no luck.

docker info below:

Containers: 16
 Running: 15
 Paused: 0
 Stopped: 1
Images: 99
Server Version: 17.12.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: v0.13.0 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-23-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.604GiB
Name: ourkid
ID: <lots_of_random_strings>
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[ehough]

Thank you. Do you have AppArmor enabled by chance?

sudo aa-status

[theriverman]

It is loaded, indeed:

root@ourkid ~ # sudo aa-status
apparmor module is loaded.
22 profiles are loaded.
22 profiles are in enforce mode.
   /sbin/dhclient
   /snap/core/4650/usr/lib/snapd/snap-confine
   /snap/core/4650/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/lxc-start
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/tcpdump
   docker-default
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
   man_filter
   man_groff
   snap-update-ns.core
   snap-update-ns.jq
   snap.core.hook.configure
   snap.jq.jq
0 profiles are in complain mode.
72 processes have profiles defined.
72 processes are in enforce mode.
   /sbin/dhclient (865)
   docker-default (5114)
   docker-default (5117)
   docker-default (5119)
   docker-default (5121)
   docker-default (5128)
   docker-default (9193)
   docker-default (9261)
   docker-default (9314)
   docker-default (9315)
   docker-default (10826)
   docker-default (11335)
   docker-default (11354)
   docker-default (11358)
   docker-default (11360)
   docker-default (11361)
   docker-default (11362)
   docker-default (11363)
   docker-default (12620)
   docker-default (12788)
   docker-default (12789)
   docker-default (12790)
   docker-default (12791)
   docker-default (12792)
   docker-default (13806)
   docker-default (13892)
   docker-default (14768)
   docker-default (14769)
   docker-default (15963)
   docker-default (16076)
   docker-default (16222)
   docker-default (16238)
   docker-default (19667)
   docker-default (19729)
   docker-default (20031)
   docker-default (20154)
   docker-default (20230)
   docker-default (20255)
   docker-default (20326)
   docker-default (20328)
   docker-default (20331)
   docker-default (20347)
   docker-default (20435)
   docker-default (20606)
   docker-default (20721)
   docker-default (20812)
   docker-default (20813)
   docker-default (21123)
   docker-default (21174)
   docker-default (21175)
   docker-default (21176)
   docker-default (21177)
   docker-default (22731)
   docker-default (22776)
   docker-default (22870)
   docker-default (22871)
   docker-default (22872)
   docker-default (22955)
   docker-default (22964)
   docker-default (22965)
   docker-default (22966)
   docker-default (22967)
   docker-default (22970)
   docker-default (23130)
   docker-default (24111)
   docker-default (24375)
   docker-default (24513)
   docker-default (24752)
   docker-default (25182)
   docker-default (27607)
   docker-default (28151)
   docker-default (31713)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

I tried executing service apparmor stop service apparmor teardown

but no luck

[ehough]

OK, thanks. I'm still trying to reproduce the issue on my end. Should have it figured out soon. Stand by!

[ehough]

I'm able to reproduce the error. Now to find the fix ...

[ehough]

Got it to work. AppArmor is indeed not allowing the container to perform the required mounts. We can supply a custom AppArmor profile for the container to use. Here's how:

WARNING I am not very familiar with AppArmor and cannot make any claims of the security impact of these instructions. However, this should be quite safe.

1. Ensure you have apparmor-utils installed:

   
   sudo apt-get install apparmor-utils

2. Create a file on the Docker host with the following contents:

   #include <tunables/global>
   profile erichough-nfs flags=(attach_disconnected,mediate_deleted) {
     #include <abstractions/lxc/container-base>
     mount fstype=nfs*,
     mount fstype=rpc_pipefs,
   }

3. Load this profile into AppArmor:

   
   sudo apparmor_parser -r -W /path/to/file/from/previous/step

4. Add --security-opt apparmor=erichough-nfs to your docker run command. e.g.

   
   docker run                                \
     -v /path/to/exports.txt:/etc/exports:ro \
     -v /path/to/share:/nfs                  \
     --cap-add SYS_ADMIN                     \
     -p 2049:2049                            \
     --security-opt apparmor=erichough-nfs   \
     erichough/nfs-server

Give it a try and let me know? If it works for you, I'll add these instructions to the README. Thanks.

[ehough]

Any luck? I'd love to add these instructions to the README, and am curious if they worked for you as well.

[michaelrcarroll]

For what it's worth: I was experiencing the same issues described as @theriverman. The apparmor config did the trick and solved my issue.

As an extra-added bonus, here's my incredibly vanilla docker-compose config if you're looking for an example:

version: "2"

services:
  nfs:
    image: erichough/nfs-server
    volumes:
      - ./files:/nfs/files
      - ./exports:/etc/exports
    ports:
      - 2049:2049
    cap_add:
      - SYS_ADMIN
    security_opt:
      - apparmor=erichough-nfs

[ehough]

@michaelrcarroll that's great to hear, thank you! I've added some docs for AppArmor, including a sample docker-compose.yml based on yours.

Closing this issue but please feel free to re-open or continue the discussion. Thanks, all!

[Chrislevi]

Hey, @ehough. I'm having the same issue on Centos 7 though. The selinux is disabled but still having the same error. How can I achieve the same thing without selinux enabled..?

[ehough]

@Chrislevi could you open a new issue with further details of your setup along with what errors you're seeing? I'd be very interested in getting the image to work under selinux as well as apparmor. Thanks!

[cybericius]

Any idea (/path/to/file/from/previous/step=.apparmor)? AppArmor parser error for .apparmor in profile .apparmor at line 3: Could not open 'abstractions/lxc/container-base'

[cybericius]

Any idea (/path/to/file/from/previous/step=.apparmor)? AppArmor parser error for .apparmor in profile .apparmor at line 3: Could not open 'abstractions/lxc/container-base'

Solution: sudo apt install lxc