standard_init_linux.go:211: exec user process caused "exec format error"

testautomation commented 4 years ago

Getting this error on a Pi 4 running hypriotOS

standard_init_linux.go:211: exec user process caused "exec format error"

the command I run

sudo docker run -v /mnt/SSD:/ssd -v /etc/exports:/etc/exports:ro \
    --privileged -p 2049:2049 \
    --name nfs \
    erichough/nfs-server

environment

$ uname -a
Linux black-pearl 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

$ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:37:22 2019
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:31:17 2019
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

testautomation commented 4 years ago

My fault. For some reason I assumed that your docker image was made to run on a Raspberry Pi. But of course this is not the case and the image from Docker Hub does not want to run on arm architecture.

Luckily there is a workaround: The solution is to build the docker image on your own RPi.

# clone this repo and build docker image locally
git clone https://github.com/ehough/docker-nfs-server.git
cd docker-nfs-server/
docker build -t wlad/nfs .

# run your own image instead the one from Docker Hub
sudo docker run -v /mnt/SSD:/ssd -v /etc/exports:/etc/exports:ro \
    --privileged -p 2049:2049 \
    --name nfs \
    wlad/nfs

testautomation commented 4 years ago

and now I kind of stuck

nfs_1  | ==================================================================
nfs_1  |       SETTING UP ...
nfs_1  | ==================================================================
nfs_1  | ----> setup complete
nfs_1  |
nfs_1  | ==================================================================
nfs_1  |       STARTING SERVICES ...
nfs_1  | ==================================================================
nfs_1  | ----> starting rpcbind
nfs_1  | ----> starting exportfs
nfs_1  | exportfs: Failed to stat ~/ssd: No such file or directory
nfs_1  | ---->
nfs_1  | ----> ERROR: /usr/sbin/exportfs failed
nfs_1  | ---->
nfs_1  |
nfs_1  | ==================================================================
nfs_1  |       TERMINATING ...
nfs_1  | ==================================================================
nfs_1  | ----> terminating nfsd
nfs_1  | ----> rpc.statd was not running
nfs_1  | ----> rpc.mountd was not running
nfs_1  | ----> un-exporting filesystem(s)
nfs_1  | ----> terminating rpcbind
nfs_1  |
nfs_1  | ==================================================================
nfs_1  |       TERMINATED
nfs_1  | ==================================================================

ehough commented 4 years ago

Thanks for the report. I think providing ARM support for this image would be a great addition. But in the meantime let's see if we can get you up and running ...

exportfs: Failed to stat ~/ssd: No such file or directory

I would try using debug mode to see if that reveals anything further. Feel free to post the log here and I'll be happy to take a look. My guess is that your exports aren't quite right.

testautomation commented 4 years ago

ok, here is my docker-compose.yml

version: "3"
services:
  nfs:
    image: wlad/nfs
    environment:
      - NFS_LOG_LEVEL: DEBUG
    volumes:
      - ~/ssd:/nfs
      - /etc/exports:/etc/exports:ro
      - /lib/modules:/lib/modules:ro
    cap_add:
      - SYS_ADMIN
      - SYS_MODULE
    ports:
      - 2049:2049

I get a strange error when try to run with debug mode

$ docker-compose up
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.nfs.environment contains {"NFS_LOG_LEVEL": "DEBUG"}, which is an invalid type, it should be a string

In next comment I will run without docker-compose

testautomation commented 4 years ago

Here a run without docker-compose - with debug mode on:

$ docker run                        \
>   -e NFS_LOG_LEVEL=DEBUG          \
>   -v /home/ssd:/ssd               \
>   -v /etc/exports:/etc/exports:ro \
>   --cap-add SYS_ADMIN             \
>   -p 2049:2049                    \
>   wlad/nfs

==================================================================
      SETTING UP ...
==================================================================
----> log level set to DEBUG
----> will use 4 rpc.nfsd server thread(s) (1 thread per CPU)
----> /etc/exports is bind-mounted
----> kernel module nfs is loaded
----> kernel module nfsd is loaded
----> setup complete

==================================================================
      STARTING SERVICES ...
==================================================================
----> mounting rpc_pipefs filesystem onto /var/lib/nfs/rpc_pipefs
mount: mount('rpc_pipefs','/var/lib/nfs/rpc_pipefs','rpc_pipefs',0x00008000,'(null)'):0
----> mounting nfsd filesystem onto /proc/fs/nfsd
mount: mount('nfsd','/proc/fs/nfsd','nfsd',0x00008000,'(null)'):0
----> starting rpcbind
----> starting exportfs
exportfs: Failed to stat /home/ssd: No such file or directory
exportfs: Failed to stat /mnt/nfsshare: No such file or directory
exporting *:/home/ssd
exporting *:/mnt/nfsshare
---->
----> ERROR: /usr/sbin/exportfs failed
---->

==================================================================
      TERMINATING ...
==================================================================
----> terminating nfsd
----> rpc.statd was not running
----> rpc.mountd was not running
----> un-exporting filesystem(s)
----> terminating rpcbind
----> un-mounting nfsd filesystem from /proc/fs/nfsd
----> un-mounting rpc_pipefs filesystem from /var/lib/nfs/rpc_pipefs

==================================================================
      TERMINATED
==================================================================

This is what my exports file looks like:

$ cat /etc/exports
/mnt/nfsshare *(rw,all_squash,insecure,async,no_subtree_check,anonuid=1000,anongid=1000)
/home/ssd *(rw,all_squash,insecure,async,no_subtree_check,anonuid=1000,anongid=1000)

And here a proof that the folders exist:

HypriotOS/armv7: pirate@black-pearl in ~/projects/docker-nfs-server
$ ls -la ~/
total 56
drwxr-xr-x  7 pirate pirate 4096 Dec 21 22:13 .
drwxr-xr-x  5 root   root   4096 Dec 21 22:04 ..
-rw-------  1 pirate pirate 7955 Dec 21 04:19 .bash_history
-rw-r--r--  1 pirate pirate  220 Apr 18  2019 .bash_logout
-rw-r--r--  1 pirate pirate 2799 Jul 21 09:50 .bash_prompt
-rw-r--r--  1 pirate pirate  596 Dec 21 00:24 .bashrc
drwxr-xr-x 11 pirate pirate 4096 Dec 17 23:19 .fzf
-rw-r--r--  1 pirate pirate  334 Dec 17 23:20 .fzf.bash
drwx------  3 pirate pirate 4096 Dec 17 00:22 .gnupg
drwxr-xr-x  3 pirate pirate 4096 Dec 21 00:23 .local
-rw-r--r--  1 pirate pirate  476 Jul 21 09:50 .profile
drwxr-xr-x  5 pirate pirate 4096 Dec 21 00:23 projects
drwxr-xr-x  2 pirate pirate 4096 Dec 21 01:17 ssd
HypriotOS/armv7: pirate@black-pearl in ~/projects/docker-nfs-server
$ ls -la /mnt/
total 20
drwxr-xr-x  5 root   root   4096 Dec 21 01:31 .
drwxr-xr-x 23 root   root   4096 Dec 18 14:17 ..
drwxrwxrwx  2 pirate pirate 4096 Dec 21 02:51 nfsshare
drwxr-xr-x  2 root   root   4096 Dec 20 23:30 SSD
drwxr-xr-x  2 root   root   4096 Dec 21 00:51 wlad

Can you see what may cause the issue?

ehough commented 4 years ago

I get a strange error when try to run with debug mode


$ docker-compose up
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.nfs.environment contains {"NFS_LOG_LEVEL": "DEBUG"}, which is an invalid type, it should be a string

Try

environment:
  - NFS_LOG_LEVEL=DEBUG

instead of

environment:
  - NFS_LOG_LEVEL: DEBUG

I believe that docker-compose's behavior varies slightly based on which version you declare. I always set

version: '3.7'

at the top of my docker-compose.yml files.

Can you see what may cause the issue?

Yes! Remember that your exports should only reference container directories.

As an example let's examine your /home/ssd directory. That's a host directory that you're bind-mounting into the container at /ssd. But then in your exports you're telling the NFS server that you want to export /home/ssd, which doesn't exist in the container. The solution is to change that line of your exports from:

/home/ssd *(rw,all_squash,insecure,async,no_subtree_check,anonuid=1000,anongid=1000)

to

/ssd *(rw,all_squash,insecure,async,no_subtree_check,anonuid=1000,anongid=1000)

Do the same for your other bind-mount and you should be in business. Give it a try and let us know?

ehough commented 4 years ago

Were you able to get thing working?

testautomation commented 4 years ago

environment:
  - NFS_LOG_LEVEL=DEBUG

:+1: :heavy_check_mark:

my updated docker-compose file

version: "3.7"
services:
  nfs:
    image: wlad/nfs
    environment:
      - NFS_LOG_LEVEL=DEBUG
    volumes:
      - /home/ssd:/ssd
      - /etc/exports:/etc/exports:ro
      - /lib/modules:/lib/modules:ro
    cap_add:
      - SYS_ADMIN
      - SYS_MODULE
    ports:
      - 2049:2049

my updated /etc/exports file

/ssd *(rw,all_squash,insecure,async,no_subtree_check,anonuid=1000,anongid=1000)

:heavy_check_mark:

$ docker-compose up
Creating network "docker-nfs-server_default" with the default driver
Creating docker-nfs-server_nfs_1 ... done
Attaching to docker-nfs-server_nfs_1
nfs_1  | 
nfs_1  | ==================================================================
nfs_1  |       SETTING UP ...
nfs_1  | ==================================================================
nfs_1  | ----> log level set to DEBUG
nfs_1  | ----> will use 4 rpc.nfsd server thread(s) (1 thread per CPU)
nfs_1  | ----> /etc/exports is bind-mounted
nfs_1  | ----> kernel module nfs is loaded
nfs_1  | ----> kernel module nfsd is loaded
nfs_1  | ----> setup complete
nfs_1  | 
nfs_1  | ==================================================================
nfs_1  |       STARTING SERVICES ...
nfs_1  | ==================================================================
nfs_1  | ----> mounting rpc_pipefs filesystem onto /var/lib/nfs/rpc_pipefs
nfs_1  | mount: mount('rpc_pipefs','/var/lib/nfs/rpc_pipefs','rpc_pipefs',0x00008000,'(null)'):0
nfs_1  | ----> mounting nfsd filesystem onto /proc/fs/nfsd
nfs_1  | mount: mount('nfsd','/proc/fs/nfsd','nfsd',0x00008000,'(null)'):0
nfs_1  | ----> starting rpcbind
nfs_1  | ----> starting exportfs
nfs_1  | exporting *:/ssd
nfs_1  | ----> starting rpc.mountd on port 32767
nfs_1  | ----> starting rpc.statd on port 32765 (outgoing from port 32766)
nfs_1  | ----> starting rpc.nfsd on port 2049 with 4 server thread(s)
nfs_1  | rpc.nfsd: knfsd is currently down
nfs_1  | rpc.nfsd: Writing version string to kernel: -2 +3 +4 +4.1 +4.2
nfs_1  | rpc.nfsd: Created AF_INET TCP socket.
nfs_1  | rpc.nfsd: Created AF_INET UDP socket.
nfs_1  | rpc.nfsd: Created AF_INET6 TCP socket.
nfs_1  | rpc.nfsd: Created AF_INET6 UDP socket.
nfs_1  | ----> all services started normally
nfs_1  | 
nfs_1  | ==================================================================
nfs_1  |       SERVER STARTUP COMPLETE
nfs_1  | ==================================================================
nfs_1  | ----> list of enabled NFS protocol versions: 4.2, 4.1, 4, 3
nfs_1  | ----> list of container exports:
nfs_1  | ---->   /ssd   *(rw,async,wdelay,hide,nocrossmnt,insecure,root_squash,all_squash,no_subtree_check,secure_locks,acl,no_pnfs,anonuid=1000,anongid=1000,sec=sys,rw,insecure,root_squash,all_squash)
nfs_1  | ----> list of container ports that should be exposed:
nfs_1  | ---->   111 (TCP and UDP)
nfs_1  | ---->   2049 (TCP and UDP)
nfs_1  | ---->   32765 (TCP and UDP)
nfs_1  | ---->   32767 (TCP and UDP)
nfs_1  | 
nfs_1  | ==================================================================
nfs_1  |       READY AND WAITING FOR NFS CLIENT CONNECTIONS
nfs_1  | ==================================================================
nfs_1  | rpc.statd: Version 2.3.4 starting
nfs_1  | rpc.statd: Flags: No-Daemon Log-STDERR TI-RPC 
nfs_1  | rpc.statd: Failed to read /var/lib/nfs/state: Address in use
nfs_1  | rpc.statd: Initializing NSM state
nfs_1  | rpc.statd: Local NSM state number: 3
nfs_1  | rpc.statd: Failed to open /proc/sys/fs/nfs/nsm_local_state: Read-only file system
nfs_1  | rpc.statd: Running as root.  chown /var/lib/nfs to choose different user
nfs_1  | rpc.statd: Waiting for client connections

That looks better than my last attempt. Only a few things fail - not sure how critical they are, i.e. rpc.statd: Failed to read /var/lib/nfs/state: Address in use and rpc.statd: Failed to open /proc/sys/fs/nfs/nsm_local_state: Read-only file system

And I can't establish a connection from a client:

I have tried nfs://192.168.178.52/home/ssd and nfs://192.168.178.52/ssd

Found this line in your readme but not sure how to adjust it correctly

mount <container-IP>:/some/export /some/local/path

Then I try sudo mount 192.168.178.52:/ssd /mnt/extssd I get this error

mount: /mnt/extssd: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

ehough commented 4 years ago

Looking better! I'd say your server setup is done.

Then I try sudo mount 192.168.178.52:/ssd /mnt/extssd I get this error

mount: /mnt/extssd: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

Your mount syntax and arguments look good. On Debian-based distributions, you would need to install the nfs-common package in order to mount an NFS filesystem; it contains the /sbin/mount.nfs* binaries that the mount command uses. Not sure which OS your client is using but you would need to install whichever package supplies those executables.

You could also add -vvv to your mount command for extra verbosity.

testautomation commented 4 years ago

I am using Pop-OS as client (Ubuntu based). Thus I was able to sudo apt install nfs-common package easily.

$ sudo mount 192.168.178.52:/ssd -vvv /mnt/extssd

mount.nfs: timeout set for Fri Jan  3 14:05:28 2020
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.178.52,clientaddr=192.168.178.29'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'addr=192.168.178.52'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: requested NFS version or transport protocol is not supported

ehough commented 4 years ago

I think we're close to figuring this out. Seems to me like 99% of the solution is in place, so we just have to iron out the last detail.

In your docker-compose.yml, you're opening only port 2049. This means that you'll only be able to use NFSv4. NFSv4 uses a pseudo file system which, IMO, only makes mounting from a client more confusing.

I would next try one of the following:

sudo mount 192.168.178.52:/ -vvv /mnt/extssd. I think this will work.
If that doesn't work, try dropping to NFSv3 by adding the relevant ports to your docker-compose.yml. See the "Expose the server ports" docs in this section. In my experience the performance difference between versions 3 and 4 is negligible.

Give it a try and let me know?

testautomation commented 4 years ago

sorry to say I have no better news this time :(

I have tried both of your last suggestions with no success.

for 1. I get following results

# from Ubuntu (WSL on Windows)
$ sudo mount 192.168.178.52:/ -vvv /mnt/extssd
mount.nfs: timeout set for Sat Jan 18 21:51:48 2020
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.178.52,clientaddr=192.168.178.48'
mount.nfs: mount(2): No such device
mount.nfs: No such device

# from Windows
PS C:\Users\wlad> net use x: \\192.168.178.52\ssd
System error 67 has occurred.
The network name cannot be found.

for 2.

made changes to docker-compose.yml (port 111 was already in use)

ports:
      - 2049:2049
      - 2049:2049/udp
      - 32765:32765
      - 32765:32765/udp
      - 32767:32767
      - 32767:32767/udp

but got same results as in 1.

Also did rebuild the docker image with

EXPOSE 2049
EXPOSE 2049/udp
EXPOSE 111
EXPOSE 111/udp
EXPOSE 32765
EXPOSE 32765/udp
EXPOSE 32767
EXPOSE 32767/udp

but again same results as in 1.

Thank you very much! I appreciate your support but I dont' want to bother you any more and would like to stop here cause I found an easier solution that fits my needs - samba :)

Here is an article which helped me to set up a nsf server manually (without docker): https://pimylifeup.com/raspberry-pi-nfs/. May be it is useful for somebody or can even help further investigae this issue.

Let me know any time in case you need help with testing an ARM based image. I will be really glad to help.

❤️ ❤️ ❤️

p.s. I think as far as the initial post is concerned (exec user process caused "exec format error") this issue can be closed.

ehough commented 4 years ago

Well I'm bummed that we didn't get to the bottom of your issue, but I'm glad you found something that worked for you in the meantime.

I've opened a new issue to track progress on building an ARM image. Feel free to subscribe to that issue for updates as I work on it.

Thanks!

ehough / docker-nfs-server

standard_init_linux.go:211: exec user process caused "exec format error" #32

for 1. I get following results

for 2.