search

droboports / nfs

NFS Server DroboApp build scripts
GNU General Public License v2.0
10 stars 3 forks source link

[DroboFS] - V4 Requires TCP Protocol #2

Closed coolacid closed 9 years ago

coolacid commented 9 years ago

Commenting out #1 I end up with:

Forcing V3 yields ::

bash-4.3# /mnt/DroboFS/Shares/DroboApps/nfs/sbin/rpc.nfsd -d 3 -N 4 rpc.nfsd: Checking netconfig for visible protocols. rpc.nfsd: knfsd is currently down rpc.nfsd: Writing version string to kernel: -2 +3 -4 rpc.nfsd: unable to set any sockets for nfsd

rpcinfo ::

bash-4.3# ./rpcinfo
rpcinfo: can't contact rpcbind: RPC: Remote system error - No such file or directory bash-4.3# ./rpcinfo -p status rpcinfo: couldn't find a suitable transport

ricardopadilha commented 9 years ago

Is rpcbind running? Could you please post your /tmp/DroboApps/nfs/log.txt here?

ricardopadilha commented 9 years ago

Also: do you have a file called /mnt/DroboFS/Shares/DroboApps/nfs/etc/netconfig?

coolacid commented 9 years ago

cat /mnt/DroboFS/Shares/DroboApps/nfs/etc/netconfig exits.

rpcbind is not running, and running it manually doesn't start it.

Both nfsd and exportfs kernel modules are running.

log file : 

2015-02-23 06-24-12: ./service.sh start
+ _service_start
+ _is_running
+ _is_nfsd_running
+ /bin/ps w
+ /bin/grep [n]fsd
+ return 1
+ set +x
+ /bin/chmod 4511 /mnt/DroboFS/Shares/DroboApps/nfs/sbin/mount.nfs
+ chown -R nobody /mnt/DroboFS/Shares/DroboApps/nfs/var/lib/nfs/sm /mnt/DroboFS/Shares/DroboApps/nfs/var/lib/nfs/sm.bak /mnt/DroboFS/Shares/DroboApps/nfs/var/lib/nfs/state
+ lsmod
+ grep ^exportfs
+ [[ -z exportfs 4832 1 nfsd, Live 0x7f000000 ]]
+ lsmod
+ grep ^nfsd
+ [[ -z nfsd 105732 0 - Live 0x7f003000 ]]
+ grep ^nfsd /proc/mounts
+ [[ -z  ]]
+ /bin/mount -t nfsd nfsd /proc/fs/nfsd
+ _kill_pid /mnt/DroboFS/Shares/DroboApps/nfs/sbin/sm-notify /tmp/DroboApps/nfs/sm-notify.pid
+ _is_pid_running /mnt/DroboFS/Shares/DroboApps/nfs/sbin/sm-notify /tmp/DroboApps/nfs/sm-notify.pid
+ /sbin/start-stop-daemon -K -s 0 -x /mnt/DroboFS/Shares/DroboApps/nfs/sbin/sm-notify -p /tmp/DroboApps/nfs/sm-notify.pid -q
+ _is_pid_running /mnt/DroboFS/Shares/DroboApps/nfs/bin/rpcbind /tmp/DroboApps/nfs/rpcbind.pid
+ /sbin/start-stop-daemon -K -s 0 -x /mnt/DroboFS/Shares/DroboApps/nfs/bin/rpcbind -p /tmp/DroboApps/nfs/rpcbind.pid -q
+ /mnt/DroboFS/Shares/DroboApps/nfs/bin/rpcbind -d
+ echo 622
+ sleep 1
+ _is_pid_running /mnt/DroboFS/Shares/DroboApps/nfs/sbin/rpc.mountd /tmp/DroboApps/nfs/pid.txt
+ /sbin/start-stop-daemon -K -s 0 -x /mnt/DroboFS/Shares/DroboApps/nfs/sbin/rpc.mountd -p /tmp/DroboApps/nfs/pid.txt -q
+ /mnt/DroboFS/Shares/DroboApps/nfs/sbin/rpc.mountd -F
rpc.mountd: Failed to access local netconfig database: Netconfig database not found
rpc.mountd: Failed to access local netconfig database: Netconfig database not found
rpc.mountd: Failed to access local netconfig database: Netconfig database not found
rpc.mountd: mountd: No V2 or V3 listeners created!
rpc.mountd: Kernel does not have pseudo root support.
rpc.mountd: NFS v4 mounts will be disabled unless fsid=0
rpc.mountd: is specfied in /etc/exports file.
rpc.mountd: Version 1.3.2 starting
+ echo 625
+ sleep 1
+ _is_pid_running /mnt/DroboFS/Shares/DroboApps/nfs/sbin/rpc.statd /tmp/DroboApps/nfs/rpc.statd.pid
+ /sbin/start-stop-daemon -K -s 0 -x /mnt/DroboFS/Shares/DroboApps/nfs/sbin/rpc.statd -p /tmp/DroboApps/nfs/rpc.statd.pid -q
+ /mnt/DroboFS/Shares/DroboApps/nfs/sbin/rpc.statd -F
+ echo 628
+ sleep 1
+ _is_nfsd_running
+ /bin/ps w
+ /bin/grep [n]fsd
+ /mnt/DroboFS/Shares/DroboApps/nfs/sbin/rpc.nfsd -d 3
rpc.nfsd: Checking netconfig for visible protocols.
rpc.nfsd: version 4 requires the TCP protocol
ricardopadilha commented 9 years ago

Ok, so the problem is rpcbind. Until that one is running, none of the rest will work properly.

coolacid commented 9 years ago

Agreed - I've set it aside for a moment (need my drobo) -- I can't seem to get rpcbind to return any errors or logs. (rpcbind -d or -ad yield nothing)

ricardopadilha commented 9 years ago

I'm going to publish a new pre-release version for the DroboFS, replacing the current one. Give it a try when you have the time.

flakrat commented 9 years ago

I'm experiencing the same issue on my 5N running the latest firmware (3.2.0 [8.45.72385]) and NFS 1.3.2. See my post for full details: http://www.drobospace.com/forums/showthread.php?tid=143410&pid=185877#pid185877

Everything I see in this ticket seems to apply to mine, rpcbind won't start and doesn't log anything when run manually. I do see a PID number in /tmp/DroboApps/nfs/rpcbind.pid, but by the time I've ssh'd in, the process is not running.

flakrat commented 9 years ago

I tried downgrading from 3.2.0 to 3.1.1 and still get the same result with NFS 1.3.2. Oh well, was worth a shot.

A word of caution for anyone considering downgrading firmware on the 5N, your file system may not be usable after downgrading! Mine wasn't, but it was new so I didn't lose anything.

ricardopadilha commented 9 years ago

There are still three pieces of information that could help shed some light into this:

1) The last 50 lines of dmesg: dmesg | tail -n 50

2) The content of /proc/filesystems

3) The content of /proc/mounts

flakrat commented 9 years ago

Howdy, here's the data after a fresh reboot

$ dmesg | tail -n 50
[2015-03-04,13:54:36.92]      host=shockwave, domain=, nis-domain=(none),
[2015-03-04,13:54:36.93]      bootserver=169.254.189.174, rootserver=169.254.189.174, rootpath=
[2015-03-04,13:54:45.69] VFS: Mounted root (jffs2 filesystem) on device 31:3.
[2015-03-04,13:54:45.70] Freeing init memory: 196K
[2015-03-04,13:54:47.25] eth0: no IPv6 routers present
[2015-03-04,13:55:00.44] SHARED_MEM: CPU1: INITIALIZATION STARTED BY CPU0. WAITING ON INIT DONE
[2015-03-04,13:55:00.66] SHARED_MEM: CPU1: INITIALIZATION OF SHARED MEMORY BY CPU0 DONE DETECTED
[2015-03-04,13:55:00.67] wait_for_init_shared_mem: Exiting Mutex for shared memory init
[2015-03-04,13:55:00.68] SHARED_MEM: shared_mem_open: About to take semaphore.
[2015-03-04,13:55:00.68] SHARED_MEM: shared_mem_open: Got Semaphore.
[2015-03-04,13:55:00.69] wait_for_init_shared_mem: Entering Mutex for shared memory init
[2015-03-04,13:55:00.70] wait_for_init_shared_mem: Exiting Mutex for shared memory init
[2015-03-04,13:55:00.71] SHARED_MEM: region name SHJ1  phy_addr 6e7fd000 virt_addr de000000 size ce00000
[2015-03-04,13:55:00.72] SHARED_MEM: region name read  phy_addr 637de000 virt_addr eb000000 size a100000
[2015-03-04,13:55:00.72] SHARED_MEM: region name SGLD  phy_addr 544d5000 virt_addr d8a00000 size 100000
[2015-03-04,13:55:00.73] SHARED_MEM: region name SHMQ  phy_addr 53ad5000 virt_addr d8c00000 size 200000
[2015-03-04,13:55:00.74] SHARED_MEM: region name SHLX  phy_addr 538d5000 virt_addr dd400000 size 200000
[2015-03-04,13:55:00.75] SHARED_MEM: region name SHVX  phy_addr 536d5000 virt_addr dd800000 size 200000
[2015-03-04,13:55:00.76] SHARED_MEM: shared_mem_open: Incrememnted open count: 0x1
[2015-03-04,13:55:00.76] Shared Memory Partition Map Addresses:
[2015-03-04,13:55:00.77] -----------------------------------------------------------------------------
[2015-03-04,13:55:00.78] SHJ1: Start:    0xde000000, End:        0xeae00000, Phys:       0x6e7fd000
[2015-03-04,13:55:00.78] HLRC: Start:    0xeb000000, End:        0xf5100000, Phys:       0x637de000
[2015-03-04,13:55:00.79] SGLD: Start:    0xd8a00000, End:        0xd8b00000, Phys:       0x544d5000
[2015-03-04,13:55:00.80] SHLX: Start:    0xdd400000, End:        0xdd600000, Phys:       0x538d5000
[2015-03-04,13:55:00.80] SHVX: Start:    0xdd800000, End:        0xdda00000, Phys:       0x536d5000
[2015-03-04,13:55:00.81] -----------------------------------------------------------------------------
[2015-03-04,13:55:00.82] SHARED_MEM: shm_queue_sync_cores: Mapped SHMQ partition at phys addr: 0x7bffd000 in kernel mem for size 0x200000
[2015-03-04,13:55:00.83] SHARED_MEM: CPU1: shm_queue_sync_cores: Ready to wait for init of SHMQ
[2015-03-04,13:55:01.16] SHARED_MEM: CPU1: shm_queue_sync_cores: Queue Init Started CPU0. Waiting on Init Done
[2015-03-04,13:55:01.17] SHARED_MEM: CPU1: shm_queue_sync_cores: Init of SHMQ done
[2015-03-04,13:55:01.18] handle_intercore_msg: Got Time Valid message
[2015-03-04,13:55:01.18] handle_intercore_msg: Got a NET_INFO request
[2015-03-04,13:55:23.78] handle_intercore_msg: Got ISCSI_ENABLE message
[2015-03-04,13:55:23.79] handle_intercore_msg: QUEUING ISCSI_ENABLE to sysfs
[2015-03-04,13:55:26.80] scsi0 : dri_dnas, version 2.0.0 [20120918], Commands: 0, Aborts: 0, Device Resets: 0
[2015-03-04,13:55:26.81] scsi 0:0:0:0: Direct-Access     Drobo    5N             1 .00  PQ: 0 ANSI: 5
[2015-03-04,13:55:26.82] sd 0:0:0:0: Attached scsi generic sg0 type 0
[2015-03-04,13:55:26.82] sd 0:0:0:0: [sda] physical block alignment offset: 3584
[2015-03-04,13:55:26.82] sd 0:0:0:0: [sda] 34359738368 512-byte logical blocks: (17.5 TB/16.0 TiB)
[2015-03-04,13:55:26.82] sd 0:0:0:0: [sda] 4096-byte physical blocks
[2015-03-04,13:55:26.83] sd 0:0:0:0: [sda] Write Protect is off
[2015-03-04,13:55:26.83] sd 0:0:0:0: [sda] Mode Sense: 30 00 00 00
[2015-03-04,13:55:26.83] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[2015-03-04,13:55:26.87]  sda: sda1 sda2
[2015-03-04,13:55:26.88] sd 0:0:0:0: [sda] Attached SCSI disk
[2015-03-04,13:56:12.54] handle_intercore_msg: Got Time Valid message
[2015-03-04,13:56:29.75] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: discard,barrier=1
[2015-03-04,13:56:35.21] Adding 262140k swap on /mnt/DroboFS/swapfile.  Priority:-1 extents:2 across:270332k
[2015-03-04,13:56:40.76] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).

$ cat /proc/filesystems
nodev   sysfs
nodev   rootfs
nodev   bdev
nodev   proc
nodev   tmpfs
nodev   devtmpfs
nodev   debugfs
nodev   sockfs
nodev   usbfs
nodev   pipefs
nodev   anon_inodefs
nodev   rpc_pipefs
nodev   devpts
        ext3
        ext2
        ext4
nodev   ramfs
        vfat
        msdos
        iso9660
nodev   nfs
nodev   jffs2
        xfs
nodev   mqueue
nodev   mtd_inodefs
nodev   oprofilefs
nodev   ubifs
nodev   nfsd

$ cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / jffs2 rw,relatime 0 0
proc /proc proc rw,relatime 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620 0 0
tmpfs /tmp tmpfs rw,relatime 0 0
none /sys sysfs rw,relatime 0 0
/dev/shm /dev/shm tmpfs rw,relatime,size=51200k 0 0
/dev/mtdblock1 /var jffs2 rw,relatime 0 0
/dev/sda1 /mnt/DroboFS ext4 rw,noatime,nodiratime,barrier=1,data=ordered,discard 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
$ uname -a
Linux RatHole 3.2.27 #1 SMP Tue Dec 10 22:48:37 PST 2013 armv7l GNU/Linux

$ lsmod
nfsd 86655 3 - Live 0x7f000000

$ ps | grep rpc
  193 root         0 SW<  [rpciod]
  704 root      2728 S    /mnt/DroboFS/Shares/DroboApps/nfs/sbin/rpc.mountd -F
ricardopadilha commented 9 years ago

I think I found the problem, and it is worth a gigantic facepalm.

Once you logged in the Drobo over SSH, try this:

cd /mnt/DroboFS/Shares/DroboApps/nfs
./service.sh stop

Now make sure there are no NFS-related processes running, i.e.:

grep ^nfsd /proc/mounts
pgrep rpcbind
pgrep rpc.mountd
pgrep rpc.statd
pgrep nfsd

... all return nothing. Now start NFS:

./service.sh start

Run the same commands as above, and make sure that something is there. Then press CTRL+C right there in the SSH session once. Try to find the NFS processes and they should all be gone, with the exception of rpc.statd and nfsd.

TL;DR: I think I'm missing a few setsid in service.sh. I would be very grateful if you guys could confirm it.

flakrat commented 9 years ago

Here are the results

cd /mnt/DroboFS/Shares/DroboApps/nfs
./service.sh stop

grep ^nfsd /proc/mounts
pgrep rpcbind
pgrep rpc.mountd
pgrep rpc.statd
pgrep nfsd

pwd
/mnt/DroboFS/Shares/DroboApps/nfs
./service.sh start

# grep ^nfsd /proc/mounts
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
# pgrep rpcbind
# pgrep rpc.mountd
1414
# pgrep rpc.statd
# pgrep nfsd

CTRL+C pressed

# grep ^nfsd /proc/mounts
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
# pgrep rpcbind
# pgrep rpc.mountd
1414
# pgrep rpc.statd
# pgrep nfsd
#
# exit
flakrat commented 9 years ago

I'm currently running the 3.1.1 [8.35.45] firmware on the 5N, do you want me to upgrade to the 3.2.0 and run the same commands or stay at the older firmware?

ricardopadilha commented 9 years ago

The current advice is to run 1.2.8 on the old firmware (3.1.1), and 1.3.2 on the new firmware (3.2.0). Once 1.3.2 is working correctly on 3.2.0, we'll work on making it work on the old firmware.

ricardopadilha commented 9 years ago

Found the problem. Libtirpc was looking for netconfig under /etc, and on my 5N/FS I did have a copy of that file there. Once the file was removed I could reproduce the issue.

flakrat commented 9 years ago

Looks good. I upgraded again to 3.2.0, installed the new nfs.tgz and:

$ grep ^nfsd /proc/mounts
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
$ pgrep rpcbind
1325
$ pgrep rpc.mountd
1331
$ pgrep rpc.statd
1335
$ pgrep nfsd
1339
1340
1341

And nfs mount from a client

$ sudo mount -t nfs 10.0.1.7:/mnt/DroboFS/Shares/Public /data/iso

$ mount | grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
sunrpc on /proc/fs/nfsd type nfsd (rw,relatime)
10.0.1.7:/mnt/DroboFS/Shares/Public on /data/iso type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.1.7,mountvers=3,mountport=51612,mountproto=udp,local_lock=none,addr=10.0.1.7)

Thanks!!!

ricardopadilha commented 9 years ago

@coolacid : b4ec144 fixes the DroboFS version. Please re-download from the releases page.