OpenBSD: Readdir oddity

[ncw]

I've been experimenting trying to make rclone mount work with OpenBSD

Plus or minus a few unsupported options it seems to be working :-)

I've come across an oddity with Readdir which I'm not sure of the cause - I think it may be an OpenBSD bug but it might be a cgofuse bug so I thought I'd ask you first.

I replicated this with the hellofs example.

This is the result of mounting and doing ls on the root directory (note that you can't mount FUSE fs as a user on OpenBSD).

openbsd$ sudo ./hellofs -d /mnt/tmp                   
Opcode: init    
Opcode: getattr Inode: 1    /
Opcode: getattr Inode: 1    /
Opcode: opendir Inode: 1    /
Opcode: getattr Inode: 1    /
Opcode: readdir Inode: 1    Offset: 0   Size: 4096  /
Opcode: readdir Inode: 1    Offset: 96  Size: 4000  /
Opcode: readdir Inode: 1    Offset: 96  Size: 4096  /   <----- unexpected
Opcode: releasedir  Inode: 1    /

What is odd here is the 3rd call to readdir.

I put a debug line into the Readdir function

diff --git a/examples/hellofs/hellofs.go b/examples/hellofs/hellofs.go
index d33566f..6d6eb39 100644
--- a/examples/hellofs/hellofs.go
+++ b/examples/hellofs/hellofs.go
@@ -14,6 +14,7 @@ package main

 import (
        "os"
+       "log"

        "github.com/billziss-gh/cgofuse/fuse"
 )
@@ -66,6 +67,7 @@ func (self *Hellofs) Readdir(path string,
        fill func(name string, stat *fuse.Stat_t, ofst int64) bool,
        ofst int64,
        fh uint64) (errc int) {
+       log.Printf("\n*** Readdir(path=%q, ofst=%d, fh=0x%X) ***\n", path, ofst, fh)
        fill(".", nil, 0)
        fill("..", nil, 0)
        fill(filename, nil, 0)

And it produces this on OpenBSD - note the Readdir function is called 3 times rather than just once.

Opcode: init    
Opcode: getattr Inode: 1    /
Opcode: getattr Inode: 1    /
Opcode: opendir Inode: 1    /
Opcode: getattr Inode: 1    /
Opcode: readdir Inode: 1    Offset: 0   Size: 4096  /2020/09/01 12:47:31 
*** Readdir(path="/", ofst=0, fh=0xFFFFFFFFFFFFFFFF) ***

Opcode: readdir Inode: 1    Offset: 96  Size: 4000  /2020/09/01 12:47:31 
*** Readdir(path="/", ofst=96, fh=0xFFFFFFFFFFFFFFFF) ***

Opcode: readdir Inode: 1    Offset: 96  Size: 4096  /2020/09/01 12:47:31 
*** Readdir(path="/", ofst=96, fh=0xFFFFFFFFFFFFFFFF) ***

Opcode: releasedir  Inode: 1    /

Whereas if I try the same code on Linux I get the Readdir function being called just once as expected with two READDIR fuse calls.

unique: 10, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115481
getattr /
   unique: 10, success, outsize: 120
unique: 12, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 115481
opendir flags: 0x18800 /
   opendir[-1] flags: 0x18800 /
   unique: 12, success, outsize: 32
unique: 14, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 115481
readdir[18446744073709551615] from 0
2020/09/01 11:50:41 
*** Readdir(path="/", ofst=0, fh=0xFFFFFFFFFFFFFFFF) ***
   unique: 14, success, outsize: 112
unique: 16, opcode: LOOKUP (1), nodeid: 1, insize: 46, pid: 115481
LOOKUP /hello
getattr /hello
   NODEID: 2
   unique: 16, success, outsize: 144
unique: 18, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 115481
   unique: 18, success, outsize: 16
unique: 20, opcode: RELEASEDIR (29), nodeid: 1, insize: 64, pid: 0
releasedir[18446744073709551615] flags: 0x0
   unique: 20, success, outsize: 16

So I think this might be a bug in the directory filling routines in OpenBSD, whether that is in cgofuse, or OpenBSD libfuse I don't know!

This causes a problem for rclone since it checks to see that Readdir is never called with a non-zero offset which it never should be as it always passes 0 as an offset to the fill function. This is easy enough to work-around but I thought I'd report it in case it is indicative of an underlying problem.

I tested this on a OpenBSD 6.7 VM I installed from scratch using go version go1.13.9 openbsd/amd64 (which is what you get with pkg_add. I can send you this VM if you would like.

[billziss-gh]

I've come across an oddity with Readdir which I'm not sure of the cause - I think it may be an OpenBSD bug but it might be a cgofuse bug so I thought I'd ask you first.

Yes, this looks odd.

This causes a problem for rclone since it checks to see that Readdir is never called with a non-zero offset which it never should be as it always passes 0 as an offset to the fill function. This is easy enough to work-around but I thought I'd report it in case it is indicative of an underlying problem.

I agree that you should not see a non-0 offset in Readdir, if you never pass a non-0 offset to the fill function.

In the Linux/FreeBSD/macOS case it is actually libfuse that makes this guarantee: the kernel protocol always includes the offset in the FUSE_READDIR request message, but libfuse abstracts this detail away by buffering readdir data, when it sees a 0 offset and satisfying the FUSE_READDIR request from the buffered data when it sees a non-0 offset. (Libfuse only does this buffering when you specify 0 offsets in the fill function.) See the libfuse fuse_readdir_common.

The cgofuse Readdir implementation is minimal and the same across all cgo platforms. So I suspect (but may be wrong) that in this case the problem lies with OpenBSD's implementation. I note that OpenBSD does not use libfuse.

[billziss-gh]

Here is the readdir implementation of OpenBSD's libfuse, which appears to pass the kernel's view of the offset:

ifuse_ops_readdir

It appears to me that this is a problem with OpenBSD.

[ncw]

Thank you for your analysis Bill.

I finally found the docs for the fill behaviour in the libfuse docs

So what it looks like is that OpenBSD libfuse equivalent doesn't implement this behavior

1) The readdir implementation ignores the offset parameter, and passes zero to the filler function's offset. The filler function will not return '1' (unless an error happens), so the whole directory is read in a single readdir operation.

@SylvestreG - I think you wrote this code originally. Does that sound right to you?

@lkostal - as the most recent OpenBSD user to be interested in this, do you want to report this as a bug?

[djdv]

Just wanted to corroborate that I'm experiencing this as well. I took some notes during testing an implementation. Noticed an oddity when using NetBSD and OpenBSD. https://gist.github.com/djdv/8341762e571ef799cf7613f2f3c1f506#netbsd This lists some of the various versions I tried and has a small trace.