Maximum buffer size for contents of readdir?

[sarnobat]

I'm finding that only the first 3200 file names (approximately) added to my directory filler in readdir() get displayed in my file system, with names like (soccer\ scores\ Liverpool.txt). When I run a test filller with 9000 numerically named files (00001.txt) it's fine.

I don't know what's going on but it looks like it's buffer size related. Are there any limitations on the buffer size of the directory filler?

[EtiennePerot]

This isn't documented much in fuse-jna or fuse itself either. I haven't confirmed this but I would guess that it is because your code isn't checking the return value of DirectoryFiller.add. It should return true on success, false on failure (because the buffer is full). Your code should return -ErrorCodes.ENOMEM() (out of memory) in this case, such that the caller knows it needs to ask again with a bigger buffer.

Let me know if my guess is correct. If it is I'll update the documentation and examples, otherwise I'll investigate further (but for that, I need a minimal example to reproduce it).

[sarnobat]

Thanks for the quick reply. I'll post here after trying it, which may take some time.

[sarnobat]

I just tried it with 30,000 files named with the following name length, and it was fine.

11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-helloworld.txt

So at this point it seems like there's something wrong in my specific code.

It's not fair for me to keep this bug open. But I may need to reopen it to get more help. Thanks for the info.

[EtiennePerot]

Did you add code to return ENOMEM on failure to add directory entries?

If so, leaving this open as reminder to improve documentation around this.

[sarnobat]

Checking the return status of DirectoryFiller.add and returning ENOMEM if non-zero didn't have any effect.

I should be able to gather a test case. I'll provide it soon. Thanks for maintaining interesting in resolving this.

[sarnobat]

Reproducible Test case

Instructions

1. Download this file, and remove the ".txt" extension FuseTruncationProblem.java.txt
2. Add the fuse-jna library to your classpath
3. Run the main method

Once the file system is mounted, then execute the following from the shell:

find -L /tmp/fusejnaproblem

You should see that only the first 3500 files get printed, even though there should be 6000 files (populated via a List using the getList() method).

Findings, Notes etc

• My latest suspicion is that something in my specific file name data is breaking the fuse functionality (with a loop that creates numerical file names, everything works

• It's nothing to do with buffer size afterall. A single file with this name causes issues:

  files.add("3503::Mainstream American pop culture seem completely worthless! Why aren't people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing");

Conclusion

File names can be 255 characters maximum. This is not a limitation of FUSE. NTFS and EXT4 have this maximum.

Maybe consider adding a safety check somewhere in your library so that rookies like me don't get perplexed with an otherwise fantastic library. Thanks very much for creating this.

[EtiennePerot]

Sorry, I didn't notice this reply until just now.

Thanks a lot for writing this testcase. However, I wasn't able to reproduce the problem with it. I ran it and the directory listing contains all ~6000 files. Maybe this is platform-dependent? Are you on OS X? I checked this on Linux 4.6. AFAIK it has a max path limit of 4096 characters, but the filename length limit should be filesystem-dependent.

If this is an OS-X-specific problem, I could add a check on filename length that only applies on OS X.

[sarnobat]

You can close it. It's nothing to do with buffer size. Here are the relevant remarks:

It's nothing to do with buffer size afterall. A single file with this name causes issues: files.add("3503::Mainstream American pop culture seem completely worthless! Why aren't people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing");

Conclusion

File names can be 255 characters maximum. This is not a limitation of FUSE. NTFS and EXT4 have this maximum.

Maybe consider adding a safety check somewhere in your library so that rookies like me don't get perplexed with an otherwise fantastic library. Thanks very much for creating this.

[EtiennePerot]

I understood what you said, but it works for me.

With the following code, which adds only a single file under 1dir:

package net.fusejna.examples;

import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.LinkedList;
import java.util.List;

import net.fusejna.DirectoryFiller;
import net.fusejna.ErrorCodes;
import net.fusejna.FuseException;
import net.fusejna.StructFuseFileInfo.FileInfoWrapper;
import net.fusejna.StructStat.StatWrapper;
import net.fusejna.types.TypeMode.NodeType;
import net.fusejna.util.FuseFilesystemAdapterFull;

public class FuseTruncationProblem extends FuseFilesystemAdapterFull
{
    public static void main(final String... args) throws FuseException, IOException
    {
        Files.createDirectories(Paths.get("/tmp/fusejnaproblem"));
        new FuseTruncationProblem().log(false).mount("/tmp/fusejnaproblem");
    }

    final String filename = "/hello1.txt";
    final String contents = "Hello World\n";

    @Override
    public int getattr(final String path, final StatWrapper stat)
    {
        if (path.equals(File.separator)) { // Root directory
            stat.setMode(NodeType.DIRECTORY);
            return 0;
        }
        if (path.equals("/hello1.txt")) { // hello.txt
            stat.setMode(NodeType.FILE).size(contents.length());
            return 0;
        }
        if (path.endsWith("dir")) {
            stat.setMode(NodeType.DIRECTORY);
            return 0;
        }
        stat.setMode(NodeType.FILE).size(contents.length());
        return 0;
    }

    private List<String> getList()
    {
        final List<String> files = new LinkedList<String>();
        files.add(
                "3503::Mainstream American pop culture seem completely worthless! Why aren't people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing, Congress, President - Page 5 - City-Data Forum");
        return files;
    }

    @Override
    public int read(final String path, final ByteBuffer buffer, final long size, final long offset, final FileInfoWrapper info)
    {
        // Compute substring that we are being asked to read
        final String s = contents.substring((int) offset,
                (int) Math.max(offset, Math.min(contents.length() - offset, offset + size)));
        buffer.put(s.getBytes());
        return s.getBytes().length;
    }

    @Override
    public int readdir(final String path, final DirectoryFiller filler)
    {
        filler.add(filename);
        if (path.length() == 1) {
            filler.add("1dir");
        }
        if (path.endsWith("1dir")) {
            for (final String file : getList()) {
                final boolean added = filler.add(file);
                if (!added) {
                    System.out.println("HelloWorldFuse.readdir() Failed to add");
                    return ErrorCodes.ENOMEM();
                }
            }
        }
        return 0;
    }
}

I get:

$ ls /tmp/fusejnaproblem/1dir    
'3503::Mainstream American pop culture seem completely worthless! Why aren'\''t people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing, Congress, President - Page 5 - City-Data Forum'
hello1.txt

Hence me asking if the behaviour is platform-specific.

[sarnobat]

Oh I see, my mistake. I was running this on Mac OS 10.7 I believe.