Crunch fails to open files on NFS and SSHFS network file systems

[illwieckz]

Crunch fails to open files on NFS and SSHFS network file systems.

I verified it on both macOS and FreeBSD with NFS, and on Linux with SSHFS:

$ pwd
/home/illwieckz/Mapping/Unvanquished/UnvanquishedAssets/src/tex-pk01_src.dpkdir/textures/shared_pk01_src

$ file door01a_d.png
door01a_d.png: PNG image data, 256 x 512, 8-bit/color RGB, non-interlaced

$ crunch -file door01a_d.png -out door01a_d.crn
crunch: Advanced DXTn Texture Compressor (Unity format variant)

Brought to you by:
- 2014-2022 Daemon Developers and contributors
  https://github.com/DaemonEngine/crunch
- 2017-2018 Alexander Suvorov and Unity Software Inc.
  https://github.com/Unity-Technologies/crunch/tree/unity
- 2010-2017 Richard Geldreich, Jr. and Binomial LLC and contributors
  https://github.com/BinomialLLC/crunch

crnlib version v1.04U x64 Built from git-1479e08

Warning: No files found: /usr/home/illwieckz/Mapping/Unvanquished/UnvanquishedAssets/src/tex-pk01_src.dpkdir/textures/shared_pk01_src/door01a_d.png

While it works on local filesystem:

$ cp -av door01a_d.png /tmp
door01a_d.png -> /tmp/door01a_d.png

$ crunch -file /tmp/door01a_d.png -out /tmp/door01a_d.crn
crunch: Advanced DXTn Texture Compressor (Unity format variant)

Brought to you by:
- 2014-2022 Daemon Developers and contributors
  https://github.com/DaemonEngine/crunch
- 2017-2018 Alexander Suvorov and Unity Software Inc.
  https://github.com/Unity-Technologies/crunch/tree/unity
- 2010-2017 Richard Geldreich, Jr. and Binomial LLC and contributors
  https://github.com/BinomialLLC/crunch

crnlib version v1.04U x64 Built from git-1479e08

Reading source texture: "/tmp/door01a_d.png"
Texture successfully loaded in 0.006s
Source texture: 256x512, Levels: 1, Faces: 1, Format: R8G8B8
Apparent type: 2D map, Flags: R G B Non-Flipped 
Generating mipmaps using filter "kaiser"
Generated 9 mipmap levels in 0.030s
Writing DXT1 texture to file: "/tmp/door01a_d.crn"
Compressing using quality level 128
Processing: 100%              
Texture successfully written in 0.091s
Texture successfully processed in 0.122s
Input texture: 256x512, Levels: 1, Faces: 1, Format: R8G8B8
Input pixels: 131072, Input file size: 219476, Input bits/pixel: 13.396
Output texture: 256x512, Levels: 10, Faces: 1, Format: DXT1
Output pixels: 174763, Output file size: 32467, Output bits/pixel: 1.486

Total time: 0.128s
1 total file(s) successfully processed, 0 file(s) skipped, 0 file(s) failed.

Exit status: 0

We can notice the absolute file path is properly computed anyway.

On Windows the crunch command line tool properly open files stored on native CIFS network share, so that looks specific to Unix APIs and/or NFS storage.

[illwieckz]

The code does:

    struct dirent* ep = readdir(dp);
[…]
    const bool is_directory = (ep->d_type & DT_DIR) != 0;
    const bool is_file = (ep->d_type & DT_REG) != 0;

If after that I add this line:

    console::warning("is_directory: %d, is_file: %d, %s", is_directory, is_file, ep->d_name);

And I create a test folder and a test file:

touch test-file
mkdir test-directory

I get that on disk filesystem APFS:

Warning: is_directory: 0, is_file: 1, test-file
Warning: is_directory: 1, is_file: 0, test-directory

But I get that on network filesystem NFS:

Warning: is_directory: 0, is_file: 0, test-file
Warning: is_directory: 0, is_file: 0, test-directory

This test was done on macOS but it's probably the same problem I face on Linux with SSHFS.

On Linux the man of readdir says:

       In the glibc implementation, the dirent structure is defined as follows:

           struct dirent {
               ino_t          d_ino;       /* Inode number */
               off_t          d_off;       /* Not an offset; see below */
               unsigned short d_reclen;    /* Length of this record */
               unsigned char  d_type;      /* Type of file; not supported
                                              by all filesystem types */
               char           d_name[256]; /* Null-terminated filename */
           };

Especially the type of file is not supported by all filesystem types. It's probably the same one macOS…

[illwieckz]

I edited the logger this way:

    console::warning("d_type: %d, is_directory: %d, is_file: %d, %s", ep->d_type, is_directory, is_file, ep->d_name);

And I get on APFS:

Warning: d_type: 8, is_directory: 0, is_file: 1, test-file
Warning: d_type: 4, is_directory: 1, is_file: 0, test-directory

and on NFS:

Warning: d_type: 0, is_directory: 0, is_file: 0, test-file
Warning: d_type: 0, is_directory: 0, is_file: 0, test-directory

So basically the feature is not supported at all…

[illwieckz]

Assumed to have been fixed by #41 long time ago:

• https://github.com/DaemonEngine/crunch/pull/41