p and pp could show the number of files

[muellerto]

Is your feature request related to a problem? Please describe. The p and pp commands could show the number of contained files when the operand is or points to a directory.

Describe the solution you'd like The p command should count the direct children while pp should be full recursive with the same algorithm as used for determining file sizes. There's then just an additional row needed in the output:

files: 3736

or clearer:

subdirs/files: 385/27362

OK, it could still be more detailed when symbolic links are also respected:

subdirs/files: 385/27362, symlinked subdirs/files: 12/5

A question is about following the symlinks. But the same question you already had when determining the over all size of the contained files. I wouldn't implement something new here, so you can give the same answer here just for counting.

Describe alternatives you've considered I saw the stats command showing a number of files but this seems to be always the current directory and not recursive.

One of the easiest alternatives is to open fzf with no parameters in such a directory, this shows the number of files very fast. You can also call find | bat to get a number (bat without parameters shows a line number).

Additional context It's not that I count my files all the day. But sometimes you need to know if there has come an additional file or not, so you need exact numbers.

[leo-arch]

I did now the following

Clever. Unifying the file size calculation with the files counter is the logical step to go.

Does dir_info() dereference symbolic links when the link points onto a directory?

No. We're using lstat(2) (instead of stat(2)), so that symlinks are not dereferenced. However, as I've said before, I think following symlinks in this case doesn't make much sense: 1. If the symlink points to somewhere outside our directory, it shouldn't be counted; 2. If it points rather to somewhere under our directory, it will be counted anyway. So that, in a way, symlinks are followed.

I saw then that du returns exactly my result when it is called with the -L parameter (dereference links).

I'm not exactly sure what du(1) is doing internally. Some testing is required. Nonetheless, the important thing is to get consistent results, independently of what du says.

One point to consider is this: when calculating full directory sizes in the long view, we still use du, which means that the directory size might not agree with that informed by pp. These results must be kept in sync. The logical solution would be to use our own file size algorithm for the long view as well, but we need to make sure it won't significantly degrade performance.

Also, the file size algorithm should take into account whether we are calculating apparent sizes (the amount of bytes actually consumed by the file - this is what you're currently doing) or real sizes (the amount of disk space taken up to store the file). In the same vein, directories are ignored when calculating apparent sizes, but they do take some space (4k, for this is the size of a disk block) when calculating real sizes. Finally, hardlinks should also be taken into account: if two or more files are hardlinked to the same file under our parent directory, they should be counted only once. This means that you must keep track of all hardlinks found when traversing the directory.

Removing a dependency, in this case du, is always a good thing, provided it doesn't carry any major drawback.

[leo-arch]

Ok. The new dir size/counter algorithm is ready. Everything seems to be working exactly as before, except that dir size and files count are made in a single algorithm/scan.

For the time being, this new algorithm is used only for pp: full dir size calculation in long view still relies on du.

We still keep the old code, just in case.

EDIT: Clifm is now 100% du-free. You can unblock the old du code compiling with -DUSE_DU1.

EDIT 2: The dir_info() code has been moved to aux.c (though I'm planning to move it to a separate file, say xdu.c).

[muellerto]

EDIT: Clifm is now 100% du-free.

I think this is a milestone :)

Ok. The new dir size/counter algorithm is ready. Everything seems to be working exactly as before, except that dir size and files count are made in a single algorithm/scan.

Here some results from my Windows machine.

directory	du	du -L	Explorer	clifm
no symlinks at all	47892105372	47892105372	47’892’105’372	47892105372
some inner symlinks, no special files	98255311860	98255311040	98’255’311’040	98255311860
/c/Program Files (special files)	(stack dump)	(stack dump)	54’933’899’005	18435519277
/c/Windows (special files)	(stack dump)	(stack dump)	34’614’516’443	25915670978

The last two lines have of course ! marks, so we see that something is not good.

The du calls were du -s --apparent-size --block-size=1 DIR and du -s --apparent-size --block-size=1 -L DIR. I can't explain the difference of 820 bytes in the second line in two points:

1. the difference must be much bigger (some hundred kB)
2. dereferencing symlinks leads to a smaller result???

Probably we must say handling symlinks is a general problem and every algorithm has it's own problems with that.

[leo-arch]

1. dir_info() was moved to a separate file: xdu.c
2. I have no clue at how Explorer is calculating file sizes (and maybe no one outside MS does), and thereby, why it's getting bigger sizes than Clifm. But the fact that we get the exact same results as du is awesome. It's also great that Clifm doesn't crash when du does.

dereferencing symlinks leads to a smaller result???

I have no idea why du is doing that. But it's not our business anymore: it is now a du specific issue.

The last two lines have of course ! marks, so we see that something is not good

This just means that some subdirectory (most likely due to permissions) couldn't be read, and that the resulting size might therefore not be accurate (this is totally expected).

EDIT: What do you mean exactly by special files?

EDIT 2: Looking at the table, we can see that the Explorer is dereferencing symlinks, which might explain why it is getting much bigger sizes. Maybe one or more symlinks are pointing to some directory outside of DIR. If this is the case, the Explorer is getting wrong results, because we want the size of DIR and whatever is under it. Or, the symlink might be pointing to a subdirectory of DIR, in which case the same subdirectory is counted twice.

[muellerto]

One problem with the Windows directories seems to be related to hard links.

I copied your dir_info() function into a simple test file with a main function. The only difference I made was: I had no check_xdu_hardlinks() and add_xdu_hardlink() functions, so I removed the entire

if (a.st_nlink > 1) {
...
}

in the assumption I have no hard links at all. With that simple change I get the following results:

directory	Explorer	with if	without if
/c/Windows	34’614’516’443	25915670978	34466934113
/c/Program Files	54’933’899’005	18435519277	18568841801

Waahhh! Indeed I have 93000 (!) entries in my /c/Windows where the condition (a.st_nlink > 1) is true. This does really surprize because the Microsoft documentation says: st_nlink is always 1 on NTFS. The MinGW runtime has probably another opinion. But if we really get into the body of that if probably your check_xdu_hardlink() leads then in some cases to the continue which excludes of a lot of content from being counted.

Update: I have now the hardlink-functions. Indeed I saw that my xdu_hardlinks list gets a length of more than 36000 entries. I added a usecount member into hlink_t, this shows that in my /c/Windows hundreds of files are hard linked 2, 3, 4 and even 5 times. - The longer I think about this it gets clearer that the Explorer counts all these hard links as normal files while you just try to avoid this. That's the difference.

But the second line in the table shows: that's still not all. /c/Program Files is different, the usecount of hardlinks is always 1. The reason for the big difference must have another reason.

[muellerto]

EDIT: What do you mean exactly by special files?

Files or directories which causes errors because of missing permissions, policies or exclusive being opened. This is something special in these Windows directories. Most of a disk doesn't have such files or directories.

[muellerto]

I found out now that under Windows the S_ISXXX macros are indeed not good, nevertheless MinGW defines them for compatibility:

#define _IFMT       0170000 /* type of file */

#define S_ISBLK(m)  (((m)&_IFMT) == _IFBLK)
#define S_ISCHR(m)  (((m)&_IFMT) == _IFCHR)
#define S_ISDIR(m)  (((m)&_IFMT) == _IFDIR)
#define S_ISFIFO(m) (((m)&_IFMT) == _IFIFO)
#define S_ISREG(m)  (((m)&_IFMT) == _IFREG)
#define S_ISLNK(m)  (((m)&_IFMT) == _IFLNK)
#define S_ISSOCK(m) (((m)&_IFMT) == _IFSOCK)

I added code directly after lstat() has been called which shows the bits in a.st_mode. Under Windows it seems to be a typical case the S_IFREG and S_IFLNK are both 1, probably it can also happen that S_IFDIR and S_IFLNK are both 1. If such an a.st_mode is now checked with the S_ISREG macro it gets wrong because the result of a.st_mode & _IFMT is indeed S_ISREG | S_ISLNK. So the bits in a.st_mode must really be bit-tested which has also consequences for the logic.

[leo-arch]

The longer I think about this it gets clearer that the Explorer counts all these hard links as normal files while you just try to avoid this. That's the difference.

The thing with hardlinks is this: they should be counted as different files, but they shouldn't add to the total size, because all of them are indeed just one file (i.e. same device and inode number). This is what both du and Clifm do. The Explorer, by contrast, is adding the size of the same file multiple times, and this is just wrong and deceiving, because you're not really using that space.

Under Windows it seems to be a typical case the S_IFREG and S_IFLNK are both 1

You're right, but only partially: this is the case for MinGW, but not for Cygwin. On MinGW symlinks are not recognized at all (this includes not only Clifm, but also stat and ls -l, and it most likely is the case for most Unix tools). So, if we want to tackle this issue we need to be able to tell whether we're running on MinGW or not. Problem is:

1. Some compilers define __MINGW32__, but not gcc. We can define it manually, but it's far from ideal.
2. Once we know we're on MinGW, we need a way to tell if a file is a symlink.

EDIT: I've found this, where we read: and actually I think msys doesn't symlink but copies. I've made a few tests and this seems to be right:

1. When you try yo create a symbolic link under MinGW (using either ln -s or Clifm's l command - which internally uses symlink(2), just as ln), it doesn't actually create any link, but a copy of the original file. Indeed, if you modify the content of the "symlink", the original file is not affected at all, because they're just two different regular files.
2. st_mode has exactly the same value for both symlinks and regular files (maybe just because symlinks are indeed regular files on MinGW), so that it's not possible to make any difference using this field.
3. If you create a symlink under Cygwin and then inspect this file (with pp) on MinGW, it is actually shown as a symlink. This leads me to think that the actual problem is the way in which MinGW creates symlinks (or, more precisely, the way in which is does not create symlinks, but copies). If this is the case, then the problem is not related to our implementation and there's not much we can do: pp is telling the truth, because the symlink (created under MinGW) is not actually a symlink, but a regular file.

[muellerto]

On MinGW symlinks are not recognized at all (this includes not only Clifm, but also stat and ls -l, and it most likely is the case for most Unix tools).

But (using your algorithm in a MinGW compiled program) lstat() shows entries which carry just a single S_IFLNK bit. And these entries (pointing to either a file or a directory) are exactly the symlinks I know from my daily work. Also ls -l or eza -l do show symlinks correctly:

> ls -l
insgesamt 632
lrwxrwxrwx 1 tm Domain Users     65 12. Mai 2023  favicon.ico -> /e/work/Dev/18_X/Base/Modules/IIPWebGuiModule/WebRoot/favicon.ico
lrwxrwxrwx 1 tm Domain Users     62 12. Mai 2023  standard -> /e/work/Dev/18_X/Base/Modules/IIPWebGuiModule/WebRoot/standard/

> eza -l
la---    0 12 May  2023 favicon.ico -> E:\work\Dev\18_X\Base\Modules\IIPWebGuiModule\WebRoot\favicon.ico
l----    0 12 May  2023 standard -> E:\work\Dev\18_X\Base\Modules\IIPWebGuiModule\WebRoot\standard

I would say this works very well.

When you try yo create a symbolic link under MinGW (using either ln -s

I almost never made a symlink using MinGW's ln because this causes always questions on another disk. I use other software [1,2,3] for that, and that's legal. So we can't assume that a symlink has been made by a concrete application, we must just take the heterogeneous things just being there in the file system.

[1] Microsoft's own mklink.exe (not very popular but belongs to Windows) [2] Link Shell Extension 3.9.3.5 by Hermann Schinagl [3] Multi Commander 13.5.0.2983 by Matthias Svensson

I got my results using a while loop in dir_info() as in the attached file while.c.gz (great, I can't attach a .c file here ...) Note: this has been tested only on MinGW (64bit). I never checked this on Cygwin (because I don't have any) or Linux. If there would be a different behavior it would be interesting.

Yes, detecting MinGW seems to be not so easy. I have neither __MINGW32__ nor __MINGW64__, too. If this really gets important the Makefile could run gcc -v or cpp -v, this gives an output containing a readable string like "x86_64-pc-msys" which can be examined by grep. But that's not very safe because it names a software where MinGW is included, there are also other MinGW distributions. OR (probably better): you provide a second, different Makefile especially for MinGW containing a concrete define for the compiler.

[leo-arch]

Wait, I'm a bit lost now. What are we trying to fix exactly? Is it a wrong dir size calculation, the files counter, or what? Please provide a concrete example.

As to your code, I've never seen file type checks written like that (I mean, the st_mode field AND'ed directly with an S_IF file type macro), besides the fact that it does not consider character/block devices, sockets, and pipes. See for example the Glibc manual. Is this code solving your issue?

EDIT: Tested your code. I'm getting 1 extra directory, less regular files, less files in the total count, and a smaller size. Maybe because of non-tested file types. Not sure.

[leo-arch]

while ((ent = readdir(p)) != NULL) {
    if (SELFORPARENT(ent->d_name))
        continue;

    char buf[PATH_MAX + 1];
    snprintf(buf, sizeof(buf), "%s/%s", dir, ent->d_name);

    if (lstat(buf, &a) == -1) {
        info->status = errno;
        info->files++;
        continue;
    }

#ifdef ALT_LINK_TEST
    if ((a.st_mode & (S_IFLNK | S_IFDIR | S_IFREG)) == S_IFLNK) { /* a single S_IFLNK bit */
#else
    if (S_ISLNK(a.st_mode)) {
#endif /* ALT_LINK_TEST */
        info->links++;
    } else if (S_ISREG(a.st_mode)) {
        info->files++;
    } else if (S_ISDIR(a.st_mode)) {
        /* Even if a subdirectory is unreadable or we can't chdir into
         * it, do let its size contribute to the total (provided we're
         * not computing apparent sizes). */
        if (conf.apparent_size != 1)
            info->size += (a.st_blocks * S_BLKSIZE);

        info->dirs++;
        dir_info(buf, 0, info);

        continue;
    } else {
        info->files++;
    }

    if (usable_st_size(&a) == 0)
        continue;

    if (a.st_nlink > 1) {
        if (check_xdu_hardlinks(a.st_dev, a.st_ino) == 1)
            continue;
        else
            add_xdu_hardlink(a.st_dev, a.st_ino);
    }

    info->size += conf.apparent_size == 1 ? a.st_size
        : (a.st_blocks * S_BLKSIZE);
}

This is basically the same old code (though rearranged). Compile with -DALT_LINK_TEST to use your alternative symlinks check (since, as far as I understand, the current issue is related to symlinks only).

I'm getting the same results as with the original code (with and without ALT_LINK_TEST). Please let me know if compiling with ALT_LINK_TEST makes any difference for you.

[muellerto]

Wait, I'm a bit lost now. What are we trying to fix exactly? Is it a wrong dir size calculation, the files counter, or what? Please provide a concrete example.

I'm trying to find out how your algorithm works, to learn and to help. This all to make it reliable and explainable. And you will have to explain the enormous differences on some directories under Windows because people will just know the Explorer and miss some ten GB. A good thing is that normal user directories are simple and that's why mostly right and the use of symlinks is something by far not everyone does.

My results are:

• The st_mode variable in the stat struct is a bit mask. At least on Windows it's not seldom that more than one bit is set.

• A true symlink can safely be detected by a single S_IFLNK bit. When st_mode has a S_IFLNK and an additional S_IFREG or S_IFDIR it's not a symlink but a regular file or a directory. The reason for the S_IFLNK is still unclear in these cases.

• Because of that the S_ISxxx() macros which do an integer comparison are not safe, at least not on every OS and/or file system. Some compilers, especially the Microsoft compiler, do not even know these macros (good so). Bit testing with the & operator is indeed the better solution. Even when S_ISLNK() is used at first to filter out true symlinks it's then not safe to use S_ISREG() or S_ISDIR() later because we have lots of files and directories with two (!) bits. Also your latest while loop above doesn't respect this. I think bit testing is also portable even when the numeric values for the bits differ among the OSes.

• Not every Windows is equal and there's indeed a variety including older versions, editions and not to forget the server OSes. Windows 11 alone has 7 (!) different editions, each in versions. And the installed software and system updates bring even more variety. So it's possible that some machines do not have such directory entries, others have.

• On my machine (Windows 11 Pro, MS Office, MS Visual Studio, MS SQL-Server, 5 different browsers and tons of other stuff) there are lots of symbolic links and an enormous amount of hard links in the Windows system directories. That's the main reason for the big differences between clifm and the Explorer.

As to your code, I've never seen file type checks written like that (I mean, the st_mode field AND'ed directly with an S_IF file type macro), besides the fact that it does not consider character/block devices, sockets, and pipes. See for example the Glibc manual. Is this code solving your issue?

Yes, I know. That's also what I have in the MinGW headers (which come originally from Cygwin). Probably this works on Unix-like OSes. But it doesn't work this way on Windows. It will not make the whole algorithm fail but it leads indeed to wrong results. I'm not sure but may be my MinGW du struggles because of that.

[leo-arch]

According to my tests (and on Linux at least), the directory check (a.st_mode & S_IFDIR) takes either block device, character device, or pipe file (not sure which) as a directory, which is clearly not right.

On the other side, the symlinks check ((a.st_mode & (S_IFLNK | S_IFDIR | S_IFREG)) == S_IFLNK) and the regular file check (a.st_mode & S_IFREG) seem to work fine.

#ifdef ALT_FTYPE_CHECKS
    if ((a.st_mode & (S_IFLNK | S_IFDIR | S_IFREG)) == S_IFLNK) { /* Works fine */
        info->links++;
    } else if (a.st_mode & S_IFREG) { /* Works fine */
        info->files++;
    /* The following check takes one of block, character or pipe file as a directory.
     * S_ISDIR(a.st_mode), on the other side, works fine. */
    } else if (a.st_mode & S_IFDIR) {
#else
    if (S_ISLNK(a.st_mode)) {
        info->links++;
    } else if (S_ISREG(a.st_mode)) {
        info->files++;
    } else if (S_ISDIR(a.st_mode)) {
#endif /* ALT_FTYPE_CHECKS */

[leo-arch]

This seems to be working:

#ifdef ALT_FTYPE_CHECKS
    if ((a.st_mode & (S_IFLNK | S_IFDIR | S_IFREG)) == S_IFLNK) {
        info->links++;
    } else if ((a.st_mode & S_IFREG)
    || (a.st_mode & (S_IFBLK | S_IFSOCK | S_IFIFO)) != S_IFDIR) {
        info->files++;
    } else if (a.st_mode & S_IFDIR) {
#else
    if (S_ISLNK(a.st_mode)) {
        info->links++;
    } else if (S_ISDIR(a.st_mode)) {
#endif /* ALT_FTYPE_CHECKS */
        ...
    } else {
        info->files++;
    }

[muellerto]

Ahm ... (a.st_mode & (S_IFBLK | S_IFSOCK | S_IFIFO)) != S_IFDIR is always true.

[leo-arch]

Ahm ... (a.st_mode & (S_IFBLK | S_IFSOCK | S_IFIFO)) != S_IFDIR is always true.

Not on Linux at least. Otherwise I would be getting zero directories counted, but I get 28784 subdirs in my home dir.

EDIT: According to my tests, this clause is always false for directories, and true even for regular files.

EDIT 2: a.st_mode & (S_IFBLK | S_IFSOCK | S_IFIFO) is the same as a.st_mode & S_IFMT, so that (a.st_mode & (S_IFBLK | S_IFSOCK | S_IFIFO)) != S_IFDIR is exactly the same as !S_ISDIR(a.st_mode), since the S_ISDIR macro is internally expanded to (a.st_mode & S_IFMT) == S_IFDIR.

[leo-arch]

Disclaimer: all these tests were made on Linux. However, the following values seem to be the same on most platforms (confirmed for Linux, OpenBSD, NetBSD, FreeBSD, Haiku, Solaris, MacOS, and Cygwin).

FTYPE	MASK (octal)	MASK (binary)
S_IFDIR	0040000	0100000000000000
S_IFCHR	0020000	0010000000000000
S_IFBLK	0060000	0110000000000000
S_IFREG	0100000	1000000000000000
S_IFIFO	0010000	0001000000000000
S_IFLNK	0120000	1010000000000000
S_IFSOCK	0140000	1100000000000000

According to the above table, the S_IFDIR mask sets to zero everything but the second bit (left to right), so that a.st_mode & S_IFDIR matches not only directories, but also block devices and sockets (because these also have the second bit set).

Because of this, it is not enough to check the directory bit only, since this bit is also set for block devices and sockets. Thereby, we need to check block devices and sockets before directories.

In the same way, the S_IFLNK mask sets to zero everything but the first and third bit, so that a.st_mode & S_IFLNK will match any file type whose first and/or third bit is set (i.e. everything not being a directory, whose only bit set is the second one, or a FIFO/pipe, whose only bit set is the fourth). To make it match symbolic links only, we want (a.st_mode & S_IFLNK) == S_IFLNK.

So, taking the above into account, and if we follow your file type check style (i.e., avoiding the S_IS* macros and the S_IFMT mask), we get this code:

if ((a.st_mode & S_IFLNK) == S_IFLNK) {
    info->links++;
} else if ((a.st_mode & S_IFBLK) == S_IFBLK || (a.st_mode & S_IFSOCK) == S_IFSOCK) {
    info->files++;
} else if (a.st_mode & S_IFDIR) {
    info->dirs++;
    ...
} else {
    info->files++;
}

Nonetheless, it is important to note that this code isn't portable, for it depends on how the bits for each file type are set. Change these bits, and the code will stop working as expected (this is what the S_IS* macros are supposed to deal with).

[muellerto]

Yeah, when the S_IFxxx defines themselves do contain multiple bits it gets really complicated and some AND operations could lead to unexpected results. This is something I could not imagine. And then some of my statements in my last post(s) are not right.

The following defines come from the Microsoft compiler's <sys/stat.h>:

#define _S_IFMT   0xF000 // File type mask
#define _S_IFREG  0x8000 // Regular
#define _S_IFDIR  0x4000 // Directory
#define _S_IFCHR  0x2000 // Character special
#define _S_IFIFO  0x1000 // Pipe

These four defines are the same as in your defines above, all are single bits, but the three others are missing. That's why they don't define these S_ISxxx macros, simple bit testing is safe and clear there.

BTW: they don't have a _S_IFLNK, so recognizing a symlink cannot be done using stat() alone. Probably the MinGW and Cygwin implementations do call API functions internally to get the S_IFLNK bit.

Probably your S_ISxxx macro based code is not sooo bad. That is what these macros can do. Perhaps it's not right in every special case but in the most user cases it will do the job, especially on the Unix like OSes. But it's good to have this research result.

[leo-arch]

The following defines come from the Microsoft compiler's <sys/stat.h>

Thanks.

Probably the MinGW and Cygwin implementations do call API functions internally to get the S_IFLNK bit

If they can, we can (the only unsolvable problem is the one we decided not to solve).

On my MSYS2 install, the S_IF* macros (all of them) are defined exactly as in the table above. Could you provide the exact st_mode value of those files whose type is not being properly determined?

EDIT:

Yes, detecting MinGW seems to be not so easy

On my box I've found that __MSYS__ is defined, so that we should be able to isolate the new code under #if defined(__CYGWIN__) && defined(__MSYS__).

[muellerto]

Could you provide the exact st_mode value of those files whose type is not being properly determined?

The results my test program (attached below) shows on Windows:

1. S_IFCHR is never set alone.
2. When S_IFCHR is 1 then S_IFREG is also 1 and S_IFDIR is never 1. So the S_ISLNK() macro is indeed safe to detect a symlink (whatever it is).
3. We cannot recognize in the st_mode bits what a symlink points to. If there's a need to know this it must be checked using another function.
4. Ordinary files have without an exception only S_IFREG, ordinary directories only S_IFDIR, not a single bit more, so the S_ISREG() and S_ISDIR() macros should also be safe.

This all doesn't sound so bad. I rewrote my test program to use the S_ISxxx() macros again. With that I see the following:

• I have no entries anymore where lstat() returns -1. I can't explain why but I'm sure there were several entries some days ago. Maybe this depends on what I did before and how many applications I started. Today my while loop runs over the whole /c/Windows and /c/Program Files directories without any problem.
• The summed up size of /c/Program Filesis exactly what the Explorer says.
• The size of /c/Windows is less than what the Explorer says because we don't sum up hard links.

All good. But pp with it's current implementation leads still to ! marks in both directories. I'm not sure why. A difference is the order of the checks. Especially the hard links are checked early in my loop.

(Note: my test program is C++ because of new and delete, you must compile with g++ test.cc) test.cc.gz

[leo-arch]

The only logical difference (besides code arrangement) I see between your code and my current code is this:

1. Hardlinked files are counted/summed just once, while my code sums up only one file, but counts each of them (if a directory has three files hardlinked to the same file, I guess the user expects pp to tell there are three files, because this is what the user sees in this directory - it would be unintuitive otherwise).
2. Your code just skips (do not count/sum) block/character special and socket files (this might be fine on Windows, where there are no such files, but not on Unix).

So, it seems to be safe, according to you, to use the S_IS* macros, and we still need to check regular files before directories. Ok.

All good. But pp with it's current implementation leads still to ! marks in both directories

According the the code, there are only two ways to get an error status:

1. A directory couldn't be opened (opendir(3) failed), and this means we cannot count/sum whatever is in it.
2. lstat(2) failed for a file (or directory, we don't know): in this case we count the entry as a regular file but cannot sum up its size.

In both cases an exclamation mark (!) is prepended to the result to warn about this: the sum/count might not be accurate. This is expected and desired.

So, I guess this code should be fine now:

if (S_ISLNK(a.st_mode)) {
    info->links++;
#ifdef __CYGWIN__
/* On Unix this can be safely omitted - we get one check less and thereby better performance */
} else if (S_ISREG(a.st_mode)) {
    info->files++;
#endif /* __CYGWIN__ */
} else if (S_ISDIR(a.st_mode)) {
    info->dirs++;
    ...
} else {
    info->files++;
}

/* Pseudo-code */
hardlink_check;

sum_size;

If everything's fine, I'll commit the new code.

[muellerto]

Make a commit :)

What I saw is: the ! marks come without an exception from special system directories with missing read permissions. That's why already opendir() fails, lstat() does never fail.

But what I also saw is: in such a case when opendir() already failed you can still call stat() (without l) on it. This gives at least a value in st_size and the unreadable directory's size can be added to the sum. This results in 2GB more in my /c/Windows. And a question is then if this should be counted as error and lead to the !. OK, the amount of files, directories and links is then not correct, but the size is.

[leo-arch]

Make a commit :)

Great!

in such a case when opendir() already failed you can still call stat() (without l) on it. This gives at least a value in st_size and the unreadable directory's size can be added to the sum.

I'll give this a try. However, do note that this is not what du(1) does (I take this as reference). But I guess it makes sense.

And a question is then if this should be counted as error and lead to the !. OK, the amount of files, directories and links is then not correct, but the size is.

I see. We should be able to distinguish between a dir size error and a files count error (provided we can actually implement it). I'll consider this as well.

[leo-arch]

As to calling stat(2) when opendir(3) failed, we need to keep the following in mind:

Calling stat on a directory gives information about the directory itself (not its contents). Now, when we're calculating apparent sizes (which is the default), the size of directories themselves is not taken into account. As we read in info du:

Apparent sizes are meaningful only for regular files and symbolic links. Other file types do not contribute to apparent size.

This said, we might give this a try if we're calculating physical (not apparent) sizes (though, again, this is not what du does). Btw, I don't know whether the Explorer is calculating apparent or physical sizes.

[muellerto]

Btw, I don't know whether the Explorer is calculating apparent or physical sizes.

Both. The Explorer says:

Size:         24,3 GB (26’107’869’319 Bytes)
Size on disk: 24,4 GB (26’243’600’384 Bytes)
Contents:     141’132 files, 53’912 folders

[leo-arch]

Cool. What Clifm says on this same directory? And what about if stat is called when opendir fails? How much does the result change?

EDIT: On Unix systems, the physical size of a directory (not including its contents) is always (or almost always), 4k.

[leo-arch]

Recall that to calculate the physical size of a file we want a.st_blocks * S_BLKSIZE (instead of just a.st_size). So, our new testing code should be something like this:

if ((p = opendir(dir)) == NULL) {
    if (conf.apparent_size != 1 && stat(dir, &a) != -1)
        info->size += (a.st_blocks * S_BLKSIZE);
    info->status = errno;
    return;
}

[leo-arch]

I've realized that the new code is doing the wrong thing, namely, counting directories size twice: one in the S_ISDIR block of the while loop, and another one when opendir() fails.

[muellerto]

Actual values from my Windows machine with the latest clifm implementation:

dir	Explorer (Size on disk)	clifm pp (apparent)
no symlinks	44,6 GB (47’897’837’568 Bytes)	44,60 GiB / 47892105372 B
some inner symlinks	91,6 GB (98’395’680’768 Bytes)	91,51 GiB / 98255311860 B
/c/Program Files	51,6 GB (55’469’498’368 Bytes)	!51,28 GiB / 55058136292 B [1]
/c/Windows	24,4 GB (26’270’601’216 Bytes)	!16,78 GiB / 18018685877 B [2]

dir	Explorer	clifm pp
no symlinks	2’864 files, 586 folders	3450 (586 directories, 2864 files, 0 links)
some inner symlinks	76’675 files, 5’301 folders	81988 (5301 directories, 76673 files, 14 links)
/c/Program Files	74’063 files, 8’410 folders	!82476 (8410 directories, 74053 files, 13 links) [1]
/c/Windows	141’198 files, 53’915 folders	!195094 (53914 directories, 141179 files, 1 link) [2]

[1] some directories without read permission (opendir() fails) [2] some directories without read permission, hard links not summed up

All in all not so bad. The differences in /c/Program Files are much smaller than at the beginning. And we can explain the differences in /c/Windows with hard links.

I'm not sure about the minor size differences in the first two lines. These are typical user directories (developer working copies with .cc, .a and .o files, all accessible). The Explorer sums up more, 0.012% in the 1st line and even 0.14% in the 2nd line. Your values are exact equal to du -s --apparent-size --block-size=1, so that's a usable reference. And since we don't know the Explorer algorithm we can't explain where these numbers come from.

Close this case?

[leo-arch]

I feel satisfied with the new results (and even more with our research). Let's close it, and as always, you can reopen it if anything comes up. Thanks @muellerto!