devkitPro / wut

Let's try to make a Wii U Toolchain / SDK for creating rpx/rpl.
zlib License
236 stars 52 forks source link

wutdevoptab: implement __wut_fsa_get_stat_from_dir to allow optimized implemenation of readdir that avoid an addtional stat #340

Closed Maschell closed 5 months ago

Maschell commented 9 months ago

In the FAT32 driver that is used on the Wii U each readdir and stat can be really slow, Especially in directory with many files. This functions helps dev to avoid stat calls in a readdir loop by reading the current dir entry from the DIR*.

Example usage:

std::string dirPath = "fs:/vol/external01";
DIR* dir = opendir(dirPath.c_str());
if (dir)
{
  struct dirent *ent;
  while ((ent = readdir (dir)) != NULL)
  {
    struct stat st = {};
    if (!__wut_fsa_get_stat_from_dir (dir, &st))
    {
      std::string fullpath = dirPath + "/" + ent->d_name;
      ::stat (fullpath.c_str(), &st);
    }
    OSReport ("\"%s\" size: %lld bytes\n ", ent->d_name, st.st_size);
  }
  closedir (dir);
}

Benchmarks (tested by listing the dir via ftp, so all numbers include an overhead, but I only cared of the relative not absolute numbers) (Already using mocha 0.3.0-dev with experimental fs patches)

Directory 1, 1000 files, each file has a very long and similar name e.g. `this_is_a_normal_filename_but_long_0atcpsdz.xvg`:
     with addtional stat: 125.883 sec
 without additional stat: 1.814 sec

Directory 2, 1000 files, each file has a unique and short name (e.g. `0b2vj3w4.2nl`):
    with addtional stat: 7.747
without additional stat: 1.383 sec

I know this is not very clean and not a very optimal solution, but the impact is really huge if you look at the benchmarks. I would rather have this helper function inside wut than having it in each homebrew app.

WinterMute commented 5 months ago

Superseded by stat extension in dirent struct https://github.com/devkitPro/newlib/commit/c4ef0a7e89fd1e75c746642d20f885064f8aedfa