search

scullionw / dirstat-rs

(fastest?) disk usage cli, similar to windirstat.
MIT License
157 stars 11 forks source link

Rework windows "Physical size" calculation, added Unit Tests #9

Open Mart-Bogdan opened 1 year ago

Mart-Bogdan commented 1 year ago

Added:

  1. Unit tests
  2. Github workflow to run:
    • Tests
    • clippy
    • code formatting checks
  3. Changed implementation on windows. Fixes #5.

Current implementation provides more correct size for files of size around 2k and would show it as 4k physical size. CompressedFileSize would report smaller size.

But it still have problems of being inconsistent with Windows explorer for files smaller than 512 bytes. Such small files are actually stored inside directory, not in own clusters, and explorer reports them as 0.

Actual threshold is not 512, it's dynamic, for now I don't know how to detect it.

Also this algorithm ignores alternate data streams. Same for previous implementation. To be more precise -- alternate data streams can also be compressed, as I learned recently.

Main difference: we are using syscall that uses file handle (that we already have opened) instead of getting by file path. And getting file path for long path required reformatting path into special format. Rust's std File is performing this for us under the hood. I first tried to fix this, and format string with \\.\ prefix (for network path UNC prefix) to enable long path, but that's big tricky function that have to be maintained, and using GetFileInformationByHandleEx gives us more flexibility for further improvements.

Mart-Bogdan commented 1 year ago

It has lots of clippy warnings, but I think they should be resolved by separate PR. And then we can even reject PRs that don't stratify clippy.

Mart-Bogdan commented 1 year ago

I have written simple benchmark, and it shows that this solution is even faster:

https://github.com/Mart-Bogdan-TMP/dirstat-rs-benchmark/actions/runs/3504907900/jobs/5870888010

physical size/new       time:   [341.84 ms 348.53 ms 356.52 ms]
physical size/old       time:   [595.32 ms 603.11 ms 611.31 ms]
logical size/new        time:   [330.95 ms 332.91 ms 334.89 ms]
logical size/old        time:   [343.66 ms 345.34 ms 347.08 ms]

But we need to take this tests with grain of salt. It only measures functions/syscall overhead. Filesystem cache takes most on itself.

(if we want to exclude FS cache from equation, we would need another testing methodology, that would flush caches. It's possible, at least on Linux).

On my local machine I've got following results:

physical size/new       time:   [128.04 ms 128.74 ms 129.44 ms]
physical size/old       time:   [161.32 ms 162.06 ms 162.83 ms]
logical size/new        time:   [130.70 ms 131.47 ms 132.24 ms]
logical size/old        time:   [131.18 ms 131.90 ms 132.64 ms]
Mart-Bogdan commented 1 year ago

Hello, @scullionw. Can you please review my PR?

scullionw commented 1 year ago

Hi! Thanks for the PR!

Sorry about the delays, quite busy these days. This is a pretty big one and I can't review it all right now.

However, if you want to split the PR (tests, workflows, changes to windows impl), we can merge the first two!

Mart-Bogdan commented 1 year ago

Hi. Hm. I guess it's possible. Would need to split tests for new functionality out.

Could try it till Friday.

P.S. wanted to work on other stuff regarding this app as well.

Mart-Bogdan commented 1 year ago

image Tests are failing without code. I guess I would not commit in not working tests.

Mart-Bogdan commented 1 year ago

I have created PR #10 with tests