How to make benchmarks run fine on windows?

[luceat-lux-vestra]

I want to benchmark for some of windows default commands like rd, robocopy.

but I can't make it work because of error.

Benchmark 1: rmdir /S /Q D:/tmp/ Error: The preparation command terminated with a non-zero exit code. Append ' || true' to the command if you are sure that this can be ignored.

It works well with release files of this project, without rmdir lines in batch script.

Please help.

Not working code here.(windows batch script)

@echo off
for %%G in (0,1G) do (
 for %%F in (10,10_000,100_000,1M) do (
    hyperfine --warmup 3 -N ^
      --export-markdown "benches/remove_%%F_files_%%G_bytes.md" ^
      --export-json "benches/remove_%%F_files_%%G_bytes.json" ^
      --prepare "ftzz g -n %%F -b %%G "D:/tmp/ftzz"" ^
        "rmdir /S /Q "D:/tmp/"" ^
        "./target/release/rm_stdlib "D:/tmp/ftzz"" ^
        "./target/release/rm_rayon "D:/tmp/ftzz"" ^
        "./target/release/rm_remove_dir_all "D:/tmp/ftzz"" ^
        "./target/release/rmz "D:/tmp/ftzz""
    )

  hyperfine --warmup 3 -N ^
    --export-markdown "benches/remove_100_000_files_%%G_bytes_0_depth.md" ^
    --export-json "benches/remove_100_000_files_%%G_bytes_0_depth.json" ^
    --prepare "ftzz g -n 100_000 -b %%G -d 0 "D:/tmp/ftzz"" ^
      "rmdir /S /Q "D:/tmp/"" ^
      "./target/release/rm_stdlib "D:/tmp/ftzz"" ^
      "./target/release/rm_rayon "D:/tmp/ftzz"" ^
      "./target/release/rm_remove_dir_all "D:/tmp/ftzz"" ^
      "./target/release/rmz "D:/tmp/ftzz""

  hyperfine --warmup 3 -N ^
    --export-markdown "benches/remove_100_000_files_%%G_bytes_5_files_per_dir.md" ^
    --export-json "benches/remove_100_000_files_%%G_bytes_5_files_per_dir.json" ^
    --prepare "ftzz g -n 100_000 -b %%G -r 5 "D:/tmp/ftzz"" ^
      "rmdir /S /Q "D:/tmp/"" ^
      "./target/release/rm_stdlib "D:/tmp/ftzz"" ^
      "./target/release/rm_rayon "D:/tmp/ftzz"" ^
      "./target/release/rm_remove_dir_all "D:/tmp/ftzz"" ^
      "./target/release/rmz "D:/tmp/ftzz""
)

[SUPERCILEX]

You can use --show-output with hyperfine to see the actual failure. My guess: have you install ftzz?

[luceat-lux-vestra]

Thanks.

It seems hyperfine problem? I can't pass command line options to commands.

I installed fizz and when I run this command on terminal

hyperfine --warmup 3 -N --show-output --prepare "ftzz g -n 100 -b 100 ""D:/tmp/ftzz""" "rmdir /S /Q D:/tmp/" got this error.

Benchmark 1: rmdir /S /Q D:/tmp/
About 100 files will be generated in approximately 100 directories distributed across a tree of maximum depth 5 where each directory contains approximately 3 other directories. Each file will contain approximately 1 byte of random data.
Created 32 files (29 B) across 214 directories.
rmdir: failed to remove '/S': The system cannot find the file specified.
rmdir: failed to remove '/Q': The system cannot find the file specified.
rmdir: failed to remove 'D:/tmp/': The directory is not empty.
Error: Command terminated with non-zero exit code: 1. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.

[luceat-lux-vestra]

Problem solved. -N made options for command not usable.

Shell script converted to batch script on windows and it's working. (It's from remote windows, line breaks not applied here.)

@echo off
for %%G in (0,1G) do (
 for %%F in (10,10_000,100_000,1M) do (
    hyperfine --warmup 3 ^
      --export-markdown "./benches/remove_%%F_files_%%G_bytes.md" ^
      --export-json "./benches/remove_%%F_files_%%G_bytes.json" ^
      --prepare "ftzz g -n %%F -b %%G D:/tmp/ftzz" ^
        "rmdir /S /Q ""D:/tmp""" ^
        "D:/Repositories/fuc/target/release/rm_stdlib D:/tmp/ftzz" ^
        "D:/Repositories/fuc//target/release/rm_rayon D:/tmp/ftzz" ^
        "D:/Repositories/fuc//target/release/rm_remove_dir_all D:/tmp/ftzz" ^
        "D:/Repositories/fuc//target/release/rmz D:/tmp/ftzz"
    )

  hyperfine --warmup 3 -N ^
    --export-markdown "benches/remove_100_000_files_%%G_bytes_0_depth.md" ^
    --export-json "benches/remove_100_000_files_%%G_bytes_0_depth.json" ^
    --prepare "ftzz g -n 100_000 -b %%G -d 0 D:/tmp/ftzz" ^
      "rmdir /S /Q ""D:/tmp""" ^
      "D:/Repositories/fuc/target/release/rm_stdlib D:/tmp/ftzz" ^
      "D:/Repositories/fuc/target/release/rm_rayon D:/tmp/ftzz" ^
      "D:/Repositories/fuc/target/release/rm_remove_dir_all D:/tmp/ftzz" ^
      "D:/Repositories/fuc/target/release/rmz D:/tmp/ftzz"

  hyperfine --warmup 3 -N ^
    --export-markdown "benches/remove_100_000_files_%%G_bytes_5_files_per_dir.md" ^
    --export-json "benches/remove_100_000_files_%%G_bytes_5_files_per_dir.json" ^
    --prepare "ftzz g -n 100_000 -b %%G -r 5 D:/tmp/ftzz" ^
      "rmdir /S /Q ""D:/tmp""" ^
      "D:/Repositories/fuc/target/release/rm_stdlib ""D:/tmp/ftzz""" ^
      "D:/Repositories/fuc/target/release/rm_rayon ""D:/tmp/ftzz""" ^
      "D:/Repositories/fuc/target/release/rm_remove_dir_all ""D:/tmp/ftzz""" ^
      "D:/Repositories/fuc/target/release/rmz ""D:/tmp/ftzz"""
)

[luceat-lux-vestra]

rm_rayon is fastest on windows 11(tested on HDD).

[SUPERCILEX]

As expected for windows right now since we're just forwarding to https://github.com/XAMPPRocky/remove_dir_all/blob/master/Cargo.toml. On that note, I noticed that the parallel feature is disabled by default. If you enable it here, https://github.com/SUPERCILEX/fuc/blob/cb34600eeb590fbb3f30f88ee447154ee356c744/fuc_engine/Cargo.toml#L24 you should get about the same performance.

Can you try that and share benchmarks?

[luceat-lux-vestra]

@SUPERCILEX not relevant, but do you know some windows internals?

Hm... I thought wrongly.

I hope to get some advice on this.

Thanks.

[SUPERCILEX]

Sadly no, I don't know much of anything about Windows internals.

[luceat-lux-vestra]

benches.zip

I can't wait to finish remove benchmark running on HDD. But I guess almost last phase.

need to adjust values or run benches on SSD... HDD takes too much time.

[SUPERCILEX]

Thanks! So that's with the parallel feature enabled? Seems like a win for the large benchmarks.

[luceat-lux-vestra]

Yes, I enabled parallel feature for remove_dir_all.

[SUPERCILEX]

Sweet! I just pushed https://github.com/SUPERCILEX/fuc/commit/844b20b9c94154f34ebe60b6166355bb8f2a52df

[luceat-lux-vestra]

benches (2).zip

Added copy benchmarks, just stdlib, rayon, cpz. I'll try this with robocopy later.