Bench on Cygwin - supported?

[hashimaziz1]

I'm sorry if I'm raising too many issues in such a short space of time, but this one is by far the most important for me since I first started looking at bench, as it stops me in my tracks from running it.

I'm trying to use bench to benchmark the speeds of different grep-like programs (GNU grep, GNU grep with -F, and ripgrep), using some relatively complicated for loops. For reference, I'm planning to do something like this:

for i in {1..15}; \ do \ (bench 'rg -i "ajndoandajskaskaksnaodnasnakdnaosnaond" "5GB_file.txt"') 2>&1 | tee -a "/cygdrive/b/users/user/Tests/ripgrep (bench 5.2GB Ramdisked with Cache Flush, position 1).txt"; \ (bench 'LC_ALL=C grep -i "ajndoandajskaskaksnaodnasnakdnaosnaond" "5GB_file.txt"') 2>&1 | tee -a "/cygdrive/b/users/user/Tests/grep (bench 5.2GB Ramdisked with Cache Flush, position 1).txt"; \ (bench 'LC_ALL=C grep -Fi "ajndoandajskaskaksnaodnasnakdnaosnaond" "losers_5GB.txt"') 2>&1 | tee -a "/cygdrive/b/users/hashim/desktop/Tests/grep -F (bench 5.2GB Ramdisked with Cache Flush, position 1).txt"; \ done;

Before I go about trying to get that to work, I was trying to test with the most minimal form that I can run in a single command, which is this:

"bench LC_ALL=C grep -i 'ajndoandajskaskaksnaodnasnakdnaosnaond' '5GB_file.txt'"

...but even that fails. When I try to run it, I get the following output:

benchmarking LC_ALL=C grep -i "ajndoandajskaskaksnaodnasnakdnaosnaond" "losers_sorted.txt" bench.exe: ShellFailed {shellCommandLine = "LC_ALL=C grep -i 'ajndoandajskaskaksnaodnasnakdnaosnaond' 'losers_sorted.txt'", shellExitCode = ExitFailure 1}

As far as I can tell, the syntax is correct as per the documentation, and works fine in similar applications like hyperfine. Any idea what's going on here? Is this possibly a problem with the shell that bench uses on a Windows system, or is something else at work here?

Thanks for the help, I'd appreciate it a lot.

[Gabriella439]

@Kaos-Industries: What exit code does the grep command return when run outside of bench? grep can return a non-zero exit code it finds no matches

[sharkdp]

Just to save everyone some time here: We have been going through the exact same thing on the Hyperfine bug tracker: https://github.com/sharkdp/hyperfine/issues/90

My guess would be that bench also uses cmd.exe to run shell commands on Windows. And LC_ALL=C grep.. isn't valid syntax in cmd.exe.

As far as I can tell, the syntax is correct as per the documentation, and works fine in similar applications like hyperfine.

@Kaos-Industries Why would you claim that? It doesn't work in hyperfine as well, and there is a detailed analysis in the ticket above.

[hashimaziz1]

@Kaos-Industries: What exit code does the grep command return when run outside of bench? grep can return a non-zero exit code it finds no matches

grep does return an exit code of 1 as it should, due to the fact that the string I'm testing with is a dummy string with no matches. Surely bench is able to benchmark programs even if they return a non-zero exit code?

Just to save everyone some time here: We have been going through the exact same thing on the Hyperfine bug tracker: sharkdp/hyperfine#90

My guess would be that bench also uses cmd.exe to run shell commands on Windows. And LC_ALL=C grep.. isn't valid syntax in cmd.exe.

As far as I can tell, the syntax is correct as per the documentation, and works fine in similar applications like hyperfine.

@Kaos-Industries Why would you claim that? It doesn't work in hyperfine as well, and there is a detailed analysis in the ticket above.

You're right of course, when writing that I was juggling so many different tests and tools that I was struggling to keep track of what was going on. The idea that the cause of these bench errors would be the same as that of the one I came across a week or so ago in hyperfine did occur to me, but I figured the fact that there was no sign of CMD's 'x is not recognized as an internal or external command, operable program or batch file that it would have a separate cause. If it as you say and the problem here is that, like hyperfine, bench can only use CMD as its shell on a Windows environment, it's unfortunate that neither of the benchmarking tools that I've so far come across, apart from time, works successfully in a Cygwin environment.

[Gabriella439]

@Kaos-Industries: Yeah, bench rejects programs with a non-zero exit code mainly to prevent against a common benchmarking mistake where the program appears to be running much more quickly but it's actually just failing fast. You can always override this behavior by adding || : to the end of any command, which swallows a failing exit code.