sharkdp / hyperfine

A command-line benchmarking tool
Apache License 2.0
22.05k stars 358 forks source link

Problems with the default color style in Windows Powershell #507

Open Pietruszek opened 2 years ago

Pietruszek commented 2 years ago

After https://github.com/sharkdp/hyperfine/issues/427 has been resolved there's now a default coloring scheme enabled for Windows.

And while it looks good in the cmd or the default color scheme for the Windows Terminal, in Powershell the background is blue. What that means for the hyperfine's color scheme is that the below output:

PS D:\Users\ZPO\Documents\benchmarksgame\nbody\Picked> hyperfine -N --warmup 2 './Cs_projects/nbody_cs/bin/x64/Release/netcoreapp3.1/nbody_cs.exe 50000000' './nbody_gpp.exe 50000000' 'java nbody 50000000'
Benchmark 1: ./Cs_projects/nbody_cs/bin/x64/Release/netcoreapp3.1/nbody_cs.exe 50000000
  Time (mean ± σ):      7.086 s ±  0.029 s    [User: 7.062 s, System: 0.034 s]
  Range (min … max):    7.055 s …  7.145 s    10 runs

Benchmark 2: ./nbody_gpp.exe 50000000
  Time (mean ± σ):      6.948 s ±  0.019 s    [User: 6.939 s, System: 0.000 s]
  Range (min … max):    6.925 s …  6.976 s    10 runs

Benchmark 3: java nbody 50000000
  Time (mean ± σ):      7.198 s ±  0.025 s    [User: 7.175 s, System: 0.058 s]
  Range (min … max):    7.165 s …  7.249 s    10 runs

Summary
  './nbody_gpp.exe 50000000' ran
    1.02 ± 0.00 times faster than './Cs_projects/nbody_cs/bin/x64/Release/netcoreapp3.1/nbody_cs.exe 50000000'
    1.04 ± 0.00 times faster than 'java nbody 50000000'

displays like that:

obraz

so some info (like the User and System times) is very hard to read and some (like the max times or the strings after "faster than") is exactly the same color as the background.

This issue can be mitigated either by disabling the coloring using the --style option or by using a different shell (Windows Terminal or cmd), but considering that some users might be testing hyperfine for their first time in regular Powershell it is quite bad UX-wise.

Though I'm not really sure what can be done to fix this...

Two attempts at solving this I could imagine are:

  1. Making a color scheme that will work on both black and blue backgrounds
  2. Checking somehow if hyperfine is running in Powershell and choosing the colors based on that

But at least the second option seems to be a dead end, since the Powershell instance running in Windows Terminal actually has a black background as a default...

sharkdp commented 2 years ago

displays like that:

That is unfortunate. While I think the program at fault here is PowerShell (for providing such a strange default), I agree that we should attempt to solve this.

This issue can be mitigated either by disabling the coloring using the --style option or by using a different shell (Windows Terminal or cmd),

... or by changing the background color, I guess...

Two attempts at solving this I could imagine are:

1. Making a color scheme that will work on both black and blue backgrounds

I'd be okay with this if we have any other ANSI 8-bit colors left...

Checking somehow if hyperfine is running in Powershell and choosing the colors based on that

Do you know a way how to do this?

Gordon01 commented 1 year ago

I've reproduced the issue on a conhost.exe with blue background and made a PR to change color scheme.