search

dotnet / performance

This repo contains benchmarks used for testing the performance of all .NET Runtimes
MIT License
700 stars 272 forks source link

error C S0117: 'Ascii' does not contain a definition for 'EqualsIgnoreCase' #3105

Closed mayadav1 closed 1 year ago

mayadav1 commented 1 year ago

1# dotnet run -c release -f net8.0 -filter "System.IO.Tests.Perf_FileStream.Write" 2# dotnet run -c release -f net8.0 --filter "System.IO.Tests.Perf_FileStream.Write" --corerun C:\Users\mayadav1\Desktop\runtime\artifacts\bin\testhost\net8.0-windows-Release-x64\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts C:\Users\mayadav1\Desktop\net8.0half220623\

Both the commands emit the error as below:

C:\Users\mayadav1\Desktop\performance\src\benchmarks\micro\libraries\System.Text.Encoding\Perf.Ascii.cs(53,70): error C S0117: 'Ascii' does not contain a definition for 'EqualsIgnoreCase' [C:\Users\mayadav1\Desktop\performance\src\benchmar ks\micro\MicroBenchmarks.csproj::TargetFramework=net8.0] C:\Users\mayadav1\Desktop\performance\src\benchmarks\micro\libraries\System.Text.Encoding\Perf.Ascii.cs(56,70): error C S0117: 'Ascii' does not contain a definition for 'EqualsIgnoreCase' [C:\Users\mayadav1\Desktop\performance\src\benchmar ks\micro\MicroBenchmarks.csproj::TargetFramework=net8.0] C:\Users\mayadav1\Desktop\performance\src\benchmarks\micro\libraries\System.Text.Encoding\Perf.Ascii.cs(59,76): error C S0117: 'Ascii' does not contain a definition for 'EqualsIgnoreCase' [C:\Users\mayadav1\Desktop\performance\src\benchmar ks\micro\MicroBenchmarks.csproj::TargetFramework=net8.0] C:\Users\mayadav1\Desktop\performance\src\benchmarks\micro\libraries\System.Text.Encoding\Perf.Ascii.cs(62,69): error C S0117: 'Ascii' does not contain a definition for 'EqualsIgnoreCase' [C:\Users\mayadav1\Desktop\performance\src\benchmar ks\micro\MicroBenchmarks.csproj::TargetFramework=net8.0] C:\Users\mayadav1\Desktop\performance\src\benchmarks\micro\libraries\System.Text.Encoding\Perf.Ascii.cs(65,69): error C S0117: 'Ascii' does not contain a definition for 'EqualsIgnoreCase' [C:\Users\mayadav1\Desktop\performance\src\benchmar ks\micro\MicroBenchmarks.csproj::TargetFramework=net8.0] C:\Users\mayadav1\Desktop\performance\src\benchmarks\micro\libraries\System.Text.Encoding\Perf.Ascii.cs(68,75): error C S0117: 'Ascii' does not contain a definition for 'EqualsIgnoreCase' [C:\Users\mayadav1\Desktop\performance\src\benchmar ks\micro\MicroBenchmarks.csproj::TargetFramework=net8.0]

The build failed. Fix the build errors and run again.

Please help me to fix it.

mayadav1 commented 1 year ago

After commenting EqualsIgnoreCase benchmarks in Perf.Ascii.cs, it works for me.

cincuranet commented 1 year ago

These were added in https://github.com/dotnet/performance/pull/3016 (and via https://github.com/dotnet/runtime/commit/b0b133bb2f1da4f8e6970359f02dbdec7e65717e). Do you have fresh enough runtime?

mayadav1 commented 1 year ago

@cincuranet I am using latest commit "5ec1f1556baa7f0ee8e0117de8756a01c23022c6". I took it 27/06/23.

cincuranet commented 1 year ago

Is that one in your PATH (or do you have fresh enough net8.0 in your PATH)? You can use benchmarks_ci.py script that will handle it for you (and that's what we use when running benchmarks).

py .\scripts\benchmarks_ci.py -f net8.0 --filter "System.IO.Tests.Perf_FileStream.Write*" --corerun "C:\Users\mayadav1\Desktop\runtime\artifacts\bin\testhost\net8.0-windows-Release-x64\shared\Microsoft.NETCore.App\8.0.0\corerun.exe"
mayadav1 commented 1 year ago

Thanks @cincuranet . It is in my path. And it works fine with your suggestion . I have a need to run using "dotnet" command rather than with the script.

cincuranet commented 1 year ago

What does dotnet --info print?

mayadav1 commented 1 year ago

C:\Users\mayadav1\Desktop\runtime>dotnet --info .NET SDK: Version: 8.0.100-preview.5.23303.2 Commit: 3fe444af72

Runtime Environment: OS Name: Windows OS Version: 10.0.22621 OS Platform: Windows RID: win10-x64 Base Path: C:\Users\mayadav1\Desktop\runtime.dotnet\sdk\8.0.100-preview.5.23303.2\

.NET workloads installed: There are no installed workloads to display.

Host: Version: 8.0.0-preview.5.23280.8 Architecture: x64 Commit: bc78804f5d

.NET SDKs installed: 8.0.100-preview.5.23303.2 [C:\Users\mayadav1\Desktop\runtime.dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 8.0.0-preview.5.23302.2 [C:\Users\mayadav1\Desktop\runtime.dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 8.0.0-preview.5.23280.8 [C:\Users\mayadav1\Desktop\runtime.dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 8.0.0-preview.5.23302.2 [C:\Users\mayadav1\Desktop\runtime.dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found: x86 [C:\Program Files (x86)\dotnet] registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables: Not set

global.json file: C:\Users\mayadav1\Desktop\runtime\global.json

Learn more: https://aka.ms/dotnet/info

Download .NET: https://aka.ms/dotnet/download

mayadav1 commented 1 year ago

Hi cincuranet,

Can you reproduce it at your end. Or I am only facing this issue .

cincuranet commented 1 year ago

Try running dotnet build -c Release -f net8.0 -p:PERFLAB_TARGET_FRAMEWORKS=net8.0.

cincuranet commented 1 year ago

I have a need to run using "dotnet" command rather than with the script.

What's your scenario that you need to use dotnet instead of the script (which uses dotnet under the hood anyway)?

mayadav1 commented 1 year ago

Script run was showing regression for "System.IO.Tests.Perf_FileStream.Write" but dotnet run doesn't show . I know that it was a noise which was suppose to feature of script.

cincuranet commented 1 year ago

I'm not sure what "showing regression" are you talking about. Can you show output...

mayadav1 commented 1 year ago

Sorry, I couldn't make you clear. I am apologize.

I did comparison net8.0-preview5 vs net6.0. And I found gap between script run and dotnet run. script run. python scripts/benchmarks_ci.py --architecture x64 -c Release -f net8.0 --filter "System.IO.Tests.Perf_FileStream.Write*" --corerun "C:\Users\mayadav1\Desktop\runtime\artifacts\bin\testhost\net8.0-windows-Release-x64\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" --bdn-artifacts C:\Users\mayadav1\Desktop\net8.0half220623\ output : summary: better: 18, geomean: 1.395 worse: 1, geomean: 1.161 total diff: 19

Slower diff/base Base Median (ns) Diff Median (ns) Modality
System.IO.Tests.Perf_FileStream.Write(fileSize: 104857600, userBufferSize: 4096, options: None) 1.16 179070450.00 207968100.00 bimodal
Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.IO.Tests.Perf_FileStream.WriteByte(fileSize: 1024, options: Asynchronous) 1.58 466933.90 294880.93
System.IO.Tests.Perf_FileStream.WriteByte(fileSize: 1024, options: None) 1.58 459612.69 290588.75
System.IO.Tests.Perf_FileStream.WriteAsync(fileSize: 1024, userBufferSize: 1024, options: None) 1.57 438684.64 280055.13
System.IO.Tests.Perf_FileStream.Write(fileSize: 1024, userBufferSize: 1024, options: None) 1.53 427014.50 278886.92
System.IO.Tests.Perf_FileStream.Write(fileSize: 1024, userBufferSize: 1024, options: Asynchronous) 1.53 435049.83 285073.26
System.IO.Tests.Perf_FileStream.WriteAsync(fileSize: 1024, userBufferSize: 1024, options: Asynchronous) 1.52 441422.15 290925.24
System.IO.Tests.Perf_FileStream.Write_NoBuffering(fileSize: 1048576, userBufferSize: 16384, options: Asynchronous) 1.46 3071909.90 2103058.59
System.IO.Tests.Perf_FileStream.Write_NoBuffering(fileSize: 1048576, userBufferSize: 16384, options: None) 1.44 2771404.17 1927024.31
System.IO.Tests.Perf_FileStream.WriteAsync_NoBuffering(fileSize: 1048576, userBufferSize: 16384, options: None) 1.42 2901251.04 2042786.61
System.IO.Tests.Perf_FileStream.Write(fileSize: 1048576, userBufferSize: 512, options: None) 1.35 4077678.91 3015326.25
System.IO.Tests.Perf_FileStream.WriteAsync_NoBuffering(fileSize: 1048576, userBufferSize: 16384, options: Asynchronous) 1.34 3252391.25 2426987.95
System.IO.Tests.Perf_FileStream.Write(fileSize: 1048576, userBufferSize: 4096, options: None) 1.32 4015832.81 3039521.25
System.IO.Tests.Perf_FileStream.WriteAsync(fileSize: 1048576, userBufferSize: 512, options: None) 1.31 4600996.88 3521668.75
System.IO.Tests.Perf_FileStream.Write(fileSize: 1048576, userBufferSize: 4096, options: Asynchronous) 1.29 5080572.92 3926701.56
System.IO.Tests.Perf_FileStream.WriteAsync(fileSize: 1048576, userBufferSize: 4096, options: None) 1.28 4428953.13 3469442.19
System.IO.Tests.Perf_FileStream.Write(fileSize: 1048576, userBufferSize: 512, options: Asynchronous) 1.28 4999435.42 3919975.00
System.IO.Tests.Perf_FileStream.WriteAsync(fileSize: 1048576, userBufferSize: 512, options: Asynchronous) 1.22 5767532.29 4740925.00
System.IO.Tests.Perf_FileStream.WriteAsync(fileSize: 1048576, userBufferSize: 4096, options: Asynchronous) 1.20 5586979.17 4661594.79

dotnet run: dotnet run -c release -f net8.0 --filter "System.IO.Tests.Perf_FileStream.Write*" --corerun C:\Users\mayadav1\Desktop\runtime\artifacts\bin\testhost\net8.0-windows-Release-x64\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts C:\Users\mayadav1\Desktop\net8.0half220623\

output:

summary: better: 9, geomean: 1.375 total diff: 9

No Slower results for the provided threshold = 10% and noise filter = 2 ns.

Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.IO.Tests.Perf_FileStream.WriteByte(fileSize: 1024, options: Asynchronous) 1.60 466933.90 291041.42
System.IO.Tests.Perf_FileStream.Write(fileSize: 1024, userBufferSize: 1024, options: Asynchronous) 1.53 435049.83 284942.25
System.IO.Tests.Perf_FileStream.WriteAsync(fileSize: 1024, userBufferSize: 1024, options: Asynchronous) 1.51 441422.15 292376.18
System.IO.Tests.Perf_FileStream.Write_NoBuffering(fileSize: 1048576, userBufferSize: 16384, options: Asynchronous) 1.47 3071909.90 2088580.47
System.IO.Tests.Perf_FileStream.WriteAsync_NoBuffering(fileSize: 1048576, userBufferSize: 16384, options: Asynchronous) 1.34 3252391.25 2427166.07
System.IO.Tests.Perf_FileStream.Write(fileSize: 1048576, userBufferSize: 4096, options: Asynchronous) 1.30 5080572.92 3897138.28
System.IO.Tests.Perf_FileStream.Write(fileSize: 1048576, userBufferSize: 512, options: Asynchronous) 1.29 4999435.42 3867601.56
System.IO.Tests.Perf_FileStream.WriteAsync(fileSize: 1048576, userBufferSize: 512, options: Asynchronous) 1.21 5767532.29 4755347.92
System.IO.Tests.Perf_FileStream.WriteAsync(fileSize: 1048576, userBufferSize: 4096, options: Asynchronous) 1.18 5586979.17 4723020.83

diff command:

dotnet run --base "C:\Users\mayadav1\Desktop\net6.0.18\results" --diff "C:\Users\mayadav1\Desktop\net8.0half220623\results" --threshold 10% --noise 2ns --full-id

Is this useful for you?

cincuranet commented 1 year ago

There's really nothing special that the script is doing in regards to running the benchmark, the work is done by BDN. The numbers you see is IMO the result of the fact that the IO benchmarks are by definition very noisy. You can check for example here (click on the graph) and see that the values are all over the place.

mayadav1 commented 1 year ago

@cincuranet , Thank you for the explanation .