Crashing benchmark

[Drawaes]

https://github.com/Drawaes/PerfBenchmarks

If you grab this benchmark and then do

Benchmark ManyReadersRareWrite

I get this following stack trace.

Unhandled Exception: System.UnauthorizedAccessException: Access to the path 'C:\code\PerfBenchmarks\Benchmarks\bin\Release\net462\BDN.Generated.exe' is denied. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalDelete(String path, Boolean checkHost) at BenchmarkDotNet.Toolchains.Roslyn.Generator.Cleanup(ArtifactsPaths artifactsPaths) at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func2 toolchainProvider, IResolver resolver) at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark[] benchmarks, ILogger logger, String title, IConfig config, String rootArtifactsFolderPath, Func2 toolchainProvider, IResolver resolver) at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark[] benchmarks, IConfig config, Func`2 toolchainProvider)

[AndreyAkinshin]

I tried several different computers, but everything works fine, I can't reproduce this issue. =( @adamsitnik do you have any ideas?

[adamsitnik]

@AndreyAkinshin I could not reproduce it. However I get similar error once in a while (1/100 runs?)

@Drawaes Could you check what is locking this folder?

SO says:

An UnauthorizedAccessException means one of 4 things:

The caller does not have the required permission. The file is an executable file that is in use. Path is a directory. Path specified a read-only file.

So it's either permissions or file in use.

[mattwarren]

Could it be an anti-virus scanner that has the file/folder in-use?

[Drawaes]

I was sure I have my c:\code excluded but let me double check

[Drawaes]

Looks like an over zealous virus scanner is to blame. If I have it off I never see it, but have seen it again with it on. Thanks for investigating, sorry to waste your time.

[mtschneiders]

This error happened to me as well, after some digging, it appears to be the Razer Game Scanner service that was locking BDN.Generated.exe in the moment it was being deleted.

Stopping the service made it work.

[Drawaes]

Interesting. For me it was Avast

[mtschneiders]

Considering the lock doesn't last long, a simple retry mechanism could do the trick.

[Drawaes]

Alright game on. Is it a good idea to retry of maybe not reuse files during the benchmark?

I suspect the issue is rapid generation of executables causes a bunch of different things in different environments to trigger and do a read. If i wanted to benchmark in prod at work I couldn't disable the virus scanner so would be stuck.

[AndreyAkinshin]

Is it a good idea to retry

It's worth a try. We use the following idea in the exporters: if there is a file with target name, we try to delete it; in the case of fail, we pick another name. See https://github.com/dotnet/BenchmarkDotNet/blob/1a8559f24c1e36bf235943d89e67ad83e5fcd9ce/src/BenchmarkDotNet.Core/Exporters/ExporterBase.cs#L29

maybe not reuse files during the benchmark?

Probably it's even a better idea. So, we can generate name of executable file using the job name (instead of fixed BDN.Generated.exe.) @adamsitnik, what do you think?

[adamsitnik]

We have two problems:

1. Something is not allowing us to remove files. We can fix it by not removing them at all (use [KeepBenchmarkFiles] today), or remove them at the end of whole benchmarking process.
2. Anti Virus is not allowing us to run the executable we have just generated. I have spoken with some security guys when I met them at DotNext conference and they told me that for some AV it's the default policy: block every new process. Some AV allows to exclude given folder for this rule. But you need to have Admin rights to do it, which might not be possible in the corporate env. So the devs will have to create IT support ticket to get it running. The only solution I see is to display a nice error that explains the problem and ask the user to use our InProcessToolchain which does not emit any new executable

[Drawaes]

Mine was not the A/V not allowing it to run. It was the overwrite because it was still "scanning" I agree getting around the A/V stopping it from running is fruitless.

[adamsitnik]

I have just changed the default behavior, the default folder/program name is new guid and we remove all artifacts after printing the results.

Please let me know if you ever run into this bug again @Drawaes

[Drawaes]

Cool! I will turn my evil virus scanner on and give it a crack :) but looks good to me