search

petabridge / NBench

Performance benchmarking and testing framework for .NET applications :chart_with_upwards_trend:
https://nbench.io/
Apache License 2.0
532 stars 46 forks source link

Need integration tests to determine how NBench.Runner.exe binary location impacts test .DLL dependencies #3

Closed Aaronontheweb closed 8 years ago

Aaronontheweb commented 8 years ago

There isn't a short way to summarize this issue in the title, so I'll illustrate by example:

Akka.Benchmarks.dll depends on Akka.dll and some others, all of which are stored in directory /src/Akka.Benchmarks/bin/Debug/

We need to write tests to verify that these behaviors are identical

Current Working Directory DLL arg passed to NBench.Runner.exe
/src/Akka.Benchmarks/bin/Debug/ Akka.Benchmarks.dll
~/ /src/Akka.Benchmarks/bin/Debug/Akka.Benchmarks.dll
Aaronontheweb commented 8 years ago

Boom - looks like the plugin system exposed an issue with this.

D:\Repositories\NBench\bin\NBench.Runner\NBench.Runner.exe "D:\Repositories\NBench\tests\NBench.PerformanceCounters.Tests.Performance\bin\Release\NBench.PerformanceCounters.Tests.Performance.dll" "output-directory="D:\Repositories\NBench\PerfResults""
WARNING: Unable to find valid configurator for NBench.PerformanceCounters.PerformanceCounterMeasurementAttribute - skipping...
WARNING: Unable to find valid configurator for NBench.PerformanceCounters.PerformanceCounterMeasurementAttribute - skipping...
WARNING: Unable to find valid configurator for NBench.PerformanceCounters.PerformanceCounterMeasurementAttribute - skipping...
WARNING: Unable to find valid configurator for NBench.PerformanceCounters.PerformanceCounterMeasurementAttribute - skipping...
------------ STARTING NBench.PerformanceCounters.Tests.Performance.ProcessJitSpec+TimeInJit ----------
--------------- BEGIN WARMUP ---------------
Elapsed: 00:00:00.0020821
--------------- END WARMUP ---------------

--------------- BEGIN WARMUP ---------------
Elapsed: 00:00:00.0000957
--------------- END WARMUP ---------------

Unable to find, via reflection, the configurators used to instrument metrics defined in a third party DLL

Aaronontheweb commented 8 years ago

Looks like the location of the binary actually works as intended - it's more of an issue with how we did caching inside ReflectionDiscovery. Closing this for now - the integration tests in my next PR pick this up.