Tests/benchmarks failing

grokys commented 3 years ago

This is mentioned in #30 but that has been closed: the test and benchmarks suite currently does not run. I can see the following problems in a freshly cloned repro:

Benchmarks fails to compile:

5>Precompiling D:\projects\grokys\XamlX\src\Benchmarks\bin\Debug\netcoreapp2.2\Benchmarks.dll
5>D:\projects\grokys\XamlX\src\Benchmarks\Benchmarks.csproj(24,9): error : Could not execute because the specified command or file was not found.
5>D:\projects\grokys\XamlX\src\Benchmarks\Benchmarks.csproj(24,9): error : Possible reasons for this include:
5>D:\projects\grokys\XamlX\src\Benchmarks\Benchmarks.csproj(24,9): error :   * You misspelled a built-in dotnet command.
5>D:\projects\grokys\XamlX\src\Benchmarks\Benchmarks.csproj(24,9): error :   * You intended to execute a .NET program, but dotnet-D:\projects\grokys\XamlX\src\Benchmarks\\..\BenchmarksCompiler\bin\Debug\netcoreapp2.2\BenchmarksCompiler.dll does not exist.
5>D:\projects\grokys\XamlX\src\Benchmarks\Benchmarks.csproj(24,9): error :   * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.
5>D:\projects\grokys\XamlX\src\Benchmarks\Benchmarks.csproj(24,9): error MSB3073: The command "dotnet   "D:\projects\grokys\XamlX\src\Benchmarks\\..\BenchmarksCompiler\bin\Debug\netcoreapp2.2\BenchmarksCompiler.dll" "D:\projects\grokys\XamlX\src\Benchmarks\/obj\Debug\netcoreapp2.2\Benchmarks.dll"" exited with code 1.

File.WriteAllBytes("testasm.dll", data) throws (#30)

Message: 
    System.IO.IOException : The process cannot access the file '.\XamlX\tests\CecilNetstandardTests\bin\Debug\netcoreapp3.1\testasm.dll' because it is being used by another process.
  Stack Trace: 
    FileStream.ValidateFileHandle(SafeFileHandle fileHandle)
    FileStream.CreateFileOpenHandle(FileMode mode, FileShare share, FileOptions options)
    FileStream.ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
    FileStream.ctor(String path, FileMode mode, FileAccess access, FileShare share)
    File.InternalWriteAllBytes(String path, Byte[] bytes)
    File.WriteAllBytes(String path, Byte[] bytes)
    CompilerTestBase.Compile(String xaml) line 75
    BasicCompilerTests.Compiler_Should_Compile_Simple_Xaml(Boolean populate) line 27

CompilerTestBase.GetCallbacks fails on net47:

Message: 
    System.IO.FileNotFoundException : Could not load file or assembly 'CecilNetstandardTests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
  Stack Trace: 
    Signature.GetSignature(Void* pCorSig, Int32 cCorSig, RuntimeFieldHandleInternal fieldHandle, IRuntimeMethodInfo methodHandle, RuntimeType declaringType)
    RuntimeMethodInfo.FetchNonReturnParameters()
    RuntimeMethodInfo.GetParameters()
    TypeExtensions.GetParametersCached(MethodBase method)
    Expression.ValidateMethodAndGetParameters(Expression instance, MethodInfo method)
    Expression.Call(MethodInfo method, Expression arg0)
    CompilerTestBase.GetCallbacks(Type created) line 124
    CompilerTestBase.Compile(String xaml) line 65
    BasicCompilerTests.Compiler_Should_Compile_Simple_Xaml(Boolean populate) line 27

CecilTests/CecilNetstandardTests are being built for netstandard2.0 which is not a valid TFM for unit tests due to it being an API not a platform causing the tests to show up in the VS test explorer but not be runnable

rstm-sf commented 3 years ago

* `CompilerTestBase.GetCallbacks` fails on net47:

Why this happens is described here https://github.com/kekekeks/XamlX/pull/31#issuecomment-708623730

In fact, conditionally, they work. But they need to be fixed so that they really work

* `File.WriteAllBytes("testasm.dll", data)`  throws

I did not understand why this is happening, perhaps windows OS does not have time to release dll files, but this was decided in #31, since this strut is not needed for batch tests

* `CecilTests`/`CecilNetstandardTests` are being built for `netstandard2.0` which is not a valid TFM for unit tests due to [it being an API not a platform](https://xunit.net/docs/why-no-netstandard) causing the tests to show up in the VS test explorer but not be runnable

Also, it affects dotnet test. I have a hunch that if you select what netstandard was for, then the problems in VS will disappear

kekekeks commented 3 years ago

CecilNetstandardTests project has to be built for netstandard2.0 as well as actual runtime TFMs because the binaries built for netstandard2.0 are being feed to Cecil in order to check that compiler is capable of processing netstandard2.0-targeting libraries. CecilTests most likely doesn't has to be built for netstandard2.0

"Ghost" tests in VS test explorer is a Visual Studio bug.

rstm-sf commented 3 years ago

Also, it affects dotnet test. I have a hunch that if you select what netstandard was for, then the problems in VS will disappear

Okey. It looks like to do this, we will need to rewrite the tests to separate the test asserts from the classes

dotnet test trying run for testclasses... https://github.com/rstm-sf/XamlX/runs/1270736821?check_suite_focus=true#step:5:36

rstm-sf commented 3 years ago

"Ghost" tests in VS test explorer is a Visual Studio bug.

It seems to me that these are problems test runner

rstm-sf commented 3 years ago

CecilNetstandardTests project has to be built for netstandard2.0 as well as actual runtime TFMs because the binaries built for netstandard2.0 are being feed to Cecil in order to check that compiler is capable of processing netstandard2.0-targeting libraries. CecilTests most likely doesn't has to be built for netstandard2.0

I do not see any particular problem in separating classes from tests. See https://github.com/rstm-sf/XamlX/actions/runs/313556568

rstm-sf commented 3 years ago

dotnet test trying run for testclasses...

It works now #35

Thus, it remains to figure out the following

Benchmarks fails to compile
Running tests with VS in a XamlParserTests project

rstm-sf commented 3 years ago

"Ghost" tests in VS test explorer is a Visual Studio bug.

An example that this is most likely a dotnet test problem

rstm-sf commented 3 years ago

Thus, it remains to figure out the following

Running tests with VS in a XamlParserTests project

~~https://github.com/microsoft/vstest/issues/655#issuecomment-344556398~~

https://github.com/kekekeks/XamlX/blob/697a419675a8055b7ef9a1904360359522b30616/tests/xunit.runner.json#L1

rstm-sf commented 3 years ago

@grokys tests in VS looks good https://github.com/rstm-sf/XamlX/tree/refactor/extract_netstandard_tests

rstm-sf commented 3 years ago

tests in VS looks good https://github.com/rstm-sf/XamlX/tree/refactor/extract_netstandard_tests

Also solves the problem of non-building Benchmarks

kekekeks / XamlX

Tests/benchmarks failing #33