search

matkoch / TestFx

Next Level Testing.
Apache License 2.0
34 stars 4 forks source link

Add html output option #7

Open Uli-Armbruster opened 8 years ago

Uli-Armbruster commented 8 years ago

I would be nice if there were support for html outputs like in MSpec with "--html {report}". According to https://github.com/matkoch/TestFx/issues/6, I could support this in a separate version.

matkoch commented 8 years ago

The main question is about the design of that report. I already have some working example flying around, which uses razor files + bootstrap to render.

Uli-Armbruster commented 8 years ago

I attached a report that is acutally generated by the current MSpec runner (saved from html as pdf) Machine.pdf

matkoch commented 8 years ago

@Uli-Armbruster since I've seen that you use german language for the report, do you also use the custom delegates in code?

Uli-Armbruster commented 8 years ago

No, just because I didn't know about that. But that is really a great feature that I would like to make use of.

matkoch commented 8 years ago

Better not. If really necessary, I would suggest you to use an alias... I'm not likely to support that, unless there are many votes for it.

Uli-Armbruster commented 8 years ago

No problem. Actually I just need the options for filtering and html output

matkoch commented 8 years ago

@Uli-Armbruster you're welcome to take a look at the draft. Not yet published to nuget.

Invoke with TextFx.exe --assemblies Tests.dll --pause --htmlReport default --output report

I think the best would be to have the report looking the same as resharper test runner output, what do you think?

agross commented 8 years ago

Today we did the following:

This yielded the following output:

Unhandled Exception: RazorEngine.Templating.TemplateCompilationException: Errors while compiling a Template.
Please try the following to solve the situation:
  * If the problem is about missing/invalid references or multiple defines either try to load
    the missing references manually (in the compiling appdomain!) or
    Specify your references manually by providing your own IReferenceResolver implementation.
    See https://antaris.github.io/RazorEngine/ReferenceResolver.html for details.
    Currently all references have to be available as files!
  * If you get 'class' does not contain a definition for 'member':
        try another modelType (for example 'null' to make the model dynamic).
        NOTE: You CANNOT use typeof(dynamic) to make the model dynamic!
    Or try to use static instead of anonymous/dynamic types.
More details about the error:
 - error: (15, 18) The type or namespace name 'Evaluation' does not exist in the namespace 'TestFx' (are you missing an assembly reference?)
         - error: (16, 18) The type or namespace name 'Extensibility' does not exist in the namespace 'TestFx' (are you missing an assembly reference?)
         - error: (19, 65) The type or namespace name 'RazorEngine' could not be found (are you missing a using directive or an assembly reference?)
         - error: (19, 108) The type or namespace name 'Evaluation' does not exist in the namespace 'TestFx' (are you missing an assembly reference?)
Temporary files of the compilation can be found in (please delete the folder): C:\Cygwin\tmp\RazorEngine_dg4yt1qs.i2d
The template we tried to compile is:
------------- START -----------
@model TestFx.Evaluation.Results.IRunResult
@using TestFx.Evaluation.Results
@using TestFx.Extensibility.Providers

@{
  var suiteResults = Model.SuiteResults.SelectMany(x => x.SuiteResults).ToList ();

  var passed = suiteResults.Count (x => x.State == State.Passed);
  var failed = suiteResults.Count (x => x.State == State.Failed);
  var inconclusive = suiteResults.Count (x => x.State == State.Inconclusive);
  var ignored = suiteResults.Count (x => x.State == State.Ignored);

  var panelId = 0;
}

<!DOCTYPE html>
<html lang="en">
<head>
...

The generated source code is:
------------- START -----------
//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by a tool.
//     Runtime Version:4.0.30319.42000
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

namespace CompiledRazorTemplates.Dynamic {
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using TestFx.Evaluation.Results;
    using TestFx.Extensibility.Providers;
...

List of loaded Assemblies:
C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscorlib.dll
        Loaded Assembly: D:\Users\agross\Projekte\GROSSWEBER\Kunden\heco\comwork\erp\packages\TestFx.Core\tools\TestFx.exe
        Loaded Assembly: C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll
        Loaded Assembly: C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll
        Loaded Assembly: C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.IO.Compression\v4.0_4.0.0.0__b77a5c561934e089\System.IO.Compression.dll
        Loaded Assembly: C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.IO.Compression.FileSystem\v4.0_4.0.0.0__b77a5c561934e089\System.IO.Compression.FileSystem.dll
        Loaded Assembly: D:\Users\agross\Projekte\GROSSWEBER\Kunden\heco\comwork\erp\build\Client\comWORK.BusinessLogic.dll
        Loaded Assembly: C:\Cygwin\tmp\Costura\7E4A85F7E36AE498EC92C0319E54D454\autofac.dll
        Loaded Assembly: C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Configuration\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll
        Loaded Assembly: C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Xml\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.dll
        Loaded Assembly: C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.CSharp\v4.0_4.0.0.0__b03f5f7f11d50a3a\Microsoft.CSharp.dll
   at RazorEngine.Compilation.DirectCompilerServiceBase.CompileTypeImpl(TypeContext context)
   at RazorEngine.Compilation.DirectCompilerServiceBase.CompileType_Windows(TypeContext context)
   at RazorEngine.Compilation.DirectCompilerServiceBase.CompileType(TypeContext context)
   at RazorEngine.Templating.RazorEngineCore.CreateTemplateType(ITemplateSource razorTemplate, Type modelType)
   at RazorEngine.Templating.RazorEngineCore.Compile(ITemplateKey key, Type modelType)
   at RazorEngine.Templating.RazorEngineService.CompileAndCacheInternal(ITemplateKey key, Type modelType)
   at RazorEngine.Templating.TemplateService.CreateTemplate(String razorTemplate, Type staticType, Object model)
   at RazorEngine.Templating.TemplateService.Parse(String razorTemplate, Object model, DynamicViewBag viewBag, String cacheName)
   at RazorEngine.Razor.Parse[T](String razorTemplate, T model)
   at TestFx.Console.HtmlReport.HtmlReportRunListener.OnRunFinished(IRunResult result)
...
matkoch commented 8 years ago

@agross thanks for your report. Both issues should be fixed by now.

I'm not a complete fan of that embedded razor template, and will try to get rid of it in favor of referencing just a XML, then parsing it using JavaScript. Maybe after weekend :grinning:

agross commented 8 years ago 
D:\Users\agross\Downloads\TestFx\shared\build
$ build.cmd

D:\Users\agross\Downloads\TestFx\shared\build
$ SET bla=blu

D:\Users\agross\Downloads\TestFx\shared\build
$ powershell.exe /file Test.ps1
The argument 'Test.ps1' to the -File parameter does not exist. Provide the path to an existing '.ps1' file as an argument to the -File parameter.
agross commented 8 years ago

Why not put it in the root? New users to TestFx will have a hard time guessing that build scripts exist in shared\build. I'd say that location is rather hard to find.

agross commented 8 years ago 
D:\Users\agross\Downloads\TestFx\shared\build
$ powershell .\FullBuild.ps1

Name                           Value
----                           -----
?                              True
args                           {}
AssemblyInfoFile               src\AssemblyInfoShared.cs
Configuration                  Release
ConfirmPreference              None
ConsoleFileName
CoverageConfig                 D:\Users\agross\Downloads\TestFx\shared\coverage.xml
CoverageFile                   src\coverage.xml
DotCover                       C:\Users\agross\AppData\Local\JetBrains\Installations\dotCover05\dotCover.exe
DotCoverDir                    C:\Users\agross\AppData\Local\JetBrains\Installations\dotCover05
ErrorActionPreference          Stop
ExecutionContext               System.Management.Automation.EngineIntrinsics
false                          False
GitInfo                        GetGitInfo.VersionInformation
HOME                           C:\Users\agross
Host                           System.Management.Automation.Internal.Host.InternalHost
input                          {}
LocalDotCoverDir               C:\Users\agross\AppData\Local\JetBrains\Installations\dotCover05
LocalMsBuildDir                C:\Program Files (x86)\MSBuild\12.0\Bin
LocalNuGetDir                  D:\Users\agross\Downloads\TestFx\..
MaximumAliasCount              4096
MaximumDriveCount              4096
MaximumErrorCount              256
MaximumFunctionCount           4096
MaximumVariableCount           4096
MsBuild                        C:\Program Files (x86)\MSBuild\12.0\Bin\MSBuild.exe
MsBuildDir                     C:\Program Files (x86)\MSBuild\12.0\Bin
MyInvocation                   System.Management.Automation.InvocationInfo
NuGet                          D:\Users\agross\Downloads\NuGet.exe
NuGetDir                       D:\Users\agross\Downloads\TestFx\..
null
NuSpecDir                      D:\Users\agross\Downloads\TestFx\nuspec
NuSpecFiles                    {D:\Users\agross\Downloads\TestFx\nuspec\TestFx.Core.nuspec, D:\Users\agross\Downloads\TestFx\nuspec\TestFx.ReSharper.nuspec, D:\Users\agross\Downloads\TestFx\nuspec...
OutputDir                      D:\Users\agross\Downloads\TestFx\output
PackagesDir                    D:\Users\agross\Downloads\TestFx\packages
PID                            7048
PSBoundParameters              {}
PSCmdlet                       System.Management.Automation.PSScriptCmdlet
PSCommandPath                  D:\Users\agross\Downloads\TestFx\shared\build\_Variables.ps1
PSCulture                      de-DE
PSHOME                         C:\Windows\SysWOW64\WindowsPowerShell\v1.0
PSScriptRoot                   D:\Users\agross\Downloads\TestFx\shared\build
PSUICulture                    en-US
PSVersionTable                 {PSVersion, PSCompatibleVersions, BuildVersion, CLRVersion...}
SemVer                         0.6.1-Pre00035
ShellId                        Microsoft.PowerShell
SkipTests                      False
SolutionDir                    D:\Users\agross\Downloads\TestFx
SolutionFile                   D:\Users\agross\Downloads\TestFx\TestFx.sln
SourceDir                      D:\Users\agross\Downloads\TestFx\src
Targets                        Rebuild
TestAssemblies                 {D:\Users\agross\Downloads\TestFx\src\TestFx.SpecK.Tests\bin\Release\TestFx.SpecK.Tests.dll, D:\Users\agross\Downloads\TestFx\src\TestFx.MSpec.Tests\bin\Release\Test...
true                           True

Clean

##teamcity[progressMessage 'Clean']]
##teamcity[buildNumber '0.6.1-Pre00035']

Build

##teamcity[progressMessage 'Build']]
##teamcity[buildNumber '0.6.1-Pre00035']
D:\Users\agross\Downloads\TestFx\shared\build\Compile.ps1 : The term 'D:\Users\agross\Downloads\NuGet.exe' is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At D:\Users\agross\Downloads\TestFx\shared\build\FullBuild.ps1:17 char:1
+ & $PSScriptRoot\Compile.ps1 -NuGetDir $NuGetDir -MsBuildDir $MsBuildD ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (D:\Users\agross\Downloads\NuGet.exe:String) [Compile.ps1], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException,Compile.ps1

Why is NuGet.exe expected to be outside of the project directory?

agross commented 8 years ago

With the latest change we now get build errors:

C:\WINDOWS\Microsoft.NET\Framework64\v4.0.30319\Microsoft.WinFx.targets(268,9): error MC1000: Unknown build error, 'Cannot resolve dependency to assembly 'TestFx.Core, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' because it has not been preloaded. When using the ReflectionOnly APIs, dependent assemblies must be pre-loaded or loaded on demand through the ReflectionOnlyAssemblyResolve event.'  [D:\Users\agross\Projekte\GROSSWEBER\Kunden\heco\comwork\erp\source\net_client\comWORK.Modules.Common.UI\comWORK.Modules.Common.UI.csproj]

comWORK.Modules.Common.UI.csproj doesn't reference TestFx directly but another project that does.

matkoch commented 8 years ago

@agross there is a Build.cmd in the root. The other just slipped through.

I will continue to make nuget packages again. Local build is anyways a thing that I want to improve, but with low priority.

matkoch commented 8 years ago

@agross What happens if you reference it directly again?

agross commented 8 years ago

@matkoch https://github.com/matkoch/TestFx/tree/617cea43ec6886160ba743aba28b7a87e2bf5b1f I don't see a build.cmd in the root. Where is it?

matkoch commented 8 years ago

Yes, my fault. Picked the wrong. Fixed.

agross commented 8 years ago

What happens if you reference it directly again?

@matkoch The build starts failing at the next project.

We use side-by-side specs, so many basic projects (for utils, domain model, etc.) use MSpec/TestFx. It would be unfortunate if we would need to add TestFx to basically all projects in the whole solution.

Another question: _TestExtensions and _TestLoaders showed up as untracked. What's their purpose?

matkoch commented 8 years ago

I guess having a reference to TestFx is unavoidable, especially since there are usages in _TestExtensions and _TestLoaders. Why is that a problem specifically? And how did you manage to add the package without the dependency in the first place? :smile_cat:

These two folders are kind of a configuration to the test assembly. It tells the execution engine, what TestLoader should be used (e.g., MSpec or SpecK), and which extensions affect the individual tests.

agross commented 8 years ago

Having a reference to TestFx for the assemblies that acutually contain tests is totally okay :smile:

It seems the XAML compiler hiccups when TestFx is referenced by an assembly containing view models (MC1000 == Markup Compiler). What's interesting is that the error only appeared after your latest changes to Costura.

I'm by no means a WPF expert. We need to investigate, @Uli-Armbruster.

These two folders are kind of a configuration to the test assembly.

Interesting. I thought TestFx would employ some kind of auto-detection, e.g. by checking references to detect what frameworks are used.

matkoch commented 8 years ago

No auto-detection. That was intentional, since dropping in other assemblies could lead to different behavior. This way it's more explicit.

What do you mean with hiccups? Do you have TestFx.exe in a dedicated folder, and execute it on an assembly outside that folder, or not? The application does extract some references on startup.