microsoft / testfx

MSTest framework and adapter
MIT License
716 stars 253 forks source link

Questions about settings parameters... #354

Closed Kressilac closed 6 years ago

Kressilac commented 6 years ago

In trying to use MSText V2 for our local project we're seeing two problems.

1) There doesn't seem to be a way to control the discovery in VSTS such that it lines up with the values in the .runsettings file. In our .runsettings we indicate x64 and Framework45. Our project builds Any CPU, .Net 4.6.2. Test discovery shows x86 and .Net 4.6.2 and complains that these values are not in the .runsettings file. If you try to update the .runsettings to anything other than Framework45 you get an error indicating an invalid value in VSTS. For x86, our servers are 64bit build servers, the code is flagged Any CPU and no prefer32bit attribute is set on a project.

2) Since moving to MSTest V2, we've noticed that a small subset of our unit tests are running intellitrace files and placing them into C:\Windows\Temp. I cannot seem to find the setting where I can turn this off or the .csproj setting where you set intellitrace gathering to off when running unit tests. Our build server's hard drive gets filled up with these 30MB files fairly rapidly.

Help on where to look would be appreciated.

jayaranigarg commented 6 years ago

@Kressilac : Thank you for reaching out to us. Please find our responses below:

  1. You can pass runsettings in VSTS using "Other console options". If you just want to trigger discovery, passing '/listtests' in "Other console options" might help as well. MSTestV2 is supported for .net framework 4.5 and above, so specifying Framework35 or Framework40 will not help. What value were you trying when you faced an invalid value issue?

  2. What are the extensions of intellitrace files that are getting generated? Could you share some of those files with us?

Kressilac commented 6 years ago

I was wanting to use Framework462 and x64, soon Framework471 and x64 but anything other than Framework45 throws an error while Framework45 generates the following warnings. Setting x64 seemingly has no impact on the tests being compiled for x86 versus x64 when the server is 64 bit Windows 2016 and the build configurations are all Any CPU without prefer 32 bit set in any csproj.

buildserverframeworkerror

buildserverversionmismatches

The trace files are in Windows/Temp and are called "testhost_###_Date_Time.iTrace" and are about 20 - 30MB each. Not all of our unit tests produce these files and they seem to correspond with build output that shows The Microsoft Code Coverage Collection tool splash data.

buildservertracefiles

And this output appears in our build output in VSTS.

buildserveroutput

Appreciate the help. Derek

Kressilac commented 6 years ago

Following up on this one to see if there is anything else that you need me to provide? I'd like to understand why these issues are happening.

jayaranigarg commented 6 years ago

@Kressilac : Apologies for the delay.

I was wanting to use Framework462 and x64, soon Framework471 and x64 but anything other than Framework45 throws an error while Framework45 generates the following warnings.

There is no shorthand for framework 4.6.2 or 4.7. You can specify that using full notation ".NETFramework,Version=v4.6.2". You can also skip specifying the framework entirely, it by default will pick the framework against which the dll was built.

Setting x64 seemingly has no impact on the tests being compiled for x86 versus x64 when the server is 64 bit Windows 2016 and the build configurations are all Any CPU without prefer 32 bit set in any csproj.

Unless explicitly required, you should always build your testproject for AnyCPU and should not specify any platform. It will automatically figure out which platform(x86/x64) it is supposed to use. However, if Test dlls are built for x86, then you should specify platform x86 and if test dlls are built for x64, then you should specify platform x64. As a user, you will not observe any difference when tests run for platform x86 or x64, the difference lies in the implementation of how these two cases will be handled by platform internally.

jayaranigarg commented 6 years ago

@wiktork : Some intellitrace files are getting generated for the user here as a part of his test run. How can help here to figure out which settings needs to be changes to turn off the generation of these files?

Kressilac commented 6 years ago

Thanks. That resolved the issues with the framework discrepancies. Appreciate the feedback. The only thing I need to figure out now is why intellitrace files are being generated. It doesn't happen for all of our projects and I have CaptureTraceOutput set to false for the MSTestV2 configuration element but we're still seeing the trace files generated like I stated in the above picture.

wiktork commented 6 years ago

@Kressilac @jayaranigarg If the runsettings file is specified, you have to make sure you remove TraceDebuggerDataCollector from the settings. If it's not specified, it probably means the default runsettings file turns it on. I can't think of any other reason why we would try to make .itrace files.

Kressilac commented 6 years ago

@wiktork image

That's the pertinent runsettings. Seems like it is all CodeCoverage.

jayaranigarg commented 6 years ago

@Kressilac : Can you pass "/diag:logfile.txt" switch when running vstest.console and share that logfile with us? That file should have the final runsettings that are getting used to run your tests. It will help us figure out if indeed TraceDebuggerDataCollector is present/inserted in the runsettings somehow.

Kressilac commented 6 years ago

Sorry for the delayed response. Here is our diagnostics file. Thanks for your help with this. logfile.txt logfile.host.18-02-26_09-12-28_91465_11.txt logfile.host.18-02-26_09-12-27_93047_10.txt logfile.datacollector.18-02-26_09-12-25_40934_6.txt

Included all of the other files from our build.

wiktork commented 6 years ago

@Kressilac Thank you for the log files. We're investigating a fix for this issue and it is also being tracked with https://developercommunity.visualstudio.com/content/problem/195034/cwindowstemp-fills-up-with-itrace-files-after-upgr.html

smadala commented 6 years ago

Fixed https://github.com/Microsoft/vstest/pull/1483