kendrahavens commented 6 years ago

Visual Studio 2019 Test-related API removal

In previous releases of Visual Studio there were several test-related APIs marked public, but were never documented or intended for external use. With Visual Studio 2019, we plan to deprecate and/or remove the APIs listed in the tables below. We know this could affect extension owners so we are communicating this very early and before the first preview release of Visual Studio 2019.

Only public APIs under Microsoft.VisualStudio.TestWindow.Interfaces.dll will be available to extend.

If this impacts you and your customers please comment so we can better understand your scenario.

Microsoft.VisualStudio.TestPlatform

API Name	Remarks	Recommendation
Common.Internal.ITestDiscovererCapabilities
Common.RockSteady.TestPluginsFramework.TestPluginCache
Common.RunSettings
Common.TestPluginsFramework.LazyExtension`2
Core.DiscoveryContext
Core.DiscoveryManager
Core.IFrameworkHandleInternal
Core.ITestCaseLog
Core.ITestRunLog
Core.TestExecutionContext
Core.TestExecutionManager
Core.TestRunStats
ObjectModel.Client.ITestExecutorLauncher	Not supported, as it is not required with the new design and new API surface for ITestPlatform.	ITestHostLauncher to be considered.
ObjectModel.Client.ITestPlatform	Re-designed. Slight changes in the API surface to accommodate the new design, removal of redundancies.	Recommendation for clients is to use the TranslationLayer APIs to interact with Test platform v2.
ObjectModel.TestCaseProperties	Supported in TPv2
Utilities.AppContainerUtilities	Not supported in TPv2 : Helper utilities for running tests for Windows Store apps	Do not use.
Utilities.ClientUtilities
Utilities.FakesUtilities	Not supported.
Utilities.InferSettingsHelper	Not supported.
Utilities.IPackageDebugSettings
Utilities.MSTestSettingsUtilities	Not supported.
Utilities.PackageDebugSettings
Utilities.PhoneAppUtilitiesImplementation
Utilities.UtilitiesException	Redundant exception, mainly used in AppContainer context.	Do not use.

Microsoft.VisualStudio.TestWindow (under Microsoft.VisualStudio.TestWindow.Interfaces.dll)

API Name	Remarks
Extensibility.ICustomSettingsImporter (Deprecated)	With runsettings having an extensible format to specify extension specific settings, there isn't a requirement to support multiple custom settings files. Extensions can always provide additional settings in IDE work flows through the IRunSettingsService.
Extensibility.ICommandContainer (Deprecated)
Extensibility.IDeployService (Deprecated)	This API is no longer supported for deployment of UWP based apps.
Extensibility.IOpenTarget (Deprecated)	Please use VS shell supported APIs to open files or hyperlinks in your extensions.
Extensibility.IProperty (Deprecated)
Extensibility.IResult (Deprecated)
Extensibility.IRunCommands (Deprecated)
Extensibility.ISelectionCommands (Deprecated)
Extensibility.ITest (Deprecated)
Extensibility.ITestsService (Deprecated)
Extensibility.IUICommandContainer (Deprecated)
Extensibility.RunCommand (Deprecated)
Extensibility.SelectionCommand (Deprecated)
Extensibility.ValidateArgProperty (Deprecated)
Extensibility.IIncrementalUnitTestChange
Extensibility.ISourceBasedTestDiscoverer
Extensibility.ISourceInformationProvider
Extensibility.ITestContainer2
Extensibility.ITestContainer3
Extensibility.ITestDataDelta
Extensibility.ITestDiscoveryListener
Extensibility.ITestExplorerService
Extensibility.ITestExplorerServiceContainerDiscoverer
Extensibility.ITestSourceLocationLookupService
Extensibility.SourceLocation
Extensibility.TestDataChangeReason
Extensibility.TestMethodSourceInformation
Telemetry. ITestWindowTelemetry

Microsoft.VisualStudio.TestWindow

The following APIs not under Microsoft.VisualStudio.TestWindow.Interfaces.dll are amongst the list of APIs removed.

API Name	Remarks
Controller.DiscoverAllOrRunOnInitializeOperation
Controller.DiscoveryConfiguration
Controller.IDebugLauncher
Controller.IOperationData
Controller.IRequestFactory
Controller.ITestContainerProvider
Controller.IUserRunSettings
Controller.Operation
Controller.OperationData
Controller.RequestConfiguration
Controller.RequestConfigurationFactory
Controller.SafeDispatcher
Controller.TestContainerConfiguration
Controller.TestPlatformProvider
Controller.TestRunConfiguration
Data.ChangedContainer
Data.TestsService
Extensibility.ILoggerExtensions
VsAdapters.AbstractDeviceAppxPackageUtilities
VsAdapters.VsPhoneAppxPackageUtilities
VsHost.DebugLauncher
VsHost.IToolWindow
VsHost.RemoteDebugLauncher
VsHost.TestWindowPackage

luis-cortina commented 5 years ago

@kendrahavens @cltshivash Where are this apis documented? Can I used the inside vsix to work with test objects? (Discover, start tests, get results?)

kendrahavens commented 5 years ago

@luis-cortina We plan to publish a nuget package to make referencing Test Window APIs easy for extension developers. Until this package is published (it may be a month or two looking at our current backlog) you can manually add a reference to this assembly in your Visual Studio install location: <VSInstallDir>/Common7/IDE/CommonExtensions/Microsoft/TestWindow/Microsoft.VisualStudio.TestWindow.Interfaces.dll

luis-cortina commented 5 years ago

@kendrahavens Thanks.

I'm currently using ITestsService, which looks like it will be deprecated in vs2019, to get discovered tests and run them inside an extension. If it's in fact deprecated, which interface should we use to do this moving forward?

Is there any documentation on this APIs? i.e I need to start the test discovery process inside an extension and I don't really know where to look. Any help will be appreciated.

Luis

AbhitejJohn commented 5 years ago

@luis-cortina : Thanks for letting us know. ITestsService is indeed something we are planning to replace with a more performant alternative soon. Just so I understand your scenario fully, you need an API to trigger a test run on a filtered set of tests? Can you elaborate please?

If you have an implementation of ITestContainerDiscoverer triggering the TestContainersUpdated event should get a discovery going. This blog should help.

luis-cortina commented 5 years ago

@AbhitejJohn how soon are you thinking? I'm thinking about upgrading my extension, but not sure if ITestService will be available at all for vs2019? Te blog post helps, thanks for that.

AbhitejJohn commented 5 years ago

@luis-cortina : We are actively working on getting this through - Hoping to have it in an update soon. Could you tell us what functionality in ITestsService is interesting to your extension?

luis-cortina commented 5 years ago

@AbhitejJohn here is what I'm using:

IComponentModel to get all the test window related services
ITestsService to get all discovered tests (GetTestsAsEnumerableAsync) and to run a particular test (RunTestsAsync)
ITestExplorerServiceContainerDiscoverer, I call the ClearContainers method which seems to start test discovery

AbhitejJohn commented 5 years ago

@luis-cortina : Thanks. So the usage of ITestsService is essentially to run a specific test - would supplying the test to run through a filter of Full Qualified Name or Outcome be an easier alternative? The GetTestsAsEnumerableAsync API cycles through all tests in the store which is pretty slow.

As for ITestExplorerServiceContainerDiscoverer this was meant primarily for Live Unit Testing and was accidentally public in builds of VS 2017. This wouldn't be available anymore unfortunately per the list above. Just to make sure I'm understanding this right - your extension triggers discovery to ensure everything is up to date, gets a set of tests and runs a specific one? INewTestService.RunTests with a filter, I described above, could do all of this for you - perform a discovery, filters the test you need to run and then runs it.

luis-cortina commented 5 years ago

@AbhitejJohn Yes, that would work. It would be nice that the filter accepts a predicate of ITest (or whatever the new class is to represent a test), this should also allow someway to run multiple tests as well.

The only reason I'm using ITestExplorerServiceContainerDiscoverer is to kick off the discovery at the beggining of the vsix load, and it won't be needed anymore.

luis-cortina commented 5 years ago

Hi @kendrahavens @AbhitejJohn, VS2019 is out and I'm looking to upgrade my extension to it. Are any of the deprecated apis available or should i wait for the new ones to come out?

AbhitejJohn commented 5 years ago

@luis-cortina : The deprecated ITestsService is still available, yes. The new API I've detailed above are part of a larger change that's being stabilized. We will update this thread as soon as that is available in a preview. APIs, other than the ones marked deprecated above, are removed. So ITestExplorerServiceContainerDiscoverer wouldn't be available anymore. A solution build triggers discovery as well if that's an option.

luis-cortina commented 5 years ago

@AbhitejJohn Building the solution is not currently triggering discovery. Only opening the Test Explorer window does. Is there a service that can help me do this better?

AbhitejJohn commented 5 years ago

@luis-cortina : Opening the Test Explorer ensures that a few initialization hooks are in place to trigger discovery on build - we wouldn't want to perform any test related work unless a user expresses interest in testing - one of which is opening the Test Explorer.

luis-cortina commented 5 years ago

@AbhitejJohn Yeah, is there any way I can do the same thing without asking the user to open the Test Explorer Window??

luis-cortina commented 4 years ago

@kendrahavens @AbhitejJohn @cltshivash Any updates on this? I'm looking for some sort of interface to kickstart the test engine without explicitly opening the Test Explorer. Also, any documentation on the Apis that we can use to interact with the testing engine would be great.

AbhitejJohn commented 4 years ago

@luis-cortina : I haven't been great with github notifications off late. I've logged an issue to track exposing this new API here. We've currently scheduled to preview this for Q2 of 2020 given our other set of priorities. We should be able to review docs as well as part of that effort. Thanks a lot for your feedback and for following up on this.

suehshtri commented 4 years ago

There is, in general, a need to be able to extend test discovery to different test sources, as well as to have those test results recorded in TFS.

AbhitejJohn commented 4 years ago

@suehshtri : The test platform adapter extensiblity helps with that - https://github.com/microsoft/vstest-docs/blob/master/RFCs/0004-Adapter-Extensibility.md. Is there a case where this does not work for you?

tonyhallett commented 3 years ago

Fine Code Coverage extension for 2019 was hooking into tests with ITestContainerDiscoverer. Visual Studio 2022 is not working in this manner. Is there a method of obtaining test projects being run and runsettings in Vs 2022 ?

AbhitejJohn commented 3 years ago

@tonyhallett : Oh we haven't intentionally changed anything for Test Explorer extensibility in VS 2022. Which interface or API are you using that doesn't work the same? On a related note, for code coverage scenarios I'd imagine IRunSettingsService would work better. Here's an example usage of that API: https://github.com/pimterry/chutzpah/blob/a521844899d85c308307e857cfb07df47421e994/VS2012.TestAdapter/ChutzpahMapperSettingsService.cs#L40. It provides a list of test containers that you can condition runsettings on as well.

tonyhallett commented 3 years ago

@AbhitejJohn Thank you for your response and the link. All is ok now. The ITestContainerDiscoverer was not being loaded as it was missing

[Export(TestContainerDiscoverer)]

I thought that this would be sufficient

[Export(ITestContainerDiscoverer)]

AbhitejJohn commented 3 years ago

The Test Window logic does not require the concrete implementation to be exported. Perhaps there are changes in your extension where an import of TestContainerDiscoverer is required? MEF logs are usually found under %localappdata%\Microsoft\VisualStudio\\ComponentModelCache

tonyhallett commented 3 years ago

@AbhitejJohn yes that didn't make sense. I will check further. Thanks for log path.

Evangelink commented 2 years ago

Make the list of APIs listed above for Test Platform side internal instead of public. This should probably be done for 17.5.

nohwnd commented 4 months ago

This is breaking change to the public api and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform

microsoft / vstest