[Bug]: Migrating Test Plans throws 'Method CreateTestResults can not be reflected'

vifor2 commented 3 months ago

Version

[X] I confirm that I am using the latest version

Source Version

Azure DevOps Server 2022

Target Version

Azure DevOps Service

Relevant configuration

{
  "Version": "15.0",
  "WorkItemTypeDefinition": {},
  "workaroundForQuerySOAPBugEnabled": false,
  "Endpoints": {
    "InMemoryWorkItemEndpoints": [
      {
        "Name": "Source",
        "EndpointEnrichers": null
      },
      {
        "Name": "Target",
        "EndpointEnrichers": null
      }
    ]
  },
  "ChangeSetMappingFile": null,
  "Source": {
    "$type": "TfsTeamProjectConfig",
    "Collection": "https://tfs.<companyname>.com/DefaultCollection",
    "Project": "<sourceproject>",
    "ReflectedWorkItemIDFieldName": "Custom.ReflectedWorkItemId",
    "AllowCrossProjectLinking": false,
    "AuthenticationMode": "AccessToken",
    "PersonalAccessToken": "<redacted>",
    "LanguageMaps": {
      "AreaPath": "Area",
      "IterationPath": "Iteration"
    }
  },
  "Target": {
    "$type": "TfsTeamProjectConfig",
    "Collection": "https://dev.azure.com/<targetorg>/",
    "Project": "<targetproject>",
    "ReflectedWorkItemIDFieldName": "Custom.ReflectedWorkItemId",
    "AllowCrossProjectLinking": false,
    "AuthenticationMode": "AccessToken",
    "PersonalAccessToken": "<redacted>",
    "LanguageMaps": {
      "AreaPath": "Area",
      "IterationPath": "Iteration"
    }
  },
  "FieldMaps": [],
  "GitRepoMapping": null,
  "LogLevel": "Debug",
  "Processors": [
    {
      "$type": "TestConfigurationsMigrationConfig",
      "Enabled": true
    },
    {
      "$type": "TestPlansAndSuitesMigrationConfig",
      "Enabled": true,
      "OnlyElementsWithTag": "Migration",
      "TestPlanQuery": null,
      "RemoveAllLinks": false,
      "MigrationDelay": 0,
      "RemoveInvalidTestSuiteLinks": false,
      "FilterCompleted": false
    }
  ]
}

Relevant log output

[15:21:48 DBG] [v15.0.4] Plan to copy 22 Configurations
[15:21:48 DBG] [v15.0.4] Default configuration created @ 2020-10-06 9:36:38 AM - Copy Configuration
[15:21:48 FTL] [v15.0.4] Error while running TestConfigurationsMigrationContext
Microsoft.TeamFoundation.TestManagement.Client.TestManagementServerException: Server was unable to process request. ---> Method TestManagementWebService.CreateTestResults can not be reflected. ---> There was an error reflecting 'results'. ---> There was an error reflecting type 'Microsoft.TeamFoundation.TestManagement.Server.TestCaseResult'. ---> There was an error reflecting property 'AnalysisFailureGroups'. ---> There was an error reflecting type 'Microsoft.TeamFoundation.TestManagement.Server.OneMRXTestAnalysisFailureGroup'. ---> Cannot serialize member Microsoft.TeamFoundation.TestManagement.Server.OneMRXTestAnalysisFailureGroup.Dimensions of type System.Collections.Generic.Dictionary`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], because it implements IDictionary. ---> System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> Method TestManagementWebService.CreateTestResults can not be reflected. ---> There was an error reflecting 'results'. ---> There was an error reflecting type 'Microsoft.TeamFoundation.TestManagement.Server.TestCaseResult'. ---> There was an error reflecting property 'AnalysisFailureGroups'. ---> There was an error reflecting type 'Microsoft.TeamFoundation.TestManagement.Server.OneMRXTestAnalysisFailureGroup'. ---> Cannot serialize member Microsoft.TeamFoundation.TestManagement.Server.OneMRXTestAnalysisFailureGroup.Dimensions of type System.Collections.Generic.Dictionary`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], because it implements IDictionary.
   --- End of inner exception stack trace ---
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.HandleReply(TfsClientOperation operation, TfsMessage message, Object[]& outputs)
   at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.Invoke(TfsClientOperation operation, Object[] parameters, TimeSpan timeout, Object[]& outputs)
   at Microsoft.TeamFoundation.TestManagement.Client.TestManagementWebService.QueryTestConfigurations(ResultsStoreQuery query)
   at Microsoft.TeamFoundation.TestManagement.Client.TestConfigurationHelper.Query(String queryText)
   at VstsSyncMigrator.Engine.TestConfigurationsMigrationContext.GetCon(ITestConfigurationHelper tch, String configToFind) in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\TestConfigurationsMigrationContext.cs:line 81
   at VstsSyncMigrator.Engine.TestConfigurationsMigrationContext.InternalExecute() in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\TestConfigurationsMigrationContext.cs:line 45
   at MigrationTools._EngineV1.Processors.MigrationProcessorBase.Execute() in D:\a\1\s\src\MigrationTools\_EngineV1\Processors\MigrationProcessorBase.cs:line 50
[15:21:48 ERR] [v15.0.4] TestConfigurationsMigrationContext The Processor MigrationEngine entered the failed state...stopping run

What happened?

I expected my test plans with the tag "Migration" to be migrated, but instead I received the error mentioned. It sounds to me like an error with the tool, but I have not seen issues already opened on this subject for this repository. I've done multiple migrations in the past using this tool over the years, but this time the issue does not seem to be related to my configuration file or the state of the work items to migrate.

Debug in Visual Studio

[X] Visual Studio Debug

vifor2 commented 3 months ago

Update: I can reproduce the same issue on a different machine and inside a Docker container. Reverting to an older version of the tool (I tried 14.3.0) did not help either.

Right now it fails at the TestConfigurationsMigrationConfig processor, but if I disable that processor it then fails during the TestPlansAndSuitesMigrationConfig processor.

Simplifying my configuration file as much as possible and only running the latter processor in Visual Studio debug mode I can see that the source's configuration is properly fetched, but it fails when trying to get the target's configuration:

From there on even simply stepping into (F11) returns this error:

NOTE: I cleaned everything test-related on my target team project beforehand: deleted all of the test runs, cases, suites, plans, variables and configuration. I am authenticating through a full access Personal Access Token created with a user that is both collection administrator and team project administrator and that has a "Basic + Test Plans" license assigned.

MrHinsh commented 2 months ago

Do you have .NET 4.8 installed? What's the os of the host that you are running it on?

vifor2 commented 2 months ago

Hello, sorry I missed the notification.

Yes, I have both the Microsoft .NET Framework 4.8 SDK and Targeting Pack installed. I am running the tool on Windows 11 Enterprise.

MrHinsh commented 1 month ago

This error looks to be coming from the server side...

vifor2 commented 1 month ago

Indeed it seems like it.

Since others did not report this issue and I am over it I think we can close this thread. Here's what I did as a workaround if it can help someone else, I know this is not a great solution, but it did the job for my context:

On the target organization manually create a copy of each configuration needed by the test plans (configurations are found in the navigation menu on the left underneath “Test Plans”)
Download the tool’s source code: git clone https://github.com/nkdAgility/azure-devops-migration-tools.git
Open the following file in Visual Studio: src\VstsSyncMigrator.Core\Execution\MigrationContext\TestPlansAndSuitesMigrationContext.cs
Replace this line: _targetTestConfigs = _targetTestStore.Project.TestConfigurations.Query("Select * From TestConfiguration"); by this one: _targetTestConfigs = _sourceTestConfigs; This will allow us to bypass the function call that makes the application crash. However, this gives us incorrect IDs for our configurations, and we need to adjust them in the next step.

At the start of the SaveNewTestSuiteToPlan() method add the following:

var defaultTc = new IdAndName(1, "Default configuration");
var secondTc = new IdAndName(2, "Second configuration");
var thirdTc = new IdAndName(3, "Third configuration");
if (newTestSuite.DefaultConfigurations != null) {
for (int i = 0; i < newTestSuite.DefaultConfigurations.Count; i++)
{
    switch (newTestSuite.DefaultConfigurations[i].Name)
    {
        case "Second configuration":
            newTestSuite.DefaultConfigurations[i] = secondTc;
            break;
        case "Third configuration":
            newTestSuite.DefaultConfigurations[i] = thirdTc;
            break;
        default:
            newTestSuite.DefaultConfigurations[i] = defaultTc;
            break;
    }
}
}

Replacing the values with what matches your configuration. The IDs in this example (1, 2, 3) correspond to the IDs of the test configurations created on the target organization.

Copy your configuration settings into the configuration.json file of the cloned repository
Build and execute the solution

nkdAgility / azure-devops-migration-tools