AzureDevOpsPipelineProcessorOptions throws a serverPath null API exception for the build pipelines

IstuntmanI commented 3 years ago

Describe your issue:

When running the pipelines processor with the default settings, an error similar to Value cannot be null. Parameter name: definition.Repository.Mappings.Mapping.ServerPath appears. I just started using this tool and it seems to me that this processor is in some kind of alpha stage? Hopefully it isn't deprecated or anything.

Source Details

Source Version: Azure DevOps Services
Source Language: Eng

Target Details

I'm using this tool to move a project from an organization to another one, so:

Target Version: Azure DevOps Services
Target Language: Eng

I have completed the following:

Simply running the tool with this default processor:

{
    "$type": "AzureDevOpsPipelineProcessorOptions",
    "Enabled": true,
    "MigrateBuildPipelines": true,
    "MigrateReleasePipelines": true,
    "MigrateTaskGroups": true,
    "MigrateVariableGroups": true,
    "MigrateServiceConnections": true,
    "BuildPipelines": null,
    "ReleasePipelines": null,
    "ProcessorEnrichers": null,
    "SourceName": "Source",
    "TargetName": "Target"
}

Attempted fix

I noticed that in the https://dev.azure.com/{org}/{proj}/_apis/build/definitions/{id}/ URL we have this information:

"repository":{
  "properties":{
     "cleanOptions":"0",
     "tfvcMapping":"{\"mappings\":[{\"serverPath\":\"$/{proj}/{branch}\",\"mappingType\":\"map\",\"localPath\":\"\\\\test\"}]}",
     "labelSources":"0",
     "labelSourcesFormat":"$(build.buildNumber)"
  },
  "id":"$/",
  "type":"TfsVersionControl",
  "name":"{proj}",
  "url":"https://dev.azure.com/{org}/",
  "defaultBranch":"$/{proj}/{branch}",
  "rootFolder":"$/{proj}",
  "clean":"false",
  "checkoutSubmodules":false
},

Then I took a look in the BuildDefinitions.cs file and I noticed that the accessor for the tfvcMapping string field is missing from the Properties class (which is used in the Repository class, which is used in the BuildDefinition class), so I created a derived PropertiesBuildDefinition class (in case that its current form is still needed for some reason):

public partial class PropertiesBuildDefinition : Properties
{
    public string TfvcMapping { get; set; }
}

public partial class Repository
{
    public PropertiesBuildDefinition Properties { get; set; }
    // ...
}
}

This will simply write the server and local paths just like they were before, in the old project. This probably means that it may throw another exception if the project to migrate to has another name. I'm not creating a pull request for this reason and also because I'm not sure if this is fine for other Azure DevOps/TFS source versions.

github-actions[bot] commented 3 years ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days

tomfrenzel commented 2 years ago

The project name would have to be mapped to the target one. Shouldn't be a Big Deal. But, I'm not sure whether Azure DevOps or TFS versions which don't use TfvcMapping will throw an exception when trying to HTTP POST a Pipeline with that Filed (even if it's empty). I'll have to validate that.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days

tomfrenzel commented 2 years ago

I'll look into that the next weeks

backasch commented 3 days ago

Hi @MrHinsh, @tomfrenzel

any updates on this topic, I received the same error also with version 16.0.3. We are using Azure DevOps Server 2022.

Used configuration:

{
    "MigrationTools": {
      "Version": "16.0",
      "Endpoints": {
      "Source_AdoEndpoint": {
        "EndpointType": "AzureDevOpsEndpoint",
        "Organisation": "https://tfsurl.com/tfs/SourceCollection/",
        "Project": "PROJECT",
        "AuthenticationMode": "AccessToken",
        "AccessToken": "MyToken",
        "ReflectedWorkItemIdField": null,
        "EndpointEnrichers": null
      },
      "Target_AdoEndpoint": {
        "EndpointType": "AzureDevOpsEndpoint",
        "Organisation": "https://tfsurl.com/tfs/TargetCollection/",
        "Project": "PROJECT",
        "AuthenticationMode": "AccessToken",
        "AccessToken": "MyToken",
        "ReflectedWorkItemIdField": null,
        "EndpointEnrichers": null
      }
    },
      "Processors": [
        {
          "ProcessorType": "AzureDevOpsPipelineProcessor",
          "BuildPipelines": "",
          "Enabled": "True",
          "MigrateBuildPipelines": "True",
          "MigrateReleasePipelines": "True",
          "MigrateServiceConnections": "True",
          "MigrateTaskGroups": "True",
          "MigrateVariableGroups": "True",
          "ReleasePipelines": "",
          "RepositoryNameMaps":  {
            "PROJECT": "PROJECT"
          },
          "SourceName": "Source_AdoEndpoint",
          "TargetName": "Target_AdoEndpoint"
        }
      ]
    }
  }

Error Message:

POST https://tfsurl.com/tfs/TargetCollection//PROJECT/_apis/build/definitions/
{"$id":"1","innerException":null,"message":"Value cannot be null.\r\nParameter name: definition.Repository.Mappings.Mapping.ServerPath","typeName":"System.ArgumentNullException, mscorlib","typeKey":"ArgumentNullException","errorCode":0,"eventId":0}

I need to add the RepositoryNameMaps, otherwise the migration tool faults with an NRE. Both projects have the same name and also the repository names are equal.

[16.0.3] Error while running AzureDevOpsPipelineProcessor
System.NullReferenceException: Object reference not set to an instance of an object.
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.MapRepositoriesInBuidDefinition(IEnumerable`1 sourceRepositories, IEnumerable`1 targetRepositories, BuildDefinition definitionToBeMigrated) in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 386
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.<CreateBuildPipelinesAsync>d__17.MoveNext() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 304
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.<MigratePipelinesAsync>d__9.MoveNext() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 86
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.InternalExecute() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 40
   at MigrationTools.Processors.Infrastructure.Processor.Execute() in D:\a\azure-devops-migration-tools\azure-devops-migration-tools\src\MigrationTools\Processors\Infrastructure\Processor.cs:line 98

Kind Regards Martin

nkdAgility / azure-devops-migration-tools