search

nkdAgility / azure-devops-migration-tools

Azure DevOps Migration Tools allow you to migrate Teams, Backlogs, Tasks, Test Cases, and Plans & Suits from one Project to another in Azure DevOps / TFS both within the same Organisation, and between Organisations.
https://nkdagility.com/learn/azure-devops-migration-tools/
MIT License
491 stars 327 forks source link

[Bug]: AzureDevOpsPipelineProcessor System.NullReferenceException #1565

Closed rowanjacobs closed 1 year ago

rowanjacobs commented 1 year ago

Version

Source Version

Azure DevOps Service

Target Version

Azure DevOps Service

Relevant configuration

{
  "TelemetryEnableTrace": true,
  "LogLevel": "Debug",
  "Endpoints": {
    "AzureDevOpsEndpoints": [
      {
        "Name": "Source Project",
        "Organisation": "https://dev.azure.com/org/",
        "AuthenticationMode": "AccessToken",
        "Project": "Source%20Project",
        "AccessToken": "personal-access-token",
        "PersonalAccessToken": "personal-access-token",
        "ReflectedWorkItemIDFieldName": "Custom.ReflectedWorkItemId",
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        }
      },
      {
        "Name": "Target Project",
        "Organisation": "https://dev.azure.com/org/",
        "AuthenticationMode": "AccessToken",
        "Project": "Target%20Project",
        "AccessToken": "personal-access-token",
        "PersonalAccessToken": "personal-access-token",
        "LanguageMaps": {
          "AreaPath": "Area",
          "IterationPath": "Iteration"
        },
        "ReflectedWorkItemIDFieldName": "Custom.ReflectedWorkItemId",
        "Query": {
          "Query": "SELECT [System.Id], [System.Tags] FROM WorkItems WHERE [System.TeamProject] = 'Target Project' AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan') ORDER BY [System.ChangedDate] desc"
        }
      }
    ]
  },
  "ReflectedWorkItemIDFieldName": "Custom.ReflectedWorkItemId",
  "WorkItemTypeDefinition": {
    "Bug": "Bug",
    "User Story": "User Story",
    "Requirement": "Requirement",
    "Task": "Task",
    "Test Case": "Test Case",
    "Chore": "Chore",
    "Shared Steps": "Shared Steps",
    "Shared Parameter": "Shared Parameter",
    "Spike": "Spike",
    "Design": "Design"
  },
  "Processors": [
    {
      "$type": "AzureDevOpsPipelineProcessorOptions",
      "Enabled": true,
      "MigrateBuildPipelines": true,
      "MigrateReleasePipelines": true,
      "MigrateTaskGroups": true,
      "MigrateVariableGroups": true,
      "MigrateServiceConnections": true,
      "BuildPipelines": null,
      "ReleasePipelines": null,
      "RefName": null,
      "SourceName": "Source Project",
      "TargetName": "Target Project"
    }
  ],
  "Version": "13.0",
  "workaroundForQuerySOAPBugEnabled": false
}

Relevant log output

[17:53:18 INF] Application Starting
[17:53:18 INF]                                       &@&
[17:53:18 INF]                                    @@(((((@
[17:53:18 INF]                                   @(((((((((@
[17:53:18 INF]                                 @(((((((((((((&
[17:53:18 INF]                               ##((((((@ @((((((@@
[17:53:18 INF]                              @((((((@     @((((((&
[17:53:18 INF]                             @(((((#        @((((((@
[17:53:18 INF]                            &(((((&           &(((((@
[17:53:18 INF]                           @(((((&             &(((((@
[17:53:18 INF]                           &(((((@#&@((.((&@@@(#(((((@
[17:53:18 INF]                          #((((#..................#@((&
[17:53:18 INF]                        &@(((((&......................(@
[17:53:18 INF]                      @.(&((((&...&&        &@&..........&@
[17:53:18 INF]                    @...@(((((@                   @#.......((
[17:53:18 INF]                  &.....@(((((@                   @((@.......&
[17:53:18 INF]                 @......@(((((                    #((((&.......&
[17:53:18 INF]                #.....( &(((((         @@@        ((((((@@......@
[17:53:18 INF]               &.....@  @(((&@@#(((((((((((((((((#@(((((&  ......@
[17:53:18 INF]              @.....@  &@&((((((((((((((((((((((((@(((((@#  ......@
[17:53:18 INF]             @.....&@(((((((((((((((&&@@@@@(((((@((((#(((#@(....&
[17:53:18 INF]             @.....&((((((((&@@&                 @(((((@(((((((@...#
[17:53:18 INF]             &....((((((@@(((((@                &@(((((@&((((((((#&&
[17:53:18 INF]            @(....&((@    @(((((@               @(((((@    @(((((((##
[17:53:18 INF]          @(#(....&        &(((((@             @(((((&       &@(((((((&
[17:53:18 INF]        &@(((&.....        @((((((&           @(((((       &.(&((((((@
[17:53:18 INF]       @(((((@.....&        (((((@        &@(((((&         @....@((((((@
[17:53:18 INF]      @(((((#@.....(          &(((((@&     ##(((((&         @.....@@((((((@
[17:53:18 INF]    (&(((((@  &.....@&         @((((((@   @((((((@         @......   @(((((@
[17:53:18 INF]    &(((((@    @.....#&         @#((((((@((((((#          @......&    @(((((@
[17:53:18 INF]   @(((((@      &......&          @(((((((@#((@         &@......       @(((((@
[17:53:18 INF]  @(((((@        @......@&        @@@(((((((&@&        @......(         #(((((@
[17:53:18 INF]  #((((&           &.......@  &@&(((((@#((((((((@@& &@.......@          ((((&
[17:53:18 INF] &(((((@@           @(....&@#((((((((((@ @(((((((#@........@            &@(((((@
[17:53:18 INF] &(((((((((((((((((((((((((((((((((&@@@@@@@@@&...........@(((((((((((((((((((((@
[17:53:18 INF] @(((((((((((((((((((((((((((((&@(....................@#((((((((((((((((((((((#@
[17:53:18 INF]       @((((((((((((((&@&  &&...................@   @@#((((((((((((((#@@
[17:53:18 INF]
[17:53:18 INF] ===============================================================================
[17:53:18 INF] Azure DevOps Migration Tools [Object Model]
[17:53:18 INF] v13.0.5.0
[17:53:18 INF] naked Agility with Martin Hinshelwood
[17:53:18 INF] ===============================================================================
[17:53:18 INF] Telemetry Note: We use Application Insights to collect telemetry on performance & feature usage for the tools to help our developers target features. This data is tied to a session ID that is generated and shown in the logs. This can help with debugging.
[17:53:18 INF] Start Time: 06/14/2023 17:53:18
[17:53:18 INF] Running with args: execute
[17:53:18 INF] OSVersion: Microsoft Windows NT 6.2.9200.0
[17:53:18 INF] Version: 13.0.5.0
[17:53:18 DBG] Hosting starting
[17:53:18 INF] Application started. Press Ctrl+C to shut down.
[17:53:18 INF] Hosting environment: Production
[17:53:18 INF] Content root path: C:\tools\MigrationTools
[17:53:18 INF] Press Ctrl+C to shut down.
[17:53:18 DBG] Hosting started
[17:53:18 INF] Config Found, creating engine host
[17:53:19 INF] Creating Migration Engine 5425a7a8-ee2a-42ee-9d06-efbb5e0aa724
[17:53:19 INF] ProcessorContainer: Of 1 configured Processors only 1 are enabled
[17:53:19 INF] ProcessorContainer: Adding Processor AzureDevOpsPipelineProcessor
[17:53:19 INF] Processor::Configure
[17:53:19 INF] Processor::Configure Processor Type AzureDevOpsPipelineProcessor
[17:53:19 INF] Creating endpoint with name Source Project
[17:53:19 DBG] Endpoint::Configure
[17:53:19 WRN] No Enrichers have been Configured
[17:53:19 DBG] AzureDevOpsEndpoint::Configure
[17:53:19 INF] Creating endpoint with name Target Project
[17:53:19 DBG] Endpoint::Configure
[17:53:19 WRN] No Enrichers have been Configured
[17:53:19 DBG] AzureDevOpsEndpoint::Configure
[17:53:19 DBG] ProcessorEnricherContainer::ConfigureEnrichers
[17:53:19 WRN] No Enrichers have been Configured
[17:53:19 INF] AzureDevOpsPipelineProcessor::Configure
[17:53:19 INF] Logging has been configured and is set to: Information.
[17:53:19 INF]                               Max Logfile: Verbose.
[17:53:19 INF]                               Max Console: Debug.
[17:53:19 INF]                  Max Application Insights: Error.
[17:53:19 INF] The Max log levels above show where to go look for extra info. e.g. Even if you set the log level to Verbose you will only see that info in the Log File, however everything up to Debug will be in the Console.
[17:53:19 DBG] TypeDefinitionMapContainer: Adding Work Item Type Bug
[17:53:19 DBG] TypeDefinitionMapContainer: Adding Work Item Type User Story
[17:53:19 DBG] TypeDefinitionMapContainer: Adding Work Item Type Requirement
[17:53:19 DBG] TypeDefinitionMapContainer: Adding Work Item Type Task
[17:53:19 DBG] TypeDefinitionMapContainer: Adding Work Item Type Test Case
[17:53:19 DBG] TypeDefinitionMapContainer: Adding Work Item Type Chore
[17:53:19 DBG] TypeDefinitionMapContainer: Adding Work Item Type Shared Steps
[17:53:19 DBG] TypeDefinitionMapContainer: Adding Work Item Type Shared Parameter
[17:53:19 DBG] TypeDefinitionMapContainer: Adding Work Item Type Spike
[17:53:19 DBG] TypeDefinitionMapContainer: Adding Work Item Type Design
[17:53:19 INF] Beginning run of 1 processors
[17:53:19 INF] Processor: AzureDevOpsPipelineProcessor
[17:53:19 INF] Migration Context Start: AzureDevOpsPipelineProcessor
[17:53:19 INF] Processor::InternalExecute::Start
[17:53:19 INF] Processor::EnsureConfigured
[17:53:19 INF] ProcessorEnricherContainer::ProcessorExecutionBegin
[17:53:19 INF] Processing Service Connections..
[17:53:22 INF] 8 of 11 source ServiceConnection(s) are going to be migrated..
[17:53:23 ERR] Error migrating ServiceConnection: dev-AKS-EUS-CSP-DV-01-marketplace-1646765294031. Please migrate it manually.
Url: POST https://dev.azure.com/org-name//Target Project/_apis/serviceendpoint/endpoints/
{"$id":"1","innerException":null,"message":"Service connection with name dev-AKS-EUS-CSP-DV-01-marketplace-1646765294031 already exists. Only a user having Administrator/User role permissions on service connection dev-AKS-EUS-CSP-DV-01-marketplace-1646765294031 can see it. ","typeName":"Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi.DuplicateServiceConnectionException, Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi","typeKey":"DuplicateServiceConnectionException","errorCode":0,"eventId":3000}
[17:53:23 ERR] Error migrating ServiceConnection: acrpd. Please migrate it manually.
Url: POST https://dev.azure.com/org-name//Target Project/_apis/serviceendpoint/endpoints/
{"$id":"1","innerException":null,"message":"Service connection with name acrpd already exists. Only a user having Administrator/User role permissions on service connection acrpd can see it. ","typeName":"Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi.DuplicateServiceConnectionException, Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi","typeKey":"DuplicateServiceConnectionException","errorCode":0,"eventId":3000}
[17:53:24 ERR] Error migrating ServiceConnection: AKS-EUS-CSP-DV-01-last-mile. Please migrate it manually.
Url: POST https://dev.azure.com/org-name//Target Project/_apis/serviceendpoint/endpoints/
{"$id":"1","innerException":null,"message":"Service connection with name AKS-EUS-CSP-DV-01-last-mile already exists. Only a user having Administrator/User role permissions on service connection AKS-EUS-CSP-DV-01-last-mile can see it. ","typeName":"Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi.DuplicateServiceConnectionException, Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi","typeKey":"DuplicateServiceConnectionException","errorCode":0,"eventId":3000}
[17:53:24 ERR] Error migrating ServiceConnection: containerregistry-3791. Please migrate it manually.
Url: POST https://dev.azure.com/org-name//Target Project/_apis/serviceendpoint/endpoints/
{"$id":"1","innerException":null,"message":"Service connection with name containerregistry-3791 already exists. Only a user having Administrator/User role permissions on service connection containerregistry-3791 can see it. ","typeName":"Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi.DuplicateServiceConnectionException, Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi","typeKey":"DuplicateServiceConnectionException","errorCode":0,"eventId":3000}
[17:53:24 ERR] Error migrating ServiceConnection: containerregistry-2211. Please migrate it manually.
Url: POST https://dev.azure.com/org-name//Target Project/_apis/serviceendpoint/endpoints/
{"$id":"1","innerException":null,"message":"Service connection with name containerregistry-2211 already exists. Only a user having Administrator/User role permissions on service connection containerregistry-2211 can see it. ","typeName":"Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi.DuplicateServiceConnectionException, Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi","typeKey":"DuplicateServiceConnectionException","errorCode":0,"eventId":3000}
[17:53:25 ERR] Error migrating ServiceConnection: containerregistry-4894. Please migrate it manually.
Url: POST https://dev.azure.com/org-name//Target Project/_apis/serviceendpoint/endpoints/
{"$id":"1","innerException":null,"message":"Service connection with name containerregistry-4894 already exists. Only a user having Administrator/User role permissions on service connection containerregistry-4894 can see it. ","typeName":"Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi.DuplicateServiceConnectionException, Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi","typeKey":"DuplicateServiceConnectionException","errorCode":0,"eventId":3000}
[17:53:25 ERR] Error migrating ServiceConnection: AKS-EUS-CSP-DV-01-marketplace. Please migrate it manually.
Url: POST https://dev.azure.com/org-name//Target Project/_apis/serviceendpoint/endpoints/
{"$id":"1","innerException":null,"message":"Service connection with name AKS-EUS-CSP-DV-01-marketplace already exists. Only a user having Administrator/User role permissions on service connection AKS-EUS-CSP-DV-01-marketplace can see it. ","typeName":"Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi.DuplicateServiceConnectionException, Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi","typeKey":"DuplicateServiceConnectionException","errorCode":0,"eventId":3000}
[17:53:25 ERR] Error migrating ServiceConnection: containerregistry-2133. Please migrate it manually.
Url: POST https://dev.azure.com/org-name//Target Project/_apis/serviceendpoint/endpoints/
{"$id":"1","innerException":null,"message":"Service connection with name containerregistry-2133 already exists. Only a user having Administrator/User role permissions on service connection containerregistry-2133 can see it. ","typeName":"Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi.DuplicateServiceConnectionException, Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi","typeKey":"DuplicateServiceConnectionException","errorCode":0,"eventId":3000}
[17:53:25 INF] 0 of 8 ServiceConnection(s) got migrated..
[17:53:25 INF] The ServiceConnection acrdv(2734f90d-2e7b-434f-8a36-2557bb74be24) doesn't exsist in the source collection.
[17:53:25 INF] The ServiceConnection SC-EUS-AZDO-CUST-ACC-NP-01(a6b265fc-a44a-4f97-a13f-6df1deae00d4) doesn't exsist in the source collection.
[17:53:25 INF] The ServiceConnection SC-EUS-AZDO-CUST-ACC-PD-01(8fd65cf2-0b11-4c7b-a388-cf3fb91f59a8) doesn't exsist in the source collection.
[17:53:25 INF] Processing Variablegroups..
[17:53:25 INF] 0 of 0 source VariableGroups(s) are going to be migrated..
[17:53:25 INF] 0 of 0 VariableGroups(s) got migrated..
[17:53:25 INF] Processing Taskgroups..
[17:53:26 INF] 0 of 0 source TaskGroup(s) are going to be migrated..
[17:53:26 INF] 0 of 0 TaskGroup(s) got migrated..
[17:53:26 INF] Processing Build Pipelines..
[17:53:26 INF] Querying definitions in the project: Source%20Project
[17:53:26 INF] Configured BuildDefinition definitions: All
[17:53:30 INF] Querying definitions in the project: Target%20Project
[17:53:30 INF] Configured BuildDefinition definitions: All
[17:53:34 INF] 36 of 37 source BuildDefinition(s) are going to be migrated..
[17:53:34 FTL] Error while running AzureDevOpsPipelineProcessor
System.NullReferenceException: Object reference not set to an instance of an object.
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.MapRepositoriesInBuidDefinition(IEnumerable`1 sourceRepositories, IEnumerable`1 targetRepositories, BuildDefinition definitionToBeMigrated) in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 393
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.<CreateBuildPipelinesAsync>d__17.MoveNext() in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 312
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.<MigratePipelinesAsync>d__9.MoveNext() in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 95
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.InternalExecute() in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 49
   at MigrationTools.Processors.Processor.Execute() in D:\a\1\s\src\MigrationTools\Processors\Processor.cs:line 106
[17:53:34 INF] AzureDevOpsPipelineProcessor completed in 00:00:15.2145434
[17:53:34 ERR] AzureDevOpsPipelineProcessor The Processor MigrationEngine entered the failed state...stopping run
[17:53:34 INF] Application is shutting down...
[17:53:34 DBG] Hosting stopping
[17:53:34 DBG] Exiting with return code: 0
[17:53:34 INF] Application Ending
[17:53:35 INF] The application ran in 00:00:16.8595042 and finished at 06/14/2023 17:53:35
[17:53:35 DBG] Hosting stopped

What happened?

Received the following error while trying to migrate pipelines from one project to another. Additionally, the pipelines were not migrated.

Error while running AzureDevOpsPipelineProcessor System.NullReferenceException: Object reference not set to an instance of an object.

The error was preceded by a lot of errors migrating service connections. In this log, the errors are because the service connection already exists in the target, but I got similar errors even the first time I ran this migration to this particular target.

What I expected was for our pipelines have been migrated successfully to the target.

Debug in Visual Studio

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days