search

Azure / azure-devops-cli-extension

Azure DevOps Extension for Azure CLI
https://docs.microsoft.com/en-us/cli/azure/ext/azure-devops/?view=azure-cli-latest
MIT License
617 stars 239 forks source link

Error on mass fetching az repos ref list #1355

Open jeroensmink98 opened 1 year ago

jeroensmink98 commented 1 year ago

This is autogenerated. Please review and update as needed.

Describe the bug

(See the additional content for my script) which uses the Azure CLI

I have a custom script which I use to fetch all branches for all my repositories under my project. The end goal would be to see what branches haven't been commited to in the last 120 days so we can mark them for deletion as part of our internal policy.

For that I wrote a custom script which fetches all repositories under a project and then fetches each branch for a repository but the Azure CLI doesn't really seem to like this. approach

Command Name az devops invoke Extension Name: azure-devops. Version: 0.26.0.

Errors:

The command failed with an unexpected error. Here is the traceback:
'defaultBranch=refs/heads/main; id=6b9be419-daf8-480a-aee6-01d369a64384; isDisabled=False; isFork=; isInMaintenance=False; name=tres-internet-terraform; parentRepository=; project=; remoteUrl=https'
Traceback (most recent call last):
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 663, in execute
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 718, in _run_job
  File "C:\Users\JeroenSmink\.azure\cliextensions\azure-devops\azext_devops\dev\common\exception_handler.py", line 31, in azure_devops_exception_handler
    reraise(*sys.exc_info())
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\six.py", line 703, in reraise
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 697, in _run_job
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 333, in __call__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "C:\Users\JeroenSmink\.azure\cliextensions\azure-devops\azext_devops\dev\team\invoke.py", line 101, in invoke
    response = client._send(http_method=http_method,
  File "C:\Users\JeroenSmink\.azure\cliextensions\azure-devops\azext_devops\devops_sdk\client.py", line 60, in _send
    request = self._create_request_message(http_method=http_method,
  File "C:\Users\JeroenSmink\.azure\cliextensions\azure-devops\azext_devops\devops_sdk\client.py", line 117, in _create_request_message
    request = ClientRequest(method=http_method, url=self._client.format_url(url))
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msrest/service_client.py", line 151, in format_url
KeyError: 'defaultBranch=refs/heads/main; id=6b9be419-daf8-480a-aee6-01d369a64384; isDisabled=False; isFork=; isInMaintenance=False; name=tres-internet-terraform; parentRepository=; project=; remoteUrl=https'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Expected Behavior

Environment Summary

Windows-10-10.0.22621-SP0
Python 3.10.5
Installer: MSI

azure-cli 2.40.0 *

Extensions:
azure-devops 0.26.0

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

This is my powershell script

$project    = "SOME_PROJECT_NAME"
try {
    $repos = az repos list -p $project  | ConvertFrom-Json
} catch {
    Write-Host "Failed to fetch repositories for project $project"
    exit
}

$results = @()

Write-Host "Project: $project"
foreach ($repo in $repos) {
    Write-Host "Repository: $($repo.name)"
    try {
        $refs = az repos ref list -p $project -r $repo.name --filter heads | ConvertFrom-Json
    } catch {
        Write-Host "Failed to fetch refs for repository $($repo.name)"
        continue
    }

    foreach($ref in $refs) {
        $objectId = $ref.objectId

        # fetch individual commit details
        try {
            $commit = az devops invoke `
                         --area git `
                         --resource commits `
                         --route-parameters `
                             project=$project `
                             repositoryId=$repo.name `
                             commitId=$objectId |
                      ConvertFrom-Json
        } catch {
            Write-Host "Failed to fetch commit details for ref $objectId"
            continue
        }

        $result = [PSCustomObject]@{
            repository = $repo.name
            name = $ref.name
            creator = $ref.creator.uniqueName
            lastAuthor = $commit.committer.email
            lastModified = $commit.push.date
        }

        $results += ,$result
    }
    # add a 500ms wait
    Start-Sleep -Milliseconds 500
}

$results | Sort-Object -Property lastModified -Descending | Format-Table -AutoSize

# Write to CSV
$results | Sort-Object -Property lastModified -Descending | Export-Csv -Path "output.csv" -NoTypeInformation
jeroensmink98 commented 1 year ago

Here is the output when I set --verbose for the az devops invoke 

INFO: GitDetect: Could not detect current remotes based on current working directory.
INFO: Detect: Url discovery took 0:00:00.037583
ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR: 'defaultBranch=refs/heads/master; id=547913c5-ea8f-4d51-8221-005ce0246615; isDisabled=False; isFork=; isInMaintenance=False; name=efo-umbraco; parentRepository=; project=; remoteUrl=https'
Traceback (most recent call last):
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 663, in execute
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 718, in _run_job
  File "C:\Users\JeroenSmink\.azure\cliextensions\azure-devops\azext_devops\dev\common\exception_handler.py", line 31, in azure_devops_exception_handler
    reraise(*sys.exc_info())
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\six.py", line 703, in reraise
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 697, in _run_job
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 333, in __call__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "C:\Users\JeroenSmink\.azure\cliextensions\azure-devops\azext_devops\dev\team\invoke.py", line 101, in invoke
    response = client._send(http_method=http_method,
  File "C:\Users\JeroenSmink\.azure\cliextensions\azure-devops\azext_devops\devops_sdk\client.py", line 60, in _send
    request = self._create_request_message(http_method=http_method,
  File "C:\Users\JeroenSmink\.azure\cliextensions\azure-devops\azext_devops\devops_sdk\client.py", line 117, in _create_request_message
    request = ClientRequest(method=http_method, url=self._client.format_url(url))
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msrest/service_client.py", line 151, in format_url
KeyError: 'defaultBranch=refs/heads/master; id=547913c5-ea8f-4d51-8221-005ce0246615; isDisabled=False; isFork=; isInMaintenance=False; name=efo-umbraco; parentRepository=; project=; remoteUrl=https'
To open an issue, please run: 'az feedback'