[New Functionality] Allow build definition authorization to repository

straubt1 commented 4 years ago

Description

Currently the resource azuredevops_resource_authorization only supports service endpoints.

Feature request for adding build definition authorization to a repository.

If you are extending a build pipeline from one repository to another, ADO will prompt you to permit access the first time it is run

New or Affected Resource(s)

azuredevops_resource_authorization

Potential Terraform Configuration

azuredevops_resource_authorization

resource "azuredevops_resource_authorization" "pipeline_auth" {
  project_id  = azuredevops_project.project.id
  resource_id = azuredevops_build_definition.plan.id
  authorized  = true
}

References

resource_authorization

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

EliiseS commented 4 years ago

@straubt1 "If you are extending a build pipeline from one repository to another," Could you specify what you mean by that? I'm trying to repro the "This pipeline needs permissions to access a resource before this can continue to run"

straubt1 commented 4 years ago

Here is an example.

I have a file pipeline/tfe-run-template.yml in repository terraform-pipeline.

Then in repository terraform-code, I reference that pipeline file with the extends option:

trigger: none
pr:
  branches:
    include:
      - master

resources:
  repositories:
    - repository: terraform-pipeline
      type: Git
      name: terraform-tfe-ado/terraform-pipeline
      ref: master

variables:
  - group: "TFE Variable Group"

pool:
  vmImage: "ubuntu-latest"

extends:
  template: pipeline/tfe-run-template.yml@terraform-pipeline
  parameters:
    isSpeculativePlan: True

This results in the permissions request (see above screenshot).

EliiseS commented 4 years ago

Thanks for replying so fast! :D

EliiseS commented 4 years ago

@straubt1 When I set it up without a variable group reference the pipeline just runs, no popup. If I try to run it with a variable group I get an error: An error occurred while loading the YAML build pipeline. Variable group was not found or is not authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.. I'm unable to get the variable group authorized, the button was there, but it just returned the same error when pressed.

Here's the yaml for the pipeline that extends the pipeline in a different project.

trigger: none
pr:
  branches:
    include:
      - master

resources:
  repositories:
    - repository: sample-repo
      type: Git
      name: "Test Project/Sample Repo"
      ref: master

pool:
  vmImage: "ubuntu-latest"

extends:
  template: azure-pipelines.yml@sample-repo

The yaml of the pipeline being extended. The variable group referenced is in the same repo as this pipeline.

variables:
- group: "Sample VG 1"

steps:
- script: echo $(key1) # uses macro syntax
- script: echo $(key2) # uses macro syntax
- script: echo "mooo" # uses macro syntax

EDIT: I got to the This pipeline needs permissions to access a resource before this can continue to run error by creating a variable group with the same values in the extending pipelines project and using that variable group instead!

straubt1 commented 4 years ago

Interesting, I haven't had any issues with accessing the Variable Groups, just the other git repo.

Here is the project to give (hopefully) more information:

Namely Repo A Pipeline: https://github.com/straubt1/tfe-ado-project/blob/master/repo-terraform-code/ci/tfe-run-speculative.yml

Repo B Pipeline that is the template I am wishing to extend: https://github.com/straubt1/tfe-ado-project/blob/master/repo-pipeline-code/tfe-run-template.yml

I am able to replicate this every time.

EliiseS commented 4 years ago

I've managed to get the pipeline to run without pressing the button now too!

The API endpoint for this feature is AuthorizeDefinitionResources. Documentation: https://docs.microsoft.com/en-us/rest/api/azure/devops/build/resources/authorize%20definition%20resources?view=azure-devops-rest-5.1#definitionresourcereference

Using that API I've made the request below to add the variable group to the build pipeline resource list as authorized.

Request:

PATCH https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}/resources?api-version=5.1-preview.1

[
   {
      "authorized":true,
      "id":"346",
      "type":"variablegroup"
   }
]

Response:

{
    "count": 10,
    "value": [
        {
            "name": "Booooo",
            "type": "queue",
            "id": "1992",
            "authorized": true
        },
        {
            "name": "Hosted",
            "type": "queue",
            "id": "1984",
            "authorized": false
        },
        {
            "name": "Hosted VS2017",
            "type": "queue",
            "id": "1985",
            "authorized": false
        },
        {
            "name": "Hosted Windows 2019 with VS2019",
            "type": "queue",
            "id": "1986",
            "authorized": false
        },
        {
            "name": "Hosted Windows Container",
            "type": "queue",
            "id": "1987",
            "authorized": false
        },
        {
            "name": "Hosted macOS",
            "type": "queue",
            "id": "1988",
            "authorized": false
        },
        {
            "name": "Hosted macOS High Sierra",
            "type": "queue",
            "id": "1989",
            "authorized": false
        },
        {
            "name": "Hosted Ubuntu 1604",
            "type": "queue",
            "id": "1990",
            "authorized": false
        },
        {
            "name": "Azure Pipelines",
            "type": "queue",
            "id": "1991",
            "authorized": false
        },
        {
            "type": "variablegroup",
            "id": "346",
            "authorized": true
        }
    ]
}

API endpoint to get the list of authorized resource for a pipeline: GET https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}/resources?api-version=5.1-preview.1 Documentation: https://docs.microsoft.com/en-us/rest/api/azure/devops/build/resources/list?view=azure-devops-rest-5.1

I'll start implementing this tomorrow, but should this be under azuredevops_resource_authorization or azuredevops_definition_resource_authorization since it's a different API call under the hood? I'm thinking azuredevops_definition_resource_authorization since it's a different list of types that can be changed too.

@nmiodice @straubt1

nmiodice commented 4 years ago

It would be great if we could leverage azuredevops_resource_authorization for all resource authorizations since it will allow for simpler maintenance long term

straubt1 commented 4 years ago

I think this is the right path, but not exactly where I was stumbling. The endpoint I was able to scrape in the UI was pipelines/pipelinePermissions/repository/ (which btw doesn't appear to have any API documentation on MSFT websites I could find). This specifically gives a build definition permission to communicate with another repository via the resource yaml block.

I was able to get this to work with a "local-exec" hack for the short term, this should help aid you in updating the resource.

resource "null_resource" "pipeline-perm-plan-apply" {
  triggers = {
    id = azuredevops_build_definition.plan-apply.id
  }

  provisioner "local-exec" {
    command = <<EOF
id=${azuredevops_build_definition.plan-apply.id}
payload="{ \"pipelines\": [{ \"id\": $id, \"authorized\": true }]}"
echo $id
echo $payload
curl \
  -u tstraub:$AZDO_PERSONAL_ACCESS_TOKEN \
  -H "Content-Type: application/json" \
  --request PATCH \
  --data "$payload" \
  $AZDO_ORG_SERVICE_URL/${azuredevops_project.project.project_name}/_apis/pipelines/pipelinePermissions/repository/${azuredevops_git_repository.pipeline.project_id}.${azuredevops_git_repository.pipeline.id}?api-version=5.1-preview.1 | jq .
EOF
  }
}

EliiseS commented 4 years ago

@straubt1 yeah, you're right, my solution is solving the variable group issue I had. I didn't know you were using github repos. I also saw the API you pointed out being called when scraping the variable groups or something similar. I then decided to try to use the one I linked to try to see if it worked, and surprisingly it did. Even though I never saw it while scraping.

Have you tried using the API I linked for adding the permissions to the github? I think it might work for that too.

If you don't have time, I'll repro your scenario tomorrow.

@nmiodice you've convinced me.

straubt1 commented 4 years ago

@EliiseS That repo I linked is in GitHub but uses the ADO Provide to create ADO Repos that store the code. In other words, its meant to create an ADO project + repos + pipelines for integrating with Terraform Enterprise.

It is a little bit of terraform inception here, but the end state is all the work happens in ADO.

EliiseS commented 4 years ago

@straubt1 Do you think you could share the terraform you used to create the project, repos and pipeline?

straubt1 commented 4 years ago

This is the primary file: https://github.com/straubt1/tfe-ado-project/blob/tt/pipeline-perms/repo-terraform.tf

And here is the stop gap to fire the proper permissions call: https://github.com/straubt1/tfe-ado-project/blob/c8a5ff01093f0d9b85cd7009e5f50f2df4e01b0b/repo-terraform.tf#L175-L193

EliiseS commented 4 years ago

@straubt1 I've had to strip out the TFE parts since I don't have terraform enterprise, but I've created the repos in the same project, uploaded the files (manually, since the null_resource scripts to upload files didn't work for me) and created a normal variable group called "TFE Variable Group". I also didn't use the null_resource scripts here: https://github.com/straubt1/tfe-ado-project/blob/c8a5ff01093f0d9b85cd7009e5f50f2df4e01b0b/repo-terraform.tf#L154-L215. The end result for this was that the pipelines ran without any authorization required.

I'm unable to repro your issue with a pipeline requiring authorization to use a git repository.

Could you please provide minimal viable terraform to repro this without TFE? Is it possible to repro this without TFE?

straubt1 commented 4 years ago

Interesting...

Yeah you can comment out the TFE specific pieces, it won't cause an issue to reproduce.

My only thought is there is something going on at the ADO organization level? I can reproduce this 100% of the time.

Just to be clear, I get the error when I run the build pipeline.

Let me see if I can get a min set of TF to demo this.

EliiseS commented 4 years ago

@nmiodice

@straubt1 and I spent some time on a call to try to get to the bottom of this, but we ended up more confused.

We've both ran the terraform in this gist against our respective Azure Devops organizations where were both owners with Full access PAT tokens. The files uploaded by the terraform need to be copied from here: https://github.com/straubt1/tfe-ado-project/tree/tt/pipeline-perms and need

He gets the error, while I don't.

I've ran out of ideas for how to repo this.

EliiseS commented 4 years ago

I was able to reproduce the repository authorization issue on @nmiodice AzDO. This allowed me to play around with the API enough to come to the same conclusion as @straubt1 in terms of which endpoint is required to add/remove/view the permissions.

I've tried to add this permission with the AuthorizeDefinitionResources endpoint, but I was unable to.

Adding pipeline to repos permissions:

PATCH: https://dev.azure.com/niiodice//_apis/pipelines/pipelinePermissions/repository/.?api-version=5.1-preview.1

BODY:

{"pipelines": [{"id": 220, "authorized": true}]}

NB: The /repository/ is followed by a both project_id and repo_id separated by a dot: project_id>.<repo_id>. Just repo_id is invalid.

Response:

{
    "resource": {
        "type": "repository",
        "id": "<project_id>.<repo_id>"
    },
    "pipelines": [
        {
            "id": 220,
            "authorized": true,
            "authorizedBy": {
                "displayName": "<name>",
                "id": "<user_id",
                "uniqueName": "<email>",
                "descriptor": "aad.Zjc1OdcWZjNDEtMWFhZSzdcdzd03ZjMwLThhYTItOTVjMWU5ZDk2N2Q4"
            },
            "authorizedOn": "2020-06-30T16:56:36.867Z"
        }
    ]
}

Delete permission: above request, but with "authorized" : false

List repo permission

GET: https://dev.azure.com/niiodice//_apis/pipelines/pipelinePermissions/repository/.?api-version=5.1-preview.1

Response: Same as above

Implementation

As @straubt1 said, this AzDO API endpoint is not in the documentation and I also couldn't find it in the AzDO Go SDK. If anyone, (@tedchamb, @nmiodice, @tmeckel), can affirm the SDK part, that would be great.

@nmiodice @xuzhang3 @tmeckel Working on the assumption it's not in the SDK, do we have a standard place we put rogue AzDO endpoints? Should we add the implementation to the SDK or keep it in AzDO repo? If in the AzDO repo, where?

tmeckel commented 4 years ago

@EliiseS "...rogue AzDO endpoints" nice wording 😁 👍

Well actually you get get a list of ALL REST API Endpoints by querying the /_apis/ResourceAreas endpoint for an organization and using an OPTION HTTP request on the (location) Url for that particular resource area to get all available REST Endpoints (routing templates). This is also the procedure that the Azure DevOps Go API uses internally. Well, actually all APIs for the different programming languages use this method. The only issue you have: you don't know the structure of the input parameters. There's no API you can query for that. You only can query for the, so called, routing templates for the REST API.

Okay because I'm a freak 🤭 I can say that the data structure you've to pass, and what @straubt1 already discovered, has to look like the following

[DataContract]
public class ResourcePipelinePermissions
{
    private List<PipelinePermission> m_pipelines;

    [DataMember(EmitDefaultValue = false)]
    public Resource Resource { get; set; }

    [DataMember(EmitDefaultValue = false)]
    public Permission AllPipelines { get; set; }

    [DataMember(EmitDefaultValue = false)]
    public List<PipelinePermission> Pipelines
    {
      get
      {
        return this.m_pipelines ?? (this.m_pipelines = new List<PipelinePermission>());
      }
      set
      {
        this.m_pipelines = value;
      }
    }
}

Nevertheless, from my point of view @nmiodice, @xuzhang3 or you should get in contact with the product team to discuss how to handle those rogue (I really like the wording!) interfaces.

tmeckel commented 4 years ago

@EliiseS I can add that there is a DotNet Client implementation available in Microsoft.Azure.Pipelines.Authorization.WebApi.dll but this DLL is not available via Nuget. So, as you said, this API isn't meant to be used by other clients than Azure DevOps itself and we consider it non public.

tmeckel commented 4 years ago

@EliiseS The following string values are valid types for target resource , when setting permissions for a pipeline

endpoint
queue
securefile
variablegroup
environment
agentpool

Based on the above shown C# data structure, the following JSON documents can be used to set permissions on resources for pipelines:

Authorize all pipelines for a specific resource

{
    "resource": {
        "type": "@@ResourceType@@",
        "id": "@@ResourceId@@"
    },
    "allpipelines": {
        "authorized": true
    }
}

Obviously by passing false to authorized the authorization for all pipelines will be revoked.

Add authorization for list of pipelines for a specific resource

{
    "resource": {
        "type": "@@ResourceType@@",
        "id": "@@ResourceId@@"
    },
    "pipelines": [
        {
            "id": "@@PipelineId@@",
            "authorized": true
        },
        {
            "id": "@@PipelineId@@",
            "authorized": true
        }
    ]
}

The REST endpoint is capable of handling a list (array) of pipeline permissions as a batch request

[
    {
        "resource": {
            "type": "@@ResourceType@@",
            "id": "@@ResourceId@@"
        },
        "allpipelines": {
            "authorized": true
        }
    },
    {
        "resource": {
            "type": "@@ResourceType@@",
            "id": "@@ResourceId@@"
        },
        "pipelines": [
            {
                "id": "@@PipelineId@@",
                "authorized": true
            },
            {
                "id": "@@PipelineId@@",
                "authorized": true
            }
        ]
    }
]

josh-barker-coles commented 3 years ago

Hi @EliiseS, @tmeckel & @xuzhang3 Is there any movement on this issue? I also need to authorize a pipeline to access a repo.

Cheers!

djtjwillia commented 3 years ago

Hey @EliiseS, @tmeckel & @xuzhang3 Is there any movement on this issue? We also could really use this feature!

petri-ojala commented 1 year ago

I wouldn't usually ask for status or anything but as the go sdk has been updated and the provider has got quite a bit of features recently, perhaps time to add this one too?

It's the only manual task we currently need to do when managing the Azure DevOps pipelines from terraform and like others, did the work to debug the exact API calls and whatnots myself as well :-)

hilariocoelho commented 7 months ago

Are there any news regarding this issue? Like @petri-ojala, it's the only task we need to perform manually 😞

microsoft / terraform-provider-azuredevops