Allow Renovate's azure-pipeline's manager to read azure repositories defined in resource blocks

JackRhodes commented 2 years ago

What would you like Renovate to be able to do?

TLDR: allow renovate to update azure repository references within azure-pipelines yaml files. It currently isn't very well documented that it doesn't support this

Renovate has support for azure-pipelines, it does this by reading the Resources block defined in an azure pipelines yaml file

From this, it is able to update:

Containers
Git repositories

A git repository in azure pipeline has a type depending on where the repo is hosted, docs here:

git, (azure repository reference)
github,
githubenterprise,
bitbucket`.

Within Renovate's current manager implementation , it only supports github and will return null for all other types. This is not currently documented within Renovate and it is not clear that this is the case unless you view the implementation.

I would like to propose that support for git type is added, which will enable it to work within Azure Devops. Alongside this work, I would like to propose that the documentation is updated to state which repository types renovate supports in the manager to better define what it supports.

GIVEN an azure pipeline, WHEN it has an azure repository reference in its resources THEN it should check if the repository has a newer version

Please let me know if any more detail is needed, or if I am misunderstanding anything.

If you have any ideas on how this should be implemented, please tell us here.

(I am not a typescript dev and this is my first attempt at a suggestion)

If the repository reference is of type git, and the renovate platform is set to azure and has an endpoint defined, then we can construct the repository url reference.

To construct this, the following inputs need to be handled

myRepository which would produce: <endpoint>/currentProject/_git/myRepository
otherProject/MyRepositoryInOtherProject which would produce <endpoint>/otherProject/_git/MyRepositoryInOtherProject

This can be added to lib/modules/manager/azure-pipelines/extract.ts

I am unsure the cleanest way to achieve this as your would be mixing the manager domain with the platform domain, but I believe the actual change should be relatively simple code change. Azure repositories can only be used within ADO so coupling the platform and manager is not as bad as it sounds.

I am happy to implement this, but would require guidance about the best way to handle this.

Is this a feature you are interested in implementing yourself?

Maybe

rarkins commented 2 years ago

Is it possible to create a public example on Azure DevOps?

JackRhodes commented 2 years ago

Yes I believe so. I'll try to setup a test ADO project with an example

github-actions[bot] commented 2 years ago

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

JackRhodes commented 2 years ago

Apologies for the delay! I had to setup an ADO project which took a while

Renovate:

The issue:

Affected yaml file : this has an example of a github reference (that works), and an ADO git reference (that renovate does not generate for)
Pull Requests : it has only generated a PR for the github dependency
Affected line in logs : as you can see, it only sees the github repo. This is because of what I've described in my original issue, where only repositories with type: github are looked at

Please let me know if anything is unclear or if you can't access any of the links!

rarkins commented 2 years ago

What is the expected result for the git dependency?

JackRhodes commented 2 years ago

In the example , renovate should be upping the version of Renovate-Playground/example-repository from 0.1.0 to 0.2.0

Currently, Renovate ignores the git dependency and only creates a PR for github https://dev.azure.com/combarcapes/Renovate-Playground/_git/issue-15028/pullrequests?_a=active

rarkins commented 2 years ago

So it's defined as a "git" dependency yet it's implicitly pointing to Azure DevOps Services?

JackRhodes commented 2 years ago

Correct

Docs here

rarkins commented 2 years ago

If the git reference is e.g. MyBitbucket/vscode then doesn't that mean we know the full name of the repo without needing to know anything about the current platform, endpoint or repository? i.e. it's always going to point to azure devops services?

hypervtechnics commented 2 years ago

as you can see in the docs it only will fetch from bitbucket if type: bitbucket. The type indicates rhe source of the repo. If type: git it will point to Azure DevOps (git sounds generic although Azure DevOps is meant)

rarkins commented 2 years ago

That doesn't answer my question

hypervtechnics commented 2 years ago

When you use type: git and name: my-repository it will point to https://dev.azure.com/my-organization/my-project/_git/my-repository. We can not infer my-project and my-organization from the azure pipelines file. That's only doable when we have the properties webUrl or remoteUrl available from the repository details.

JackRhodes commented 2 years ago

If the git reference is e.g. MyBitbucket/vscode then doesn't that mean we know the full name of the repo without needing to know anything about the current platform, endpoint or repository? i.e. it's always going to point to azure devops services?

So yes, git will always point to the current azure devops organization. We know the full name of the repository, and potentially the project, but we do not have any way to see the organization from just the yaml file. Azure pipelines with a git reference will implicitly use the organization url for you since they sit in the same organization so it never gets explicitly defined in a pipeline

rarkins commented 2 years ago

In that case you can follow the example from GitLab: https://github.com/renovatebot/renovate/blob/main/lib/modules/manager/gitlabci-include/extract.ts#L68

hypervtechnics commented 2 years ago

Thanks for the hint 😄 Unfortunetaly this works only in a subset of cases. We also have to know about the project name which is often omitted in the pipeline file if a repository in the same project is referenced. We would need to have the repository details available.

JackRhodes commented 2 years ago

@hypervtechnics For a first iteration, it could be ok to make it so we explicitly add the project name in the resource reference and not support implicit project lookups.

But yes, you are correct that for full behavior, then we would also need a way to lookup the current project that a repository is located

hypervtechnics commented 2 years ago

But I think we can already start discussion on handling it properly in the second iteration. Adding the project name everywhere is kind of a very huge task when you have a lot of projects and pipelines.

The only way I can think of is having the repository details (including the git clone uri) to fetch the project somewhat reliably. The endpoint url in the configuration is not suitable because it only points to the organization and not the project in the organization.

HonkingGoose commented 1 year ago

When you've created this feature, please edit the https://github.com/renovatebot/renovate/blob/main/lib/modules/manager/azure-pipelines/readme.md file, to remove the part of the admonition that mentions feature 15028. 😉

sheldonhull commented 1 year ago

UPDATE: Had to do quite a few iterations to get this right but finally got it working! I believe the issue was the filematch expression failing and a mismatch between local and remote (due to the way the docker container test runs and clones inside itself). Appreciate the hard work on this, as it's great to see version pinning working and updates coming in. Cheers

I'm a little confused on the state here. I reviewed the https://github.com/renovatebot/renovate/pull/20380 and indicates that as long as the Project/Repo format is followed (instead of also handling implicit project determination) then it should be merged in and working right?

I am running a test case with "renovateVersion": "35.146.1" docker version locally, and while one pipeline seems to be matched, it never takes any action on the out of date tag. The docs also still reference this admonition. If i need to open another discussion instead I am willing, just thought this would be the appropriate follow-up place.

ceng-p commented 1 year ago

@sheldonhull you got it working as in the feature works for you? Could you share a working example, as I can't seem to get it working. It correctly gets the dependencies and current tag, but fails to see updates:

DEBUG: packageFiles with updates (repository=myorg/myrepo, baseBranch=main)
       "config": {
         "azure-pipelines": [
           {
             "deps": [
               {
                 "autoReplaceStringTemplate": "refs/tags/{{newValue}}",
                 "currentValue": "1.7.0",
                 "datasource": "git-tags",
                 "depName": "myproject/mytemplates",
                 "depType": "gitTags",
                 "packageName": "https://dev.azure.com/myorg/myproject/_git/myrepo",
                 "replaceString": "refs/tags/1.7.0",
                 "updates": [],
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "sourceUrl": "https://dev.azure.com/myorg/myproject/_git/myrepo",
                 "currentVersion": "1.7.0",
                 "fixedVersion": "1.7.0"
               },

running a git tag -l
```bash
...

1.7.0
1.7.1
1.7.2
1.7.3
1.7.4
1.7.5

renovatebot / renovate