Support Private Authentication in Package Rules

[ekristen]

What would you like Renovate to be able to do?

With recent changes to github-releases using the graphql API, workarounds using hostRules to swap out authentication tokens based on an organization path query no longer works.

Being able to specify authentication data by packageRule would be a great replacement and would be a lot more flexible as the rule matching is far more flexible.

If you have any ideas on how this should be implemented, please tell us here.

Here is an example

   "packageRules": [
        {
            "matchDatasources": [
                "gitub-releases"
            ],
            "matchPackagePatterns": [
                "^org1"
            ],
            "encrypted": {
                "token": "wcFMA..."
            }
        }
    ],

Is this a feature you are interested in implementing yourself?

Yes

[rarkins]

@viceice @JamieMagee can you think of an elegant way to do this?

We'll need to add it to most of the interfaces for datasource, such as GetPkgReleasesConfig, GetReleasesConfig, etc.

Then is there any way to support this in all datasources using class inheritance so we don't need to add if statements to every single one?

[ekristen]

@rarkins @viceice @JamieMagee any movement on this? I was going to look into doing a PR shortly, but not if it's a waste of time.

[rarkins]

I think we need to implement our own pagination approach (ignoring the link next urls) so that the other will be consistent (using repo org and name and it repo ID). That way auth for custom repos should work

[ekristen]

Maybe? I still like the approach of being able to set in packageRules, it's definitely more flexible that way, but I don't have the knowledge of how this project is built as well as you, which is also why I think a PR from me would take a very long time, I would have to do a considerable amount of learning to understand how everything works first.

Let me know how my time might be best spent, helpful, or if I should just wait? Happy to just test as well.

[rarkins]

I'm in favour of both solutions. I think it's unintuitive that the GitHub API pagination directs to an ID-based undocumented API for subsequent pages, and using our own pagination would likely fix the problem for most. But, it would also be useful to allow credentials to be added to packageRules. The challenge is how to make sure that the same credentials are applied when pagination is used

[nabeelsaabna]

Is this related to any APIs other than https://api.github.com/graphql ?

if so, applying hostRules related to github-releases and github-tags only at:

https://github.com/renovatebot/renovate/blob/9f837f226d5718a5fd7445d0367c1d3e30488c03/lib/modules/datasource/github-releases/cache/cache-base.ts#L165-L182

buy sending a "good token" in the options will fix it, like:

      const graphqlRes = await this.http.postJson<
        GithubGraphqlResponse<QueryResponse<FetchedItem>>
      >('/graphql', {
        baseUrl,
        body: { query: this.graphqlQuery, variables },
        token: good_token_from_rules_for_org_and_repo
      });

don't believe its complicated since here we are always targeting a github organization and repository (straight forward)

[nabeelsaabna]

hard code this if statement to populate the token

    let overrideToken : string | undefined;
    if (variables.name === 'wrn251-private-b') {
      overrideToken = 'ghp_A***';
    }

and the result below,, the 3 modules got the correct update:

with config:

{
    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
    "dependencyDashboard": true,
    "hostRules": [
        {
            "matchHost": "https://gitlab.com",
            "token": "glpat-***",
            "hostType": "gitlab"
        },
        {
            "matchHost": "https://github.com/nabeelsaabna",
            "token": "ghp_A***",
            "hostType": "github"
        },
        {
            "matchHost": "https://api.github.com/repos/nabeelsaabna",
            "token": "ghp_A***",
            "hostType": "github"
        }   
    ],
    "customEnvVariables": {
        "GOPRIVATE": "github.com/nabeelsaabna,github.com/nabeelys,gitlab.com/nabeelsaabna",
        "GONOSUMDB": "github.com/nabeelsaabna,github.com/nabeelys,gitlab.com/nabeelsaabna",
        "GONOPROXY": "github.com/nabeelsaabna,github.com/nabeelys,gitlab.com/nabeelsaabna"
    },
    "postUpdateOptions": ["gomodTidy"],
    "branchPrefix": "rnv_prefix_9/"
}

[ekristen]

@nabeelsaabna I'm not sure how or why it's working at the moment for you but I've found that the host rules do not work at least not consistently anymore.

Either due to changes in renovate or changes by GitHub itself, some URLs end up being api.github.com/repo/<integer> instead of repos/org/... which then doesn't match the host rule and breaks.

This is why I'm suggesting package rules as it gives the best place to provide auth for packages and datasources and repositories that match the rule.

[rarkins]

Either due to changes in renovate or changes by GitHub itself, some URLs end up being api.github.com/repo/<integer> instead of repos/org/... which then doesn't match the host rule and breaks.

It's a change GitHub made to their pagination logic. The Link: header now returns /repo/id links instead of /repos/org/name.

[nabeelsaabna]

@nabeelsaabna I'm not sure how or why it's working at the moment for you but I've found that the host rules do not work at least not consistently anymore.

hard-coded a token for repository as an example of a code fix location

[viceice]

I've found maybe another way to use the right host rule, discussed it on slack with @rarkins .

will try that latest on monday

[nabeelsaabna]

https://github.com/renovatebot/renovate/blob/7e4aafc228f97cd6c19be16a13eb902c61a46f11/lib/util/http/host-rules.ts#L70-L74

It might be easier to pass an additional "hostRule-lookup-URL" in the option and use it in findMatchingRules() ? when calling /repo/id the additional URL will be the original /repos/org/name instead

[viceice]

no, it will handled in GitHub http. not generic for all http layer