Make the Github-URL for fetching dependeny changelogs configurable

[dprilmueller]

What would you like Renovate to be able to do?

I would like to make use of the Github.com changelog fetching feature in a gitlab-renovate-runner project. The problem is, that direct access to the internet is not possible. Instead all requests have to be proxied via nexus. To make this possible, a custom base-url for the changelog requests would be an option.

Current state when fetching changelog for dependency mapstruct for example: Fetching changelog: https://github.com/mapstruct/mapstruct (1.4.0.Final -> 1.4.2.Final) (repository=***/rvs-node-dependency-upgrades-poc) "url": "https://api.github.com/repos/mapstruct/mapstruct/tags?per_page=100",

The github.com part (and/or the api.github.com?) would be replaced with the proxy-url.

If you have any ideas on how this should be implemented, please tell us here.

-

Is this a feature you are interested in implementing yourself?

No

[rarkins]

Can you show a before/after example of the URLs?

[dprilmueller]

Before: https://github.com/mapstruct/mapstruct After: http(s)://proxyurl.com/mapstruct/mapstruct

Of course the "mapstruct/mapstruct" part is just an example for a dependency on github.

[rarkins]

So there's a dedicated proxyurl host which only proxies GitHub.com? Or is there a path prefix not shown in your example?

[dprilmueller]

"https://github.com" is the part of the URL that should be configurable.

Otherwise fetching changelogs is not possible due to network restrictions. In our case, our gitlab instance is not allowed to access external services like github. We have to use an internal proxy therefore. (We do the same for the npm-registry, in this case renovate supports custom registries which solves the problem).

I guess others have the same problem and need to set fetchReleaseNotes to false in this case, but thats no soluation.

[rarkins]

And you can't use a HTTPS_PROXY?

[thor]

To chime in, @rarkins: we have a similar use-case, except it's not Nexus we're using. In our case, perhaps similarly to @dprilmueller's, we cannot use a HTTPS_PROXY.

In our case the URLs look something like https://service.company.com/some/api/path/github/$GITHUB_ENDPOINT.

Alternatively, we'd need to look closer into "solving" the problem of missing HTTPS_PROXY for us to use.

[rarkins]

Does your alternative URL support the full API of GitHub.com? As in it's essentially a full transparent proxy?

[thor]

Yes and no: in terms of making simple reading-based requests, I'd say yes. In terms of altering any content or pushing any updates, I'm not sure: I've never tried to POST through it. When it comes to fetching, it does feel as though it is a fully transparent proxy. We've never had issues with any typically read-based method like LIST, GET or OPTIONS.

EDIT: We do have some API gateways too that can work as fully transparent proxies if need be. The service we use is Artifactory, which is similar to Nexus. We point all of our internal registryUrls to it for the different datasources we have available.

[rarkins]

You'd need token authentication for github.com too, otherwise you'd hit rate limiting very quickly. How is that handled?

[thor]

Artifactory handles that on its side; we configure Artifactory to use the specified token for communication with github.com. If multiple teams need access to the extent that we need multiple tokens, we could/would create some/api/path/new-team/github and allow users to point to that. Currently we only use the one some/api/path/github where we have a read token.

If it's a straightforward Bearer token (IIRC) it could be used to transparently authenticate to the API proxy endpoint that Artifactory provides, except it'd be a token for the specific Artifactory endpoint, rather than necessarily being for the GitHub API (as that would be handled inside Artifactory).

[rarkins]

We could potentially solve this with a new config option as part of hostRules, e.g. called replaceHost or similar.

Can someone give a more concrete example, for example what would the URL https://api.github.com/repos/mapstruct/mapstruct/tags?per_page=100 be replaced with?

And do you have a separate proxy endpoint for https://github.com/ too?

[dprilmueller]

The URL would be replaced with something like: https://proxy.company.com/some/repos/mapstruct/tags?per_page=100

Yes, if renovate also accesses https://github.com (not only api.github.com) we need a seperate proxy endpoint for that as well.

[rarkins]

So the new option could be called hostProxy and accepts a URL like https://githubapiproxy.company.com or https://proxy.company.com/githubapi/.

It would be somewhat similar to insecureRegistry in that it changes the URL (https://github.com/renovatebot/renovate/blob/4b16903ff1868a75a9598dad685fde7a60e2b2bd/lib/datasource/docker/common.ts#L252-L254) although should be implemented in lib/util/http.

Maybe also needs some code changes made so that we don't automatically disable github.com changelog lookups when a github.com token is not found (although you could potentially just configure a dummy token which will never be used).

I'm marking this as "status:ready" but it will need a community contribution from someone who needs this.. including because it should be tested in a real environment before merge.

[gaving]

Also interested in this functionality since direct net access is being shut down in favour of everything going through Nexus.