search

dependabot / dependabot-core

🤖 Dependabot's core logic for creating update PRs.
https://docs.github.com/en/code-security/dependabot
MIT License
4.69k stars 1.02k forks source link

Dependabot should work for GHES without public internet access #6829

Closed leifwalsh closed 1 year ago

leifwalsh commented 1 year ago

Is there an existing issue for this?

Feature description

I would like to run Dependabot actions on self-hosted runners connected to a GitHub Enterprise Server instance, to do version updates based on private registries. The runners cannot be given internet access, they need to stay behind a firewall, and only have access to internal resources (those registries).

This seems like it should be possible with private registry support, but I tried it and the Dependabot actions that do run fail on their first step: running actions/setup-node@v3.

I found https://docs.github.com/en/enterprise-server@3.8/admin/github-actions/managing-access-to-actions-from-githubcom/setting-up-the-tool-cache-on-self-hosted-runners-without-internet-access which is probably what you're going to ask me to do, but it sounds like a headache.

I have ~two~ three questions:

  1. Since modern GitHub Actions Runners come with I think node 16 already, could the Dependabot actions just use that node instead of fetching their own?
  2. If I get past this problem, am I going to immediately run into the problem that they can't docker pull from public registries? I can't find anything in the docs talking about this. We have internal docker registries that mirror external ones which we'd have no problem using, but I'd need some way to tell the Dependabot code to use different URL prefixes for the images it wants to run, and I don't know a) which images it is going to want or b) if there's any way to configure that.
  3. Generally, could GHES just come with all the binary artifacts that the other features of the GitHub product suite are going to need? Like, could the Dependabot runner fetch node and the Docker images it wants from our GHES instance? (This is probably a better question for the folks who package GHES, I'll admit)
honeyankit commented 1 year ago

@leifwalsh :

Point1:

Since modern GitHub Actions Runners come with I think node 16 already, could the Dependabot actions just use that node instead of fetching their own?

This is something we can look but right now, we have other high priorities task and will put this in backlog.

honeyankit commented 1 year ago

@leifwalsh :

For your point2:

We have internal docker registries that mirror external ones which we'd have no problem using, but I'd need some way to tell the Dependabot code to use different URL prefixes

You can use replaces-base. More can be found in the docs.

I don't know a) which images it is going to want or b) if there's any way to configure that.

Steps to setup the Dependabot and self-hosted action runner so they can run with limited internet access.

honeyankit commented 1 year ago

Point3:

Dependabot runner fetch node

This is something which comes under Actions team.

Docker images it wants from our GHES instance? (This is probably a better question for the folks who package GHES, I'll admit)

I understand your concern. For this you are right we need to sync with GHES team. However, at the moment, we have other priorities that we need to focus on. Perhaps we can revisit this task at a later time once we've addressed our current priorities.

leifwalsh commented 1 year ago

Thanks for all your input! This is a ton of new information, give me a couple days to try everything you suggested and thank you very much.

About "packaging GHES with the things it will need", I totally get it: it's hard and cross-team. Should I leave that idea in your court or would you like me to raise the idea elsewhere with the GHES team or the support portal? Happy to do whatever is most helpful to your team.

leifwalsh commented 1 year ago

First data point: I tried the "populate the tool-cache" workflow and it didn't work. I set up a runner on a VM on which I'd populated the tool cache in the way described by that documentation but it didn't appear to make it work:

> Run actions/setup-node@v3
##[debug]isExplicit:
##[debug]explicit?false
##[debug]evaluating 0 versions
##[debug]match not found
Attempting to download 16...
##[debug]No manifest cached
##[debug]Getting manifest from actions/node-versions@main
connect ETIMEDOUT 140.82.112.6:443
##[debug]Error: connect ETIMEDOUT 140.82.112.6:443
##[debug]    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1278:16)
Falling back to download directly from Node
##[debug]No dist manifest cached
Error: connect ETIMEDOUT 104.20.23.46:443
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Setup node

Again, not really sure what team to direct this to, if you want to redirect me elsewhere or think I should raise a support ticket, I can.

For the docker stuff, we have internal mirrors of other Docker registries but not ghcr.io. I don't think it's that hard for us to add another, and I've started that process, but assuming we get past installing node 16 I might get stuck at that next step until we resolve that internally.

Just to try to clear up any confusion: the nodes we're going to run dependabot on are isolated from the public internet from the beginning. We can't start them up with internet access, docker pull a couple things, and then firewall them before running the actual dependabot workflows. We need them to be able to fetch Docker images through our caching proxies at the outset (where I think getting an internal mirror of ghcr.io will help) - it's not something where we can fetch a bunch of images and then disconnect from the internet.

Finally, I reread your earlier comment:

We have internal docker registries that mirror external ones which we'd have no problem using, but I'd need some way to tell the Dependabot code to use different URL prefixes

You can use replaces-base. More can be found in the docs.

I think you might have been answering a question about how to get dependabot to update docker images that live on our registry? I think replaces-base is a dependabot.yml-specific idea, about telling dependabot that if it wants to upgrade docker images it should look in a different docker image registry for other versions. What I actually meant is that for the Docker images that dependabot needs to run, I'd like to have it fetch them from corpnet.internal/debian:unstable-slim rather than debian:unstable-slim from Docker Hub. But maybe I was lazy in searching and it's something more like a Docker config option where we can tell Docker that an image from one registry should actually be pulled from a different URL (which would work great for us, but I don't know how to do that - I could ask our Docker experts if that's a thing, certainly). What I was actually talking about is: the dependabot instructions suggested to me that the runner is going to want to run containers from Docker images (which makes sense to me - maybe you have specific images that know how to upgrade Gemfiles or upgrade Maven BOMs or whatever). If they're going to want to docker pull or docker run during the dependabot execution, we're going to need to make sure that we have the images those parts of the dependabot workflow wants cached or available in our caching proxies, so they can upgrade whatever language/package ecosystem is relevant to the repo the dependabot workflow is trying to upgrade.

honeyankit commented 1 year ago

Just to try to clear up any confusion: the nodes we're going to run dependabot on are isolated from the public internet from the beginning. We can't start them up with internet access, docker pull a couple things, and then firewall them before running the actual dependabot workflows.

Q. Can you access those self-hosted action runner ? if yes, how?

If they're going to want to docker pull or docker run during the dependabot execution, we're going to need to make sure that we have the images those parts of the dependabot workflow wants cached or available in our caching proxies.

I am trying to understand your problem. Your requirement is that all the docker images which is required to run the Dependabot on the self hosted runner needs to be present in the caching registry (say corpnet.internal/debian:unstable-slim) and should be pulled from there.

If my understanding is correct then there are two ways to do it if you have access to the self-hosted action runners.

  1. You will need to download all the docker images beforehand on the self-hosted action runner that is sufficient to run the Dependabot jobs. One way to do it is to download all the docker images on another machine (dev machine/staging machine), tar (docker save), copy to all self-hosted action runners and untar (docker load) those docker images. Only thing to remember is docker images tags should match from the container.json.

  2. You can also pull all the docker images on your dev machine and push to the repo of your choice (say: corpnet.internal/debian:unstable-slim) and again you will need to pull all the docker images on the self-hosted action runner and tag it to work properly with some automation or via login to those machines and manually pulling those images.

Note: When using Dependabot on a self-hosted runner, the runner must have the correct Docker images with the ghcr.io/dependabot tag, otherwise Dependabot will retrieve them from the ghcr.io registry. It is not possible to use a different caching registry to fetch these images when running Dependabot on a self-hosted action runner at this time.

honeyankit commented 1 year ago

"populate the tool-cache" workflow and it didn't work

I think you need to create a support ticket with Actions team.

honeyankit commented 1 year ago

About "packaging GHES with the things it will need

Let me talk to my team on this and will get back on best course of action on this issue.

leifwalsh commented 1 year ago

Q. Can you access those self-hosted action runner ? if yes, how?

I can ssh to them. They're (well, I just have one right now) VMs where we can run rootless docker and I've installed the runner package. They're on our network but our firewall doesn't let them access resources outside our network.

I am trying to understand your problem. Your requirement is that all the docker images which is required to run the Dependabot on the self hosted runner needs to be present in the caching registry (say corpnet.internal/debian:unstable-slim) and should be pulled from there.

We have an internal Artifactory that can mirror external Docker registries. I thought we need to add config to it so that it starts mirroring ghcr.io, and then configure Docker somehow to interpret ghcr.io/github/dependabot-update-job-proxy/dependabot-update-job-proxy as ghcr.corpnet.internal/github/dependabot-update-job-proxy/dependabot-update-job-proxy.

It is not possible to use a different caching registry to fetch these images when running Dependabot on a self-hosted action runner at this time.

It sounds like you're saying that's not possible, and maybe this is me not knowing enough about Docker. Maybe we can do the docker save/docker load trick to get them onto the self-hosted runner VM as their native names. I'll poke around.

I think you need to create a support ticket with Actions team.

Will do, thank you!

And thanks for all your help on this again!

honeyankit commented 1 year ago

Since, you have access to those VMs via ssh you can easily do docker save/docker load option.

leifwalsh commented 1 year ago

I'll give it a shot, thanks

leifwalsh commented 1 year ago

Alright, I've tried some things and have made some progress. Thank you very much for your help so far.

I was able to get the tool cache working for setup-node, when I first tried it I had a silly mistake and unpacked the tarball in the wrong location (home directory rather than inside ~/.github-runner).

I wasn't able to try docker save and docker load, because getting access to an external machine is harder than it should be for us right now. What I did instead was, using our internal caching proxy (Artifactory), I set up ghcr.internal to mirror ghcr.io, and I was able to pull the images to the dependabot VM. Getting dependabot to realize it could use those images is not possible just with docker config right now (you can only use registry-mirrors for docker hub, not other registries. @rpetrich found a patch that would presumably allow this, but we didn't try applying it).

What I did instead was I edited containers.json and image-service.ts (and Dockerfile.proxy/Dockerfile.updater, though I expect I didn't need to) in github/dependabot-action, ran npm run package, and pushed the result up to that repo and moved the ghes-3.8 tag to it. Now, dependabot is able to run the images from our caching proxy (and since the digests are the same this seems perfectly safe). I wonder if this could be a feature you could add to future versions of dependabot? Some config somewhere (ideally enterprise-wide, or per-runner, rather than in every repo's dependabot.yml) that lets users remap ghcr.io to some other domain name?

Now I'm to the point where the containers run and try to check for updates, but I think I now have a problem that the rule for updating with pip-compile doesn't correctly honor replaces-base for internal PyPI registries. I'll do more debugging on that and probably will file a fresh issue about it if I figure out what's going on.

leifwalsh commented 1 year ago

I did find the problem, it's not quite a problem with dependabot, it's this: https://github.com/jazzband/pip-tools/issues/1831

I think dependabot could patch over the problem by also setting the environment variable PIP_INDEX_URL when invoking pip-compile, but I'm also hoping pip-tools just is responsive.

leifwalsh commented 1 year ago

Ok I think I have a couple other issues to report. Let me know if these sound like I should report them as fresh issues in this repo:

  1. For a private NPM registry, dependabot.yml complains about a malformed config if you don't specify a token, and doesn't run the workflow. If you have a registry which doesn't require authentication (we do), I don't think this parameter should be required. If I set it to something, this confuses the registry, which isn't expecting auth, and requests to it fail, so the workflow does run, but errors out. (I did search around the issue backlog in this repository, particularly among the private registry and npm labels, and didn't find anything that sounded similar)
  2. This is pretty minor but weird: for a private PyPI registry that doesn't require auth, if you don't specify token, the workflow does run, but the secret mask step ends up masking the string :. This causes things like timestamps in the logs to be printed like 2006-01-02T15***04***05.000 instead of 2006-01-02T15:04:05.000.
  3. I also tried enabling this for the github-actions package-ecosystem, and dependabot tried to look for those actions on github.com, instead of on our github enterprise instance, where they actually live. I couldn't find any configuration options in dependabot.yml that I could use to tell it to query our enterprise instance instead.
honeyankit commented 1 year ago

@leifwalsh : Thank you for trying all the steps. Yes, please go-ahead and file issues.

Let me know, if you need any more help related to Dependabot on GHES else I will go-ahead and close this issue.

honeyankit commented 1 year ago

Feel free to re-open it if you have any more questions related to Dependabot on GHES. Happy to help!

leifwalsh commented 1 year ago

I filed #7290, #7291, #7292, and #7293 as follow-ups. Thanks for your help!