renovatebot / renovate

Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io
https://mend.io/renovate
GNU Affero General Public License v3.0
17.19k stars 2.25k forks source link

Possibility to use our own docker images for side containers instead of renovate ones. #6047

Closed micheelengronne closed 3 years ago

micheelengronne commented 4 years ago

What would you like Renovate to be able to do? Currently, side docker images for the binarySource: Docker are hardcoded to those made by the renovate team.

It poses a problem with closed (nearly air-gaped) networks as it maintains a link with docker-hub.

Some security policies forbid to link directly to an external registry and to use unsigned images.

Is it possible to define these images in the configuration ?

rarkins commented 4 years ago

How would you propose to do this? e.g. would you completely mirror the entire image set, or you would manually mirror them as-needed, etc?

micheelengronne commented 4 years ago

I think I would create a method that gives back the image name to use and eventually authenticate on a private registry.

It will look in the config.js if an image is configured to replace it by looking at the key in a subarray, the key being the original image name.

smthg like that in config:

dockerimages: {
  'renovate/ruby': 'ownimage/ruby'
}

Here, ownimage/ruby replaces renovate/ruby.

And a dirty skeleton of the method (just the algo, not in js):

arg='renovate/ruby'

methodskeleton(arg) {
  currentimage=arg

  if isset( config(arg) ) {
    currentimage=config(arg)
  }

  if getHost(currentimage) has an entry in hostRules {
    authenticate
  }
}
rarkins commented 4 years ago

Would you need to support multiple tags per image, e.g. to support multiple versions of Ruby, Python, Yarn, etc?

micheelengronne commented 4 years ago

For my usecase not really.

JamieMagee commented 4 years ago

Would it be an acceptable solution to add a config option, something like dockerBase or dockerUser. It would default to renovate, but could be overridden to be myuser, and we would fetch docker images like ${config.dockerBase}/ruby?

You wouldn't need to provide a full mirror of our Docker images and tags, only the ones you use.

micheelengronne commented 4 years ago

Not really for me. As I would like the possibility to completely override the images and use my own.

As a side note, AFAIK, the side images are not called with a tag and a hash. I would like to call mines with them to keep the immutability on my whole renovate install.

rarkins commented 4 years ago

If you don't need multiple versions of each tool, wouldn't an easier solution be if you had a prebuilt monolithic image assuming you can get the right tools installed?

micheelengronne commented 4 years ago

I used to do that.

The problem is that solution finally created conflicts between environments and made the whole thing harder to maintain.

viceice commented 4 years ago

@micheelengronne When we finished the buildpack stuff, you can easily build and maintain your own renovate docker image

ref: renovatebot/docker-renovate#9, renovatebot/docker-renovate#15

https://github.com/renovatebot/docker-renovate/blob/feat/buildpack/Dockerfile

micheelengronne commented 4 years ago

I already build mine. Easily. That's not an install question. The problem is that I begin to have a lot of different tests environments and their configurations start to conflict with each other.

The docker (waiting for podman :) ) way is far cleaner. It creates truly separate environments and a capacity to correctly clean them. The only thing that bother me is the fact that I canno't override the ones embedded in code.

viceice commented 4 years ago

What aboud simply build a renovate image for every environment, see out new full image as example

https://github.com/renovatebot/docker-renovate-full/blob/85d5c76c388636783930d9c82e75a4069d69cde3/Dockerfile

So you can easily add the required tool for every environment separatly

micheelengronne commented 4 years ago

I run my renovate image with a loop, making it a daemon that fetch every 10 minutes all repositories.

What you are proposing adds far more complexity. Is it complicated to add the possibility to override the current side images ? Genuine question, I did not dig enough into the code.

viceice commented 4 years ago

Currently it is not possible to override the images. It's also no easy to implement, because some side-containers have specific versioning requirements. So you would be required to fulfill those your own.

micheelengronne commented 4 years ago

Is it not possible to have the possibility to override their call in code and leave the burden of maintaining them to those (like me) that choose this option ?

viceice commented 4 years ago

As I said, it's not yet possible, maybe in future. For this feature we need to finish the docker centralized handling first. Then we can think of implementing this.

micheelengronne commented 4 years ago

ok. I will wait for it then :)

ps-jay commented 4 years ago

Would it be an acceptable solution to add a config option, something like dockerBase or dockerUser. It would default to renovate, but could be overridden to be myuser, and we would fetch docker images like ${config.dockerBase}/ruby?

For my use case, yes, it would be nice to be able to drop a prefix in front of the image names. We proxy via Artifactory.

To use renovate for example, I adjust the image name like so:

artifactory.example.net:6555/renovate/renovate:19
viceice commented 4 years ago

Idea for prefix:

https://github.com/renovatebot/renovate/blob/b806e98c6d50309131de20ce8ad77090fad4787d/lib/util/exec/docker/index.ts#L211

@rarkins @JamieMagee Other suggestions?

@micheelengronne You would then need to push matching tags to your local docker repository

micheelengronne commented 4 years ago

@viceice no problem. Does it also solve the possible auth on the local docker registry or should this registry be auth-less ?

viceice commented 4 years ago

In a first iteration it would be auth-less, we would need to support docker login and logout for this.

So we should open a different issue to support this, or you simply do the login before running renovate.

dandandy commented 4 years ago

I can work on this issue

rarkins commented 4 years ago

Please don’t start yet. Our internal handling of this is still being worked on so we’re not ready to make it configurable yet.

HonkingGoose commented 3 years ago

@rarkins What is the current status of this issue? I see that we ping-ponged around a bit with the labels from blocked to in-progress, back to blocked.

Is there something we're still waiting on? Or did we manage to get ourselves unblocked in the meantime?

rarkins commented 3 years ago

Already implemented: dockerImagePrefix