search

datawire / forge

Define and run multi-container apps in Kubernetes
http://forge.sh
Apache License 2.0
416 stars 43 forks source link

Different container registries in dev and prod #146

Closed majelbstoat closed 6 years ago

majelbstoat commented 6 years ago

In development, I'd like to use a local container registry for the location of images (see https://github.com/datawire/forge/issues/145), but in CI deployments, I need to use GCR. Currently, forge.yaml appears to support specification of only one registry.

This could be supported in a few ways:

rhs commented 6 years ago

FYI, there actually already is a --config flag and a FORGE_CONFIG environment variable that override the forge.yaml.

Personally I'm not sure it's ideal for this case though. My goal is that forge deploy should pretty much always do the right (and safe) thing based on the current git branch. Having to remember which form of the forge command to invoke would work against that.

As another FYI, there is no need to actually run forge setup, you can simply manually create a forge.yaml file for CI, the full format is documented here: https://forge.sh/docs/reference/forge-config

I'm also happy to create a non interactive option for forge setup, but I wonder if bypassing it all together might be better? The original intended use case for forge setup was to help users bootstrap through the fussy/brittle registry configuration process. I'm thinking now that gcloud integration is done, with a bit more auth work I might be able to remove all secrets from forge.yaml entirely, in which case a reasonable option for CI might be to simply check in an official CI version of forge.yaml and pass that explicitly to forge when invoked in CI, e.g.: forge --config ci/forge.yaml deploy or something like that.

What are your thoughts? I'm mostly thinking out loud here. ;-)

majelbstoat commented 6 years ago

I'm thinking now that gcloud integration is done, with a bit more auth work I might be able to remove all secrets from forge.yaml entirely, in which case a reasonable option for CI might be to simply check in an official CI version of forge.yaml and pass that explicitly to forge when invoked in CI, e.g.: forge --config ci/forge.yaml deploy or something like that.

Yeah, that would be neat. I would love to check in a standard file and have it Do The Right Thing.

At the moment, I've managed to work around this by creating a common dev image which include a VOLUME directive, and pushing that to GCR. Those volumes are then bound to paths in my monorepo for hot reloading etc. This might actually end up being better anyway, so there's a common image that can be updated predictably. (https://github.com/datawire/forge/issues/145) is still valuable for testing those changes before they are pushed to the container repo, and used in combination with your suggestion, would allow me to:

1) Test a new dev container image from the local repo. 2) Push that dev image to GCR or wherever once it's known to be good. 3) forge deploy dev from that new image from GCR.

I do still need to do some gymnastics to use the right Dockerfile, because they aren't templatisable (https://github.com/datawire/forge/issues/144), but the default can be dev, and then in CI, I just overwrite it with a prod one.

I am also thinking out loud :)

rhs commented 6 years ago

I just released forge 0.4.2 which includes part of what is discussed here. The docker user and password are now optional, so it should be possible to create completely secret free forge.yaml files and check them into source control.

I hope to finish up this and #145 early next week.

rhs commented 6 years ago

I've released forge 0.4.3 with a forge.yaml that supports profiles. I believe that should complete all the items discussed in this issue.

As an aside, I'm intrigued by your comments around the VOLUME directive. We have a couple of vaguely similar sounding schemes geared towards enabling fast incremental container builds and/or live reloading of stuff running in a container. I'm kinda curious about the details of what you ended up with. I'd really like to provide a more general solution to this class of problem. Would it be possible for you to share a working project skeleton?

majelbstoat commented 6 years ago

Awesome, thanks for all your work on this, can't wait to try it! Happy to help with a reference implementation. Focused on non-build-related things for the next week or so, but will try and put together a simple example soon after that.