Private registry spike

[krancour]

This issue is a place for us to hammer out our approach to resolving https://github.com/deis/controller/issues/253

cc @helgi @kmala

Discussed offline earlier today:

Will prove the concept first:

• No CLI changes initially. Configuration can be set via deis config:set. This will be replaced with a well-thought-out UX and necessary CLI changes before the feature is released.
• Controller will special-case its handling of relevant environment variables set via deis config:set-- again, this is temporary.
• Controller will construct imagePullSecret from relevant app configuration and make app RCs reference the same.
• Initially, this will bypass Workflow's internal registry. Kubernetes will pull images directly from the location specified by deis builds:create / deis pull.

With the concept proven:

• Improve UX / update CLI.
• Do we want do / can we institute private, per-app repos Workflow's internal registry, each requiring authentication in order to push or pull images? This would improve platform security overall. Ultimately, then, each app/namespace would have its own imagePullSecret for the internal registry. This would not be specified by deis users, but could be auto-generated (when an app is created?). Any registry authentication details provided by the user would be used by the controller to pull images from the private registry before pushing said images to the internal registry-- i.e. similar to what we do today, and correcting for the fact that the PoC will likely bypass this step.

@helgi correct me, please, if any of this seems to differ from what we discussed.

[helgi]

The way I like to think about this is 3 phases:

1. Use config:set per app to set a Registry authentication to prove that Controller can pull from External Registry (GCR / ECR / etc) and shovel into the Deis Internal Registry, requireing no change to Kubernetes as Deis Internal Registry has no auth
2. Better UX (as discussed above) and pull from External Registry directly via k8s imagePullSecret, bypassing Deis Internal Registry
3. Make Deis Internal Registry smarter around auth (but this will come later)

[krancour]

Ok. On point 1, we differ. I was thinking we skip the "shoveling" into the internal registry, but you're thinking we keep that and the thing we skip is any imagePullSecret at all on the app pod(s). So just the controller would use the provided auth details to pull from the external private registry...

I'm fine with that. That seems pretty reasonable.

[krancour]

Actually... I am confused now about where we're hoping to land eventually.

No. 1 has us skipping the imagePullSecrets, but still caching images in the internal registry. As I indicated, I'd be fine with that.

No. 2 has us using imagePullSecrets and skipping the internal registry.

These two are opposites of one another. So what do we achieve then? Is the idea that you want to be able to support both?

[helgi]

We wouldn't support both until we hit point 3 but that gets into more configuration nightmare.

It's mostly about time and momentum really :) If we get controller talking to external registry very fast and on all levels (the original meta ticket) without issues then we can move to the imagePullSecret.

We need to figure out how the workflows will look, if we are doing pull / tag / push at all times or if we are starting to just reference the end users image in their External Registry, etc

We can't spend beta3 and beta4 on this so we are trying to time box things. crawl, walk, run :)

[helgi]

Going to close this now that #718 has been merged