Configurable endpoint for S3 object storage

[Cryptophobia]

From @markkwasnick on September 1, 2016 16:43

Deis currently supports Minio (S3-compatible API) for object storage only within the cluster and not for production deployments. Deis also supports S3 for object storage, but not with a configurable "regionendpoint", requiring the use of specific Amazon AWS regions.

Request is to modify Deis to accept a s3.region_endpoint parameter within the generate_params.toml file on the deis/charts project, which will be stored as a Kubernetes secret like the other parts of configuration. This will allow for operators to configure Deis Object Storage to use hosted S3-compatible deployments outside of the cluster.

(Related) Is a change to the deis/object-storage-cli to pass in the configurable endpoint above to the s3 driver via the optional "regionendpoint" param.

(per doc: https://github.com/docker/distribution/blob/master/docs/storage-drivers/s3.md)

Copied from original issue: deis/workflow#481

[Cryptophobia]

From @markkwasnick on September 7, 2016 0:50

Adding a note here: Probably also want to include an option to toggle secure yes/no for the docker storage driver.

[Cryptophobia]

From @bacongobbler on March 27, 2017 22:7

If anyone has a good doc on getting this set up, that would be appreciated. We can fizzle that down into a consumable document for other users.

[kingdonb]

Notes from when I started a solution for this:

https://gist.github.com/kingdonb/845945908336cbeefb3e8cce73ec5dc2

I had started a pull request (below), but most everything is in the gist above.

https://github.com/deis/workflow/pull/857

[kingdonb]

There's enough in that Gist right now that if you have an external Minio endpoint (with or without persistence) and some credentials, you can masquerade and populate them into your variables DEIS_MINIO_SERVICE_HOST and _PORT in a couple of places, set your access keys and secret key in the template, and get off and running.

For my personal (minikube / home cluster on a very minimal system requirements) this is enough to get a database and controller off the ground pretty reliably, whereas if I leave the defaults and don't set any external object storage, with my slow disk that has to do everything, the thing tends to go into swap and builder will fall over pretty easily (that is if you can even get it started.)

My modest machine cannot really handle Database and Controller and (in-memory) Storage all at the same time. Builder also really would prefer an SSD, I think. Lots of my Deis knowledge centers around how to get it to run on a fairly modest system without actually becoming totally unusable.

But any Deis (/Hephy) guide for newbies should absolutely spend some time on setting up Object Storage properly (and the different ways that you can do that, because why not...)

You probably want deis postgres if you need postgres and don't have something like RDS stood up already that you can use

Other than getting the MINIO variables in place, all you really need to know to make it work is that the accesskey and secretkey you generated when you started Minio externally (however you did that) should be put in your chart at workflow/templates/objectstorage-secret.yaml as base64 encoded strings, and kept secret

(We can certainly do better than that, but we could also build a bikeshed...)

[kingdonb]

actually, the Gist only "almost works"

Slugbuilder can't find S3HOST and I don't see a deployment resource for it to add MINIO vars to, either...

Builder is of course dynamically starting up the slugbuild, not through a deployment resource, and

builder]$ kd logs slugbuild-owlish-hawthorn-44438bde-f8a03655
2018/03/25 23:10:42 missing s3 host

[kingdonb]

Yep not exactly stable without the deis-minio service in place

I stood up a "socat" listener to forward to my remote minio service, and that is not apparently exactly stable either. I'm not currently sure how to go about proxying minio but I figure I might have better luck with something that is intended to do multithreading and proxy http, like nginx.

It would maybe be better to just rewrite slugbuilder and slugrunner so they accept a "region endpoint" setting that is passed in somehow, and so would everything else in {builder,database,controller,registry} set that depends on the presence of the deis-minio service when the minio storage config is in use.

[Cryptophobia]

Great work on this!

[kingdonb]

Super! I'm catching up, just saw teamhephy/distribution#1

Timing couldn't be better! Thanks @rwos