feat(postgres): allow users to configure postgres to read from S3

deis / postgres

A PostgreSQL database used by Deis Workflow.

https://deis.com

MIT License

36 stars 21 forks source link

feat(postgres): allow users to configure postgres to read from S3 #44

Closed bacongobbler closed 8 years ago

bacongobbler commented 8 years ago

If a user wishes to set up an external S3 storage endpoint, they can overwrite the following environment variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
S3_URL (something like https://s3-us-west-1.amazonaws.com)

TESTING:

build this image, modify deis-database-rc.yaml to point to this image
modify deis-database-rc.yaml, injecting the following environment variables:

env:
  - name: AWS_ACCESS_KEY_ID
    value: "myaccesskey"
  - name: AWS_SECRET_ACCESS_KEY
    value: "mysecretkey"
  - name: S3_URL
    value: "https://s3-us-west-1.amazonaws.com"
  - name: AWS_DEFAULT_REGION
    value: "us-west-1"

install Deis, and observe that a db_wal bucket was created in your S3 region with database backups.

closes #33

jchauncey commented 8 years ago

can you give me an example of how to test this?

bacongobbler commented 8 years ago

@jchauncey I haven't fleshed this out fully after thinking this through. I'll write out some documentation on how we can test this in a minute. :)

krancour commented 8 years ago

deis/charts#123 is related. It introduced a secret containing user-provided object storage connection / auth details.

bacongobbler commented 8 years ago

All righty, this should be ready to review. Follow the steps in the OP to test.

krancour commented 8 years ago

Can you maybe ask for bucket name instead and derive the URL from the region and the bucket name?

Reason I ask is because what is supposed to become the Deis-wide config for object-storage options currently seems to ask for bucket name instead of URL:

https://github.com/deis/charts/blob/master/deis-dev/tpl/objectstorage.toml#L29-L35

bacongobbler commented 8 years ago

I think I can fix this up in the future to not require S3_URL at which point boto3 will derive the URL from the region and bucket name, but I don't think I can reasonably do that right now. I can open a ticket for that post-beta though!

bacongobbler commented 8 years ago

FYI @kmala volunteered to refactor this PR on top of https://github.com/deis/charts/pull/123 so we don't have users required to configure object storage in two places

bacongobbler commented 8 years ago

closing in favour of #46