S3 Upload

[samber]

Hi guys !

What about a parameter uploading to/restoring from AWS S3, instead of -file ?

[aaronhurt]

I can definitely see the benefit there. I'll give it a look in the next few days.

[samber]

Thanks a lot @leprechau ! :+1:

[aaronhurt]

@samber Testing this locally and I had a question ... Would you like to directly push the blobs to S3 or have it generate the local file and then push that to S3 afterwards?

[samber]

I've heard some people having more than 1Go of KV in consul, so pushing directly would be awesome ! Unfortunately, if it requires encryption, you will not be able to stream data into s3, because we have to put the whole KV store in a -very large- buffer. I think storing data in a file would be the best solution.

I think we can use some -s3-region -s3-path -aws-access-key -aws-secret-key flags such as:

• consul-backinator backup -addr $CONSUL_ADDR:8500 -dc eu-central-1 -file /backup/consul ➡️ only generate a file
• consul-backinator backup -addr $CONSUL_ADDR:8500 -dc eu-central-1 -s3-region eu-central-1 -s3-path s3://my-bucket/backup/consul/latest -aws-access-key $AWS_ACCESS_KEY -aws-secret-key $AWS_SECRET_KEY ➡️ only push to s3
• consul-backinator backup -addr $CONSUL_ADDR:8500 -dc eu-central-1 -file /backup/consul -s3-region eu-central-1 -s3-path s3://my-bucket/backup/consul/latest -aws-access-key $AWS_ACCESS_KEY -aws-secret-key $AWS_SECRET_KEY ➡️ doing both

WDYT ?

[aaronhurt]

I've seen a few places that are using URIs to capture all of the S3 data in combination with environment variables. Since we are already allowing plenty of environment variables for the Consul portions I like the idea of expanding that to S3 as well. This is my current implementation locally:

Everything in the URI...

consul-backinator backup -s3 "s3://access-key:secret-key@my-bucket/consul/latest?region=us-east-1"

A simple URI with exports...

export AWS_ACCESS_KEY_ID=my-access-key
export AWS_SECRET_ACCESS_KEY=my-secret-key
export AWS_REGION=us-east-1
consul-backinator backup -s3 "s3://my-bucket/consul/latest"

This only adds a single option to the list for the various commands to keep the help listing a bit more sane and keeps with the current implementation of putting most common things in environment variables.

Would this work for you?

[aaronhurt]

I changed my mind about the previous post. I've actually not added any options but instead used the existing -file option to specify either a local file OR a remote S3 location.

With CONSUL_HTTP_ADDR, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_REGION exported ...

2016/11/08 16:17:31 [Success] Backed up 16 keys from consul.local:8500/apps/hmm/ to s3://ena-testing/consul/latest
Keep your backup and signature files in a safe place.
You will need both to restore your data.

charlie:consul-backinator ahurt$ ./consul-backinator restore -file="s3://ena-testing/consul/latest" -prefix=/apps/hmm/
2016/11/08 16:17:38 [Success] Restored 16 keys from s3://ena-testing/consul/latest?region=us-east-1 to consul.local:8500/apps/hmm/

[aaronhurt]

@samber The working code is committed to master. Please test it and let me know what you think.

[samber]

Ok :+1:

I agree with everything. Thanks for this feature ! ;-)

[aaronhurt]

Awesome, think I'll push a new release that contains these changes and the TLS customizations from #16 and #18