Travis CD to AWS S3 - Versioning and Old File Cleanup

[kirlat]

Currently, Travis CD configuration does not provide any way to remove obsolete files from an AWS S3 bucket. From the Travis S3 docs:

Note that deploying to S3 only adds files to your bucket, it does not remove them. If you need to remove deprecated files you can do that manually in your Amazon S3 console.

Because of this we have to establish a process that will clean older files from the S3 bucket on its own, not relying on Travis CD.

[kirlat]

Placing releases to the versioned folders (i.e. cedict-1.0.1, cedict-1.0.2, etc.) would make this simpler, I think. We can then have a script that will scan folders and read a version. Based on that it can decide whether a folder be removed or not.

We can use Amazon API Gateway to hide those versioned folders from the outside world. Here is what Even Financing is using:

We use Amazon API Gateway as a simple proxy in front of our Amazon S3 bucket. We set up our API to have 3 main resources: /snapshot/{s3path+} /v/{s3path+} /{s3path+}

All of the resources are configured as HTTP proxies, with s3path mapped from method.request.path.s3path. /v, and /snapshot point to their corresponding https://s3.amazonaws.com/<bucket name>/<snapshot|version>/{s3path}. The remaining non-versioned resource, points to https://s3.amazonaws.com/<bucket name>/{version}/{s3path}, with version mapped from stageVariables.version.

The piece above is taken from Continuous Release Pipeline with Travis CI.

@balmas, what do you think?

[balmas]

In general, I am okay with using versioned folders, but I would like to avoid introducing another AWS dependency if we can help it (as we pay for API gateway usage)

Since the buckets are already served by a cloudfront distribution, perhaps we could figure out how to handle pointing at the right versioned folder via the cloudfront configuration?

[balmas]

here's the cloudfront terraform file https://github.com/alpheios-project/terraform/blob/master/cloudfront_lexis.tf and documentation https://www.terraform.io/docs/providers/aws/r/cloudfront_distribution.html https://docs.aws.amazon.com/cloudfront/index.html

[kirlat]

Thanks, I will check it out