search

guardian / riff-raff

The Guardian's deployment platform
Apache License 2.0
267 stars 18 forks source link

S3 upload package/deployment prefix can't be overridden #446

Open sihil opened 7 years ago

sihil commented 7 years ago

The S3 upload (both standalone and also within the autoscaling type) prefixes the package when it uploads files (by default, unless disabled by setting prefixPackage: false.

This comes from the package name (aka deployment name in riff-raff.yaml). It is not possible to customise this, but there are good reasons for doing so in more complex configuration files. I suggest the behaviour should be to use the app name instead which can already be overridden in the configuration.

The example in question is this riff-raff.yaml: https://github.com/guardian/grid/blob/82c4d53afa0e0cbab1d6f3b4f361e0c102fcbc70/usage/conf/riff-raff.yaml

It winds up uploading files into media-service/TEST/upload whereas it should probably upload into media-service/TEST/usage as app is overridden.

This is an easy change to make in all places that S3Upload is used, my only concern is that this is a breaking behaviour change and it's not clear how many places would be impacted. We might need to review all riff-raff.yaml files to figure out how many files use an S3 upload task and also override app in the same deployment.

/cc @akash1810 @philwills

philwills commented 7 years ago

We might need to review all riff-raff.yaml files

😱

I don't have any other objection.

akash1810 commented 7 years ago

re reviewing riff-raff.yaml files, how would one start this? A GH search isn't that fruitful as it doesn't find any of the files 🤔 .

philwills commented 7 years ago

https://github.com/search?utf8=✓&q=org%3Aguardian+filename%3Ariff-raff.yaml&type=Code

akash1810 commented 7 years ago

😮 you can do filename searches?? Thanks @philwills. What would I do next?

lost

sihil commented 7 years ago

I would figure out how to download a copy of all of the riff-raff.yamls from GitHub (you could also do this directly from the S3 artifacts bucket as an alternative, although there are A LOT).

Then you'd need to parse the yaml (possibly grabbing a bunch of code from Magenta) and evaluate the results highlighting any autoscaling/s3upload/elasticsearch/lambda/self-deploy deploy types where deployment.name != deployment.app.