Investigate assets.mozilla.net Automation

bensternthal commented 4 years ago

Copied over from milestone board.

glogiotatidis commented 4 years ago

My suggestion is to use Git LFS and store assets in a GitHub repo. On push, a GitHub Action would sync with a S3 bucket. Access would be managed on GitHub and no AWS credentials would be handed out to developers.

Details:

GitHub supports LFS.
Unlimited file size: I couldn't find a official statement on file size limitations when using LFS, but this Stack Overflow answer claims that there's none. I tested with current files in assets and all of them got uploaded without issues.
After the initial setup (see below), Git LFS works transparently once you have the git-lfs binary in your path. Get the binary from the official page
A GitHub action that does S3 syncing with git-lfs support will be triggered on every push on master. We can fork https://github.com/marketplace/actions/s3-sync to make it support git-lfs.

Initial Setup

git clone git@github.com:mozilla/repo/with/assets
git lfs install
mkdir assets
git lfs track "assets/*"
git add .gitattributes
git commit -m "Track assets/ with git lfs"

To add a file

dd if=/dev/urandom of=assets/50mb count=50 bs=1M
git add assets/50mb
git commit -m "Add 50mb file"
git push origin master

Thoughts @bensternthal @pmac @mozmeao/www ?

pmac commented 4 years ago

That sounds way easier than I thought! Does our GitHub plan support LFS for the mozmeao org or do we have to upgrade?

glogiotatidis commented 4 years ago

I should. I tried with a test org I run and it just worked.

pmac commented 4 years ago

Awesome. I guess my only other question then is if we should do this in the bedrock repo and just upload large files to our normal media s3 bucket, or keep it separate? I vote separate just because bedrock is already so large and we might use assets.m.o for other sites.

glogiotatidis commented 4 years ago

I agree on having separate repos.

glogiotatidis commented 4 years ago

Created assets.mozilla.com repository
Created asset-uploader AWS user and assigned the existing groups asset-uploaders
Created AWS access key for asset-uploader
Updated policy for group asset-uploader
Added GitHub Action to sync to s3

Open questions

[x] ~~Assets are actually hosted under assets.moz.works. Do we want to add assets.mozilla.com?~~ It's assets.mozilla.net
[x] Give write permissions to the right people. Gave access to @mozmeao/www
[x] ~~Make repo public?~~ Keep it private to avoid downloads from this repo which counts against our bandwidth quota
[x] buy git lfs package.
[x] disable forking to manage LFS usage (storage and bandwidth is shared with all forks)
[x] Disable versioning in s3 bucket.
[x] Notify www team in #www

mozmeao / infra