Move mission photos (ie MEDIA references) onto S3

[jaylett]

This means they need thumbnailing and uploading before adding to MEDIA. Probably needs some discussion about who gets the appropriate S3 credentials.

[russss]

We should also change the scheme we name the images with at the same time.

We should use original names where possible. All the onboard imagery has a mission-roll-frame number (e.g. AS13-59-8491) which uniquely identifies it. We should keep these names so we can easily trace the images back to their original source, and also in case we want to move an image between timestamps without renaming it.

[russss]

I'd like to say we should probably do this before we add any more missions, to avoid git bloat and more hassle uploading stuff.

A little (passworded) web app which lets you upload files to S3 and spits out the relevant json to paste into the MEDIA file would be handy, I think.

[andrewgodwin]

So, we discussed this at the little meetup, and it seems there's two issues here: separating the MEDIA bloat from the main repo, and adding versioning so that new images correctly avoid the cache.

My proposal is thus:

• Move each mission into its own git repo, under the Spacelog user (so, for example, Spacelog/a13). This avoids main repo bloat without locking people out from running stuff locally like just storing it on S3 would.
• Link to images/other things in the mission directory using the top git revision ID of that mission's repository as part of the URL. Thus, when something is updated, things correctly cache-bust, unlike if we, say, took the hash/mtime of the _meta file (we can update the images without updating the _meta)

I think this is a better solution; we can just have the server clone the mission repos directly. We need to have stuff in a repo anyway (because it's just more sane that way), and if we have separate repos we can just use the intrinsic nature of commit IDs as versioning. We can also then just keep using CloudFront and not bother with S3.

[russss]

OK, firstly, let me explicitly limit the scope of this ticket to the mission photos from external sources, and not anything we might create such as phase diagrams.

Are we ever going to change these images? In the unlikely event that we do change them, is it going to be such an important change that we want to keep track of that change and circumvent cache expiry?

I think the answer is no, and that's my reasoning behind explicitly not wanting to keep these in version control. This is externally-provided, immutable content like the scan PNGs. If it's never changing, there's no point in keeping it in version control.

I can imagine us having several hundred or even thousands of photos for some of the later Apollo missions (obviously not displaying them all the same way as we display photos currently). Having a 1GB+ git repository for a single mission doesn't seem like the best way of handling them, especially if you need to clone it to make a single-line transcript change.

I also think having one git repository per mission will get unwieldy even considering our current feasible scope of 28 missions (11 Apollo, 10 Gemini, 6 Mercury and ASTP), not counting the possibility of Shuttle or Soyuz missions.

[jaylett]

So we're talking about:

• MEDIA references use a new MEDIA_BASE_URL, defaulting into the normal static-serve stuff and overridden to S3 or S3-via-CloudFront in website/configs/live/settings.py
• global doesn't have to change at all
• we need a documented naming convention (in README.md)
• we need to rename all the existing media items (I believe Russ has started, or perhaps is just naming them appropriately from now on)
• and new media items need to be uploaded before the _meta gets committed & a reindex done (which needs documenting and maybe as Russ suggests a micro app building to support it)

The only downside, and it is annoying, is that it now requires internet access to fully test changes to the site (testing adding new media can be done without problem, but a change that broke links to external media might not be spotted). We not going to get 500 breakage because of this, but we will need to be alert to pushing changes live without testing them beforehand, either with a staging system or on someone's local setup with full internet access.

We also want to support truly external media (to avoid having to host video, etc. if it's already well-hosted). There's a separate ticket for that, but I think it's worth noting here.

[andrewgodwin]

OK, if it's just the media images, and not the rest of the mission assets, I'm happier with that not being version-controlled (though it would be nice to provide a way to mirror them all locally so you can switch MEDIA_BASE_URL).

I presume by renaming we mean prefixing things with mission names?

[jaylett]

No, naming is about using original identifiers rather than timestamps of where we've used them, as per the first comment (https://github.com/Spacelog/Spacelog/issues#issue/104/comment/592922).

[russss]

OK, this is now done. There is a s3-upload.py in tools which will upload files to S3, although only I can currently use it because only I have the credentials...

I've left A13 and MA6 file naming as is, if we want to change it later, we can. Apollo 11 imagery is now incoming.