nprapps / elections18-general

2018 midterm election back-end: Associated Press data ETL, database, admin panel, and JSON output; iteration upon 2016 GE work
MIT License
3 stars 1 forks source link

Ensure that GZIP compression is turned on for CloudFront assets #17

Closed mileswwatkins closed 6 years ago

mileswwatkins commented 6 years ago

This'll improve data-load times by clients.

mileswwatkins commented 6 years ago

And make sure that we're not GZIPping the data ahead of time ourselves!

mileswwatkins commented 6 years ago

Recently removed GZIP from liveblog code, too; see if we can ensure GZIP there too.

mileswwatkins commented 6 years ago

Tested that a GZIP compression "Behavior" will work for both legacy/existing files (that are already GZIP'd before upload to S3, and have the metadata tag to indicate this encoding), and for new files (that are stored uncompressed on S3).

Want to check this by turning on compression for all apps.npr.org assets; that's the only step left at this point. Conferring with @alykat about it.

mileswwatkins commented 6 years ago

We're testing this on apps.npr.org/dailygraphics/* starting tomorrow afternoon; Aly and I have a meeting set up to review any potential fallout.

mileswwatkins commented 6 years ago

Eads:

the gzipping for s3 is in a lot of ways a legacy feature I think it's preferable to just do that at the CDN layer

mileswwatkins commented 6 years ago

Tyler:

the only thing we saw issues with in gzipping was video (so by proxy audio is probably bad, we just haven't done it)

We should check on some audio files and video files, if they're present in s3://apps.npr.org.

alykat commented 6 years ago

A couple audio examples off the top of my head: http://apps.npr.org/lookatthis/posts/album-art/ https://apps.npr.org/music-memoriam-2013/ https://apps.npr.org/dailygraphics/graphics/march-for-our-lives-soundscapes-t-20180329/ (on this page)

With the examples I can think of that involve video, the video files don't live on apps.npr.org.

mileswwatkins commented 6 years ago

I looked through all 174k files on that bucket, and there are a couple video files in dailygraphics:

dailygraphics/graphics/cosplay2-20160721/assets/riki-lecotey.mp4
dailygraphics/graphics/sound-hovercraft-20160721/assets/riki-lecotey.mp4

And there are also video files within in a handful of projects:

audiogram-videos
austin
austin15
bats
best-songs-2015-so-far
best-songs-2015
borderland
lookatthis
naughty-or-nice
radex
rockymountain-vr

So we have a good set to test with.

mileswwatkins commented 6 years ago

Actually, it doesn't look like audio/mpeg or video/mp4 are even compressed by CloudFront! We weren't compressing them vefore, and I guess we don't need to sweat 'em now.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html#compressed-content-cloudfront-file-types

mileswwatkins commented 6 years ago

No problems with dailygraphics/*, from what we can see. We're letting everything sit for one day (just in case someone reports an issue), and then tomorrow will turn on auto-compression for everything in the entire bucket (ie, *).

mileswwatkins commented 6 years ago

Auto-GZIP turned on for all of apps.npr.org; this covers liveblogs and election JSONs and election graphics.