support gzip for html games

leafo commented 8 years ago

more information here: https://cloud.google.com/storage/docs/json_api/v1/how-tos/performance

We need to have gzip ahead of time since google cloud storage can't zip on the fly. I think it makes most sense to have zipserver do the work. Another option is to push everything through cloudflare

goldfire commented 8 years ago

+1 for CloudFlare. We've used them in production for several years with no issues. We also recently did a benchmark and CloudFlare was head of the pack among CDNs (which is especially great since it is free).

leonkrause commented 7 years ago

Any news on this? Godot Engine will support WebAssembly coming next version, the binary compresses very well, shrinks from ~12MB down to ~3MB

fasterthanlime commented 7 years ago

Btw since we last posted here, CloudFlare is no longer an option (for internal reasons).

I'm sure our actual CDN has an equivalent though.

leafo commented 7 years ago

We'll have to update our zip extractor to upload the gz files and preserve the mime type. Most engines already export as pre-zipped package. Cloudflare isn't an option here since we use more traffic than what is allowed by their flat fee plans.

@eska014 does the Godot HTML export automatically generate a compressed file? Or is it depending on the server to do it?

leonkrause commented 7 years ago

@leafo Godot currently generates several uncompressed files (index.html, engine blob, game content, some JS) into a selected directory. Users are expected to zip up the files for uploading to itch.io, the server is expected to take care of compression.

We can implement an export-time toggle to generate one zip archive with gzipped files, would uploading that to an itch.io page just work? Not sure how compression works with a CDN 'in between'

fasterthanlime commented 7 years ago

@eska014 we can configure the CDN to gzip-compress some file extensions. What extension does the engine blob have?

leafo commented 7 years ago

The way unity works is that it compresses things ahead of time, creating jsgz or memgz files. In our case we aren't detecting their content type correctly, and it's set to just octet stream. Unity actually has a gzip extractor built into their html5 runtime so they extract it after downloading within the browser. I'm going to update our zip extractor to set the correct content type header, that was the original purpose of this issue.

http://leafo.net/shotsnb/2017-04-03_14-25-36.png

In the future, I think we should just automatically gzip all files we upload to our storage that are used for HTML5 games.

As for your approach, @eska014, I think doing something like what Unity does is probably the best approach. Your average user is either going to skip comprssion, or add something on their webserver that compresses during the request, which will puts unnecessary pressure on the server. Ahead of time compression seems like a good way to make sure they're serving compressed files.

leonkrause commented 7 years ago

@fasterthanlime The filename extension for the engine is .asm.js for the current asm.js-based export plus a separate .mem file for static memory. The upcoming WebAssembly format uses .wasm. The game content uses either .pck for Godot's custom binary format, or a zip archive with either a .zip or .pcz extension.

@leafo In the future, I think we should just automatically gzip all files we upload to our storage that are used for HTML5 games.

If that's possible, that would be great

Beuc commented 6 years ago

I tested a bit using emscripten and web assembly. I mainly need to serve index.js, index.wasm and index.data. On my test server I pre-compress them so I have index.js.gz, index.wasm.gz, index.data.gz.

Ideally callling xxx.ext would automatically look for pre-compressed xxx.ext.gz, leaving the original "Content-type", plus adding "Content-Encoding: gzip" so the browser transparently decodes it. This is not currently the case at itch.io's CDN, you get a 403 error.
Alternatively, you'd fiddle with emscripten's output and the program would directly call the .gz, adding a "Content-Encoding: gzip" so the browser can decompress it. This is not currently the case at ioch.io's CDN, the browser just tries to interpret the gzip'd stream without decompressing, so this fails.
Incidentally, for wasm to be efficiently loaded, it needs the "Content-type: application/wasm". itch.io's CDN returns application/octet-stream so we get an error "wasm streaming compile failed: TypeError: Response has unsupported MIME type" (which can be ignored though, just not efficient).

Currently the only way that works is upload all js/wasm/data uncompressed AND serve them uncompressed to the user. I probably could embed a JS-based gunzip but that sounds like troubles for my .data file(s) (~120M uncompressed with 50% gzip ratio), not to mention I'd have to patch emscripten's initialization process.

I can get all 3 points with a 4-line .htaccess, though I understand configuring a CDN is a more involved process (but I'm surprised this hasn't been tackled before!).

Would it be possible to implement at least point 2 ("Content-Encoding: gzip")? Otherwise what do you recommend?

fasterthanlime commented 6 years ago

(but I'm surprised this hasn't been tackled before!

I remember @leafo and I spending a good amount of time on that, trying various combinations, and eventually noticed it broke all Unity WebGL exports (with some now-deprecated but already-widely-used-for-export version of Unity. They do ship a javascript inflater as a fallback, btw).

My personal take is that:

HTTP contains all we need to serve gzip resources
- Rewriting xxx.ext to xxx.ext.gz feels wrong (we shouldn't do path mapping)
Transparently serving a gzip'd (or brotli'd) version of assets is the CDN's job
Transparently decompressing an asset is the browser's job

The problem is that some game engines have encouraged people to upload already-compressed versions of assets to their web servers, and add non-trivial configuration, so we're forced to find a solution for that scenario too, which is probably not going to be an option, but something based on automatic detection.

The situation right now (doing nothing special) works for recent Unity WebGL games, ie.:

Unity WebGL builds already-gzipped asset files
They're served directly, without Content-Encoding: gzip
Unity's runtime falls back to its own javascript decompression
- Which is a waste of CPU cycles but hey, bandwidth is saved.

This is just a drive-by update, as both leaf & I are deep into other issues at the moment, but I just wanted to reassure you that it's on our mind. Everything is, always :shipit:

Beuc commented 6 years ago

Thanks for the update! I suspected some backward compatibility issue indeed. Btw xxx.ext->www.ext.gz is not rewriting but content negotiation/MultiViews really.

leafo commented 6 years ago

our CDN is supposed to be transparently gzip compressing files for whitelisted extensions, .js is one of them. (It's not applied to everything since it was breaking unity games). But I'm checking now and not seeing it. Going to debug...

GSGBen commented 6 years ago

Hi, any update on specifying gzip encoding in the headers? UE4 outputs really small builds when using it but unfortunately can't run on itch:

Downloading failed: 
Downloaded a compressed file GT.data.jsgz without the necessary HTTP response header "Content-Encoding: gzip" specified!
Please configure gzip compression on this asset on the web server to serve gzipped assets!

JonathanADaley commented 5 years ago

I ran into the same issue today that @GSGBen ran into:

Downloading failed: Downloaded a compressed file SomeGame-HTML5-Shipping.jsgz without the necessary HTTP response header "Content-Encoding: gzip" specified! Please configure gzip compression on this asset on the web server to serve gzipped assets!

With this current project I'm okay to not serve compressed files, as it's a small project; but in the future I'll probably need to use gzipped assets. Any updates?

I'm using Unreal Engine 4.21 to make this HTML5 build, and assets are pre-compressed. But there are also uncompressed versions built out as well. And there's an option to not serve compressed versions; so there is a workaround at least.

Beuc commented 4 years ago

I opened a question for .wasm files specifically at https://itch.io/t/598164/how-to-compress-webassembly-game-code

juanitogan commented 2 years ago

Just adding links to how GitLab did it, for what it may be worth. I think this solution has already been proposed and axed for some reason but, well, like I said, for what it may be worth.

https://webd97.de/post/gitlab-pages-compression/ https://gitlab.com/gitlab-org/gitlab-foss/-/issues/15037

hilloftheking commented 1 year ago

I've noticed that all of my game's files are being transferred with gzip encoding except for the .pck file. Is there a reason for it not being on the whitelist?

itchio / itch.io

support gzip for html games #394