search

glacials / splits-io

a speedrunning data store and analysis engine
https://splits.io
GNU Affero General Public License v3.0
133 stars 27 forks source link

Parse 100% in Lambda #490

Open glacials opened 5 years ago

glacials commented 5 years ago

We currently have an AWS Lambda job that "parses" runs on upload, but it does so by hitting a webhook in Rails that downloads the file and parses it, so still technically on the web servers.

We want to move this parsing to the Lambda job itself, which would let us budget way less memory on the Rails servers, way less disk space, would not consume CPU for parsing giant runs causing web requests to fail, etc. etc.

The Lambda job already has livesplit-core included in it correctly, it just isn't using it right now. We should make it start using it, and somehow insert the parsed data into the database, then notify Rails and/or the user that the run is ready to be loaded.

CryZe commented 5 years ago

Is that still going to use the Ruby bindings or should we use the direct Rust + Lambda support? I could look more into that: https://aws.amazon.com/de/blogs/opensource/rust-runtime-for-aws-lambda/

glacials commented 5 years ago

Undecided atm, but the current setup uses the JS bindings. I think in general I'd prefer to use a set of bindings than a language not officially supported by Lambda, though.

glacials commented 5 years ago

Oh oops I didn't click your link, I didn't realize there was a Rust runtime! I'd be willing to try it out.

CryZe commented 5 years ago

Yeah if those JS bindings are the asm.js ones (and not the node.js ones) then the native rust runtime would likely be much faster. The node.js ones are probably close.

CryZe commented 5 years ago

Oh, and apparently pricing is based on memory usage and cpu time. Both of these would be much lower with the Rust version. So this should absolutely be attempted imo.