Benchmarks

[LegNeato]

We should benchmark against the "raw" go runtime as well as other language runtimes.

[rocallahan]

It would be interesting to try to replicate the results here https://medium.com/@nathan.malishev/lambda-cold-starts-language-comparison-%EF%B8%8F-a4f4b5f16a62 but including Rust.

[softprops]

Let me know if anyone is planning on jumping on this. I was also planning on doing this for https://github.com/ilianaw/rust-crowbar curious to see if it could make the runtime with the lowest cold start time, python, dip down even lower. I'm really curious to see for myself :)

[euank]

I'm planning to do a comparison between this, the default go runtime, rust-crowbar, and the python runtime.

I won't be surprised to get drastically different numbers than the above medium post since I think the cold-start time is probably dwarfed by the zip download+unzip time, which is largely a function of code size... rust-crowbar, go, and rust-aws-lambda all have rather large binary sizes compared to standard python/nodejs hello worlds, so it's quite plausible crowbar and this will have similar performance for cold start since the binary sizes will be similar.

The preceding paragraph is purely speculation, and I'll be happy to be proven wrong once I do get the numbers :)

[LegNeato]

Sadly it looks like the glibc is too old on Amazon Linux so we can't use docker to build a smaller non-musl binary. I'm still looking at it though.

That being said, with --release, --lto, strip, and the system allocator the binary could get decently small.

[euank]

At this point I've got a bunch of numbers and even a graph!

I'm putting it together in a short blog post which is pending just a bit more wordsmithing.

The tl;dr though is this methodology for benchmarking "hello world" written 4 different ways gives me this graph:

Without doing any sort of binary-size-golfing or trying to do anything else fancy, things already seem to be fast! Significantly faster than Go even!

I'll link my post, which will be a few paragraphs explaining the above methodology and charts a little more, within the next couple days.

[rocallahan]

Can you explain why crowbar is faster than rust-aws-lambda in the warm case? Because that makes no sense to me.

[srijs]

Kudos @euank, it’s great to see some first numbers comparing the different lambda approaches!

Once warmed up, the python runtime needs to do less work to invoke the handler, as the rust-aws-lambda runtime needs to read messages off a socket, then deserialize gob data, and then deserialize json data.

When we’re just looking at “Hello World” performance, it makes sense to me that python (and crowbar) would have an edge over go (and rust-aws-lambda).

So this probably paints a good picture for ‘startup overhead’, but I’d be interested in seeing how these numbers look like for a workload that’s closer to what a lambda function does in the real world, maybe accept an event, process it, and then call an AWS API such as DynamoDB or S3.

For a scenario like that, rust-aws-lambda can likely offer a few benefits such as the tokio runtime, as well as multithreading.

[rocallahan]

I see. Is there documentation somewhere of how Python communicates with the rest of the AWS infrastructure? I assumed it would have to do something similar to the Go protocol.

[euank]

@rocallahan The Go Lambda stuff is available under a permissive Apache 2 license, so it's fairly easy to look at that code, reverse engineer how it works, and so on.

The Lambda Python runtime is, to the best of my knowledge, proprietary AWS code.

Sure, if you wanted to reverse engineer the (probably C) code in /var/runtime/liblambdaipc.so you could, but you couldn't actually publish it anywhere since it's copyright AWS. If you want to poke around at what files are involved in the runtime more easily you can do so with docker run -w /var/runtime/ -it lambci/lambda:build-python3.6 bash, but again, it's proprietary code.

The python runtime is a blackbox and we can't make too many assumptions about it.

There's also the possibility that the Lambda service for some reason has different control-plane deployments/code-paths for handling different runtime's events with mildly different performance characteristics. It's unlikely, but possible.

The XRay traces don't give us more granularity than "Lambda black-box operations" and "your code doing stuff we think?", so it's difficult to know exactly what introduces the difference and whether it's something we can influence at all.

[euank]

@srijs

So this probably paints a good picture for ‘startup overhead’, but I’d be interested in seeing how these numbers look like for a workload that’s closer to what a lambda function does in the real world

That's trickier to measure than little toy benchmarks, but I also think that most or all the advantages this project has in those cases would also transfer to crowbar.

Maybe rust-cpython has issues with threading or such, but barring more information it seems fair to assume rust running under crowbar won't be that different after the handler stuff is out of the way.

I don't think half a dozen milliseconds are that big a deal once you start doing real work though, and the difference very well might entirely vanish in the noise of a single AWS API call.

[srijs]

There’s smaller differences like deserializing events into serde_json::Value rather than specialized structs, or crowbar::LambdaContext holding a reference to the python runtime, but truthfully none of these are not fixable.

I suppose the point I was trying to make is not that you can’t have these in crowbar, but that by the time you’re doing them in crowbar, it might start to dwarf the initial headstart that crowbar has in the hello world benchmarks.

Looking beyond “crowbar vs this project”, I feel a general “here’s how Rust compares to other languages in lambda” would probably be interesting and benefit from a more real-world benchmark where rust can shine with its blazing performance.

Looking forward to reading your blogpost! 👍

[LegNeato]

@euank thanks for the great work! It might be useful to try with LTO enabled as well for the rust projects.

[softprops]

wow this is super cool @euank ! thanks for posting your findings.

Here's another angle I was kicking around to put in a separate post re: rust in relation to other runtimes, and its not related to speed.

It's related to reliability and correctness. Something that rust brings to the table that is potentially more interesting sell. since all runtimes are billed at 100ms intervals the differences are almost unnoticeable so that to me as a game of golf. The real cost comes in with the cost of the triggers in the case of api gateway the cost of that dwarfs the cost of lambda execution. Theres a different kind of cost though. Reliability, I think that's the thing that makes rust really shine ( being on par coldstart / performance wise is sprinkles on the cake for me ). Being able to build reliability into products is key for business, and I feel rust has the highest potential for the future there :) There's something to be said for not being able to call a method on undefined ( looking at you node ) and calling methods on None ( looking at you python ) and calling methods on null ( looking at you java ) and calling methods on nil ( looking at you go and Im well aware that this can be made to be safer but thats weird! ) Rust on the other hand brings a lot to the table in terms of reliability where other runtimes just can't make the same guarantees. To me that's the real sell here.

[euank]

I wrote some words around the above graph here. I also suggested additional paths to explore, namely "normal" sized rust functions and handler inputs.

I might do one additional round of benchmarks with a "normal" function that takes a few kb json object in, does a database query and an s3 api call, and gives back a few kb of json as well. That seems like it would be somewhat representative, and it would pull in an s3 client + postgres client which would get us to a normal deployment zip size I think. To be clear though, that's a might; it's not something I'm actively starting on now.

I'm not sure what the actual resolution of this issue is though.

@LegNeato, which of the following do you see as closing this issue:

1. The numbers above, gathered at one point in time for a trivial function
2. A real-world benchmark, as described above, but again living as a point-in-time thing outside of this repo
3. Benchmark code maintained in this repo / by this project and run on each release.
4. Benchmark code run on each PR.

To me, 1 and 3 seem like the most reasonable resolutions. For 3, I do think the code I hacked out isn't a bad place to start, though I could also understand wanting to drive it all from rust instead of bash+terraform.

[softprops]

Very write up @euank. Thank you for the effort on this. I realize it's hard enough to put together and experiment to collect data let alone write a detailed post about it. This was very great work!

[dnagir]

This is my very non-scientific benchmark by just manually clicking "Test" lambda and watching the duration numbers (simple "Hello lambda" code).

Rust (rust-aws-lambda)

• first invocation - 6ms
• hot - ~0.4ms - fairly consistently in the range 0.3-0.6ms with occasional jumps into 5ms and sometimes 10 area
• min - ~0.3ms
• max - 15ms

Go (built-in AWS)

• first invocation - 12ms
• hot - ~5ms (hard to get good average as the numbers are anywhere between 0.5 - 20)
• min - ~0.4ms
• max - 20ms

---

Overall, Rust looks a fair bit more stable and consistent as well as a touch faster. So the results look promising 👏

Thank you very muc for your work on this and hope it'll be well supported and used in the future ❤️

[coder543]

Amazon has announced official support for Rust on Lambda, available as a result of this other announcement. So, I guess the war between Rust on Lambda hacks has probably come to a rapid conclusion, but it would be interesting to see how the performance of this new runtime compares to the existing ones.

[euank]

@coder543 I noticed that as well. I plan to get some new numbers, including with something a little less artificial... It is possible the "custom runtime" has worse characteristics than the go-based one in this repo.

I don't expect to get to it until at least 2 weeks from now, but I do expect to re-do my comparison before too long :)

[dnagir]

AWD just released official support for lambda

https://aws.amazon.com/blogs/opensource/rust-runtime-for-aws-lambda/

[srijs]

As we're deprecating this project in favour of https://github.com/awslabs/aws-lambda-rust-runtime, I'm going to close this issue.