Benchmarks

[gingerlime]

Hi again :)

I'm trying to run some simple benchmarks to compare thumbor performance to imagor. For some reason I'm not sure the VIPS_CONCURRENCY_LEVEL env has an effect? It seems like it's always using all available CPUs ... Also interesting is that whilst it seems to run faster compared it to a single-process thumbor, when I run it with concurrency of 4 against thumbor (with the same concurrency), thumbor out-performs imagor (without caching, only keeping the original image in file storage) ...

Perhaps it's due to the way I've setup the benchmark? maybe it doesn't keep the original image in storage and always fetches it? or some other trivial misconfiguration? because otherwise I would expect libvips to outperform thumbor ...

You can see the benchmark code I'm running at https://github.com/MinimalCompact/thumbor/tree/imagor-thumbor-benchmark/benchmarks/locust (it's based on an older benchmark I created for optimizing thumbor on a multi-proc environment, so it's a bit messy unfortunately, but hope it's still useful)

I tested it on a DigitalOcean droplet with 4 cpus. Here's a sheet with the stats

[gingerlime]

@cshum @jcupitt thank you! I also ran a benchmark and see no more errors 🍾

[jcupitt]

Great!

That's an impressive speedup --- of course I'm now nervous that it's too good. Do you see similar gains in production use? Or is there something funny going on in this test?

[cshum]

I have deployed imagor to production from day one, and have been seeing significant improvement in response time from the user point of view, not just for the benchmark sake. It also reduces memory consumption by half.

Our production environment mainly oriented around simple image operations such as fit-in, resize, trim, fill background. But we work with lots of images, like millions of product images, mostly in jpeg. So the benchmark environment does differ from what we do in production. However, replicating such environment equivalent in benchmark script would be overwhelmingly difficult. You will have to prepare millions of distinct, non-copyrighted images, then run and measure against all of them.

The initiative of the imagor project is to tackle thumbor performance from ground up. This is because the uncached response time of thumbor could hardly keep up with our user acceptance. We used to restart the thumbor instances like every few hours, in order to keep things intact.

Though I am no expert in optimising python web services, I do sincerely hope that some of the situations will be improved in their 7.0.0 release https://github.com/MinimalCompact/thumbor/issues/67