Tracking Issue for IIP Performance Statistics

[beaudet]

This issue can be appended with various performance tests being conducted against the IIP server. Perhaps the following format can be used:

Date of Test:

Server Description (cores, memory, cpu model, etc.):

Tool used for performance testing and / or command line:

Data file description / sample of data file contents

Testing Procedure (for replication):

IIP release or branch build & date built + configuration, e.g. Kakadu, OpenJPEG, PNG, etc.:

Results:

[beaudet]

Date: 5/12/2017 Server: 2.3 GHz Xeon x 8 cores, 24 GB RAM / Memcache 6GB Cache / Varnish 4GB malloc cache / proxy to varnish for all requests not received from localhost Tool: siege with 8 concurrent users and no delay, 900 requests total per user with file containing 900 unique URLs, each retrieving a 400x400 JPG from a wide variety of art object images, some of which are as large as 30k pixels in the longest dimension, but most of which are between 3000 and 4000 pixels in the longest dimension. Pyramidal TIFF sources generated with vips im_vips2tiff $targetFile $sourcefile$fileExt:jpeg:90,tile:256x256,pyramid

Example Line from Data File: http://server-name/iiif/public/objects/5/0/7/9/1/50791-primary-0-nativeres.ptif/full/400,/0/default.png

IIP Build: Custom build from github.com/beaudet/iipsrv/nga_prod branch incorporating IIIF url support, lanczos filter, caching enhancements, png output support, icc profile enhancement, and oversampling enhancements

IIP Config: FcgidInitialEnv MEMCACHED_SERVERS "localhost:11211" FcgidInitialEnv MEMCACHED_TIMEOUT 300 FcgidInitialEnv ALLOW_UPSCALING 0 FcgidInitialEnv LOGFILE "/var/log/iipsrv/iipsrv.log" FcgidInitialEnv VERBOSITY 1 FcgidInitialEnv FILESYSTEM_PREFIX "/my/secret/image/prefix/" FcgidInitialEnv JPEG_QUALITY 90 FcgidInitialEnv MAX_CVT 8192 FcgidInitialEnv MAX_HEADERS_IN_METADATA_CACHE 10000 FcgidInitialEnv MAX_IMAGE_CACHE_SIZE 0 FcgidInitialEnv DISABLE_PRIMARY_MEMCACHE 0 FcgidInitialEnv INTERPOLATION 2 <-- Lanczos FcgidInitialEnv OVERSAMPLING_FACTOR 1.5 FcgidInitialEnv IIIF_PREFIX "/iiif" FcgidInitialEnv RETAIN_SOURCE_ICC_PROFILE 1

Testing Procedure:

1. on image server: service httpd restart; service memcached restart; service varnish restart;
2. on load generating server: siege -c 8 -d 0 -r 900 -f jpg400x400.txt
3. repeat siege test to compare memcache performance against varnish cache MALLOC performance

Results:

• IIP producing 400x400 jpgs at a rate of 371 per second on the initial run after restart at near 100% cpu
• Repeat run after varnish has cached the results produces 907 images / second at 9% cpu load
• Disable proxying to varnish and deferring to IIP's memcache produces 890 images / second at 9% cpu load

Conclusions:

• IIP is VERY fast at serving "reasonably" sized tiles - in this case about 40% as fast as one of the fastest caches available although I've seen other tests approaching 50% as fast; however, this is primarily due to the same URL being called by multiple clients and once cached in memcache, all subsequent clients are able to take advantage of the fact that the image response already exists. I will rerun this test and document separately with all caching disabled in order to assess the impact on performance of multiple users vs. a single user making sequential requests.
• Using mod_rewrite's [P] to proxy requests to varnish quickly exhausts sockets on image server after a few performance tests - using mod_proxy instead should help since it re-uses existing connections rather than establishing a new socket every time but it's not clear how to create a non-mod_rewrite based rule for mod_proxy. Therefore, unless I plan to create a disk based image cache greater than 6 GB, it seems better to just jettison varnish entirely and rely on memcached instead. It's too bad varnish doesn't provide an option for HTTPS, otherwise I would just front Apache with varnish and let varnish handle all of the initial requests. Actually, it appears that

despite this https://www.varnish-cache.org/docs/trunk/phk/ssl_again.html

there might actually be hope here: https://www.varnish-software.com/plus/ssl-tls-support/ and therefore here https://github.com/varnish/hitch

[beaudet]

I reran the May 12 test listed above with varnish and the IIP primary cache as well as the image tile cache disabled and with 8 users as well as 1 user. I then enabled tile caching only to assess the effectiveness of that option alone.

Results and Conclusions:

• With 8 users and all but per-process image metadata caching disabled, IIP produced 400x400 jpgs at a rate of 82 per second with 8 simultaneous users and 100% cpu utilization. With a single user, this rate dropped to 32 images per second, showing that while IIP is fairly efficient at making use of all available processing cores for certain operations, the fact that only 25% of the overall CPU was used on average during the test suggests there's probably more opportunity for multi-threading.
• With only tile caching enabled, the rate increased to 93 images / second with 8 users and 36 images per second with 1 user, demonstrating that the bulk of caching performance is gained at the full response caching level rather than the tile caching level, although this will depend on the specific uses and loading conditions in practice.
• The initial conclusion I posted on May 12 saying that cached results are only 2x as fast as native IIP calls is a bit inaccurate since 7 out of 8 calls in that test were actually being served from IIP's primary cache. Rather, there is actually an 11x speedup using cache of either varnish or IIP memcached variety compared to completely uncached calls. It appears that tile caching alone only contributes marginally to the speed of IIP with the most significant boost coming from response caching. Even at 11x slower than a fully cached response, I would say that IIP is still quite fast considering the operations and I/O it is performing.

[beaudet]

Ruven has indicated that native Intel Lanczos code is being successfully integrated into IIP and provides an additional 7-8x speedup w.r.t. the resampling operations. That processing is perhaps 40% of the total time handling a non-cached requests, so I would expect it to speed up IIP by a factor of perhaps 2x once that's in place.