Investigate benchmarks

[lopcode]

JVips includes some benchmarks to do a comparison between equivalent C and JNI operations: https://github.com/criteo/JVips/blob/master/src/test/java/com/criteo/vips/benchmark/SimpleBenchmark.java

Maybe vips-ffm should do the same?

[JohannesBeranek]

I've actually got JVips running in our code and a few benchmark tests. The code won't help you much, but once it runs in our app ( #60 ) I can share some simple results

[lopcode]

Seeing results sounds great if you have the time, thanks!

[JohannesBeranek]

Some numbers ; note that these aren't scientifical by any means. S means small (40x35), M is medium (800x600). Small is interesting I think, as it shows the overhead.

Test	AWT	JVips	vips-ffm
M Resize Contain	39.8ms	21.4ms	18.6ms
M Resize Cover	44.8ms	21.8ms	13.7ms
M Gradient Resize Contain	31.1ms	20.2ms	15.3ms
M Gradient Resize Cover	30.5ms	19.8ms	12.7ms
S Resize Contain	2.9ms	11.9ms	10.8ms
S Resize Cover	7.7ms	13.6ms	10.3ms

I've a couple more in my test suite (it's for making sure the results are correct, not for benchmarking), but it's more just the same in terms of numbers. The tests also do a lot of getWidth()/getHeight() to assert stuff.

I think this shows though: vips-ffm performs better than JVips (at least in our test suite, that's ALWAYS the case), and AWT only wins for very simple stuff where the overhead dominates.

[lopcode]

Thanks for providing those, those numbers match my expectations for FFM vs JNI, which is great 🎉

The only "downside" for users appears to be a minimum JDK version of 22. But at least 23 is out now which gives folks an LTS version to target.

[lopcode]

Just to make comparison easier, the above table with % differences. Like you said, early comparisons, but still great to see as a starting point.

[lopcode]

Tried doing this on branch https://github.com/lopcode/vips-ffm/tree/task/benchmark-jvips but found out JVips doesn't support arm64 so I'll struggle to try this locally.

There's a start for benchmarks on main, but just for vips-ffm.

[lopcode]

Booted up my gaming PC and ran the project, vips-ffm works fine, but JVips fails to run the black operation with an error:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffd9882834b, pid=40576, tid=33296
#
# JRE version: OpenJDK Runtime Environment Temurin-23+37 (23.0+37) (build 23+37)
# Java VM: OpenJDK 64-Bit Server VM Temurin-23+37 (23+37, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64)
# Problematic frame:
# C  [libgobject-2.0-0.dll+0x834b]

and the problematic frame:

Current thread (0x00000220dd345df0):  JavaThread "app.photofox.vipsffm.BenchmarkTests.frameBenchmarkUsingJVips-jmh-worker-1" daemon [_thread_in_native, id=33296, stack(0x00000085f8900000,0x00000085f8a00000) (1024K)]

Stack: [0x00000085f8900000,0x00000085f8a00000],  sp=0x00000085f89fea40,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libgobject-2.0-0.dll+0x834b]  (no source info available)
C  0x0000000061e8172a  (no source info available)
C  0x000002208fcdd307  (no source info available)

The last pc belongs to native method entry point (kind = native) (printed below).
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.criteo.vips.VipsImage.blackNative(II)V+0
j  com.criteo.vips.VipsImage.black(II)Lcom/criteo/vips/VipsImage;+11
j  app.photofox.vipsffm.BenchmarkTests.frameBenchmarkUsingJVips(Lapp/photofox/vipsffm/BenchmarkTests$BenchmarkState;)V+35

Not sure what's wrong and not particularly inclined to investigate further right now.