OSPray as a benchmark?

[michaellarabel]

Hi, I was looking again at OSPray to see its viability as a benchmark. When looking some months ago I thought I might have seen some reference to any options for OSPray for serving as a benchmark for comparing system/component performance, but I either lost track of it or misremembered.

Is there any recommended means of using OSPray for performance testing? I know there is the demos page with various samples that can be tested, but any existing plumbing (or plans) within the ospExampleViewer component to expose any performance measurements? Thanks for any guidance.

[michaellarabel]

Nevermind, sorry about the noise... Somehow yesterday my searches weren't turning up ospBenchmark.

[jeffamstutz]

Hi,

ospBenchmark is something to start with, but as we are going to start our alpha release of v2.0 on Monday, I think we should do better to provide benchmarks for users to use. In the v1.x releases, our offerings were fairly limited, where most of our benchmarking was/is done in the context of publications we produce with various partners in academia.

We would be honored for you to consider OSPRay as something to benchmark! In the short term, starting with ospBenchmark is good place to start (and feel free to ask any questions that come up). However, I'd like to keep this issue open for the sake of us tracking publicly that a better benchmark app will be available for when the official v2.0 release goes out (and maintaining it after that).

Thanks!

[jeffamstutz]

We can report in this issue when we have new benchmarking capabilities to try out.

[michaellarabel]

Great, thanks @jeffamstutz - I was able to get ospBenchmark running well so far paired with the various demos from OSPray.org. So far it's been a fun renderer to test so certainly looking forward to using OSPray more moving forward.

Nice to hear that OSPray 2.0 could yield new benchmarking improvements given the limited information from ospBenchmark so far. I'm just using the time metrics and converting to FPS for easy analysis. If any other ideas or feedback on the benchmark functionality come to mind, will certainly pass it along here.

[111alan]

I am actually doing the thing you're talking about. Using the 1.8.5 binary files and benchmark.py I can run default tests and generate the results. I don't know what the results mean though(except less is better), neither do I know how to impliment other tutorials to the test.

Also intel parallel studio 2019 didn't catch any usages of AVX512 in the benchmark. ` --- Now Starting ---

=== TEST 1/2: spheres-4k-1spp (pt) === Running "ospBenchmark --generate:spheres -4k -bf 8 -wf 8 -r pt -i bench_output/test_spheres-4k-1spp_pt -sg:spp=1" Statistics: max: 2088 min: 2061 median: 2073 median abs dev: 7 mean: 2073 std dev: 8

of samples: 8

=== PASSED === === TEST 2/2: spheres-4k-1spp (scivis) === Running "ospBenchmark --generate:spheres -4k -bf 8 -wf 8 -r sv -i bench_output/test_spheres-4k-1spp_scivis -sg:spp=1" Statistics: max: 388 min: 378 median: 384 median abs dev: 3 mean: 383 std dev: 3

of samples: 8

=== PASSED === Press any key to continue . . .`

[Micros-DJB]

I'm also interested in benchmarking, as well as stressing CPUs (for design validation testing).

I am not a programmer, and so far I am struggling a bit to find my way around this tool. I will look again for a Users Manual or similar, but I haven't figured out exactly how to run the benchmark as "111alan " details above. (For example, where do I find the output and statistics?)

The only thing I found so far was that I can run the ospTutorialBouncingSpheres.exe and get a running FPS count in the upper left corner of the window. (This is indeed a good CPU stress test - things get pretty hot.)

Any help for a newby would be greatly appreciated.

Thanks.

[111alan]

I'm also interested in benchmarking, as well as stressing CPUs (for design validation testing). I am not a programmer, and so far I am struggling a bit to find my way around this tool. I will look again for a Users Manual or similar, but I haven't figured out exactly how to run the benchmark as "111alan " details above. (For example, where do I find the output and statistics?) The only thing I found so far was that I can run the ospTutorialBouncingSpheres.exe and get a running FPS count in the upper left corner of the window. (This is indeed a good CPU stress test - things get pretty hot.) Any help for a newby would be greatly appreciated. Thanks.

there is a benchmark.py batch file in the 1.8.5 package, move it to ospbenchmark.exe's directory then you can run it. Python 2.7 is required.

[Micros-DJB]

@ 111alan - thanks for the input. I'll take a look and try it.

[jeffamstutz]

We now have a new ospBenchmark app in v2.0 that represents our current benchmark suite. It is not very complete right now, but it will be the basis of all our synthetic scene benchmarks moving forward.

[michaellarabel]

@jeffamstutz I am currently exploring the new ospBenchmark in 2.0.1. I am still diving into it more, but at this point does seem to be fairly synthetic and with the obvious basic test cases. Has the old ospBenchmark functionality been spun off into any other binary with OSPray2 or was it completely removed?

Currently evaluating some differences between CPUs to see if the new synthetic ospBenchmark is as meaningful as my previous way of using various real-world demo scenes fed into the OSPray 1.x ospBenchmark for comparing hardware.

Thanks for any insight.

[jeffamstutz]

The old benchmark app was based on our scene graph, which we migrated out of the core OSPRay repository and into OSPRay Studio. We are still working on getting studio up to OSPRay v2, but we fully intend on bringing back the old functionality.

The difference between what we will benchmark is that OSPRay's benchmark suite (ospBenchmark) will strive to remain maximally synthetic in order to minimize scene file dependencies. OSPRay Studio then owns our file loaders and the rich library for scene manipulation (i.e. our scene graph) which will prove much more useful for "real world" scenarios.

Thus I think OSPRay's benchmarks will continue to be useful, but they compliment (not replace) file based benchmarks. We are aiming to soon get the new ospBenchmark setup to do our own automated benchmarking, where we will continue to add to OSPRay's benchmark suite to cover more aspects of OSPRay's performance.

[michaellarabel]

Thanks for the feedback, understood.