search

open-quantum-safe / profiling

Network-level performance testing of post-quantum cryptography using the OQS suite
https://openquantumsafe.org/benchmarking
MIT License
8 stars 7 forks source link

Visualization #5

Closed baentsch closed 4 years ago

baentsch commented 4 years ago

This is to collect work-in-progress information on speed-JSON file visualization.

Current version accessible at https://test.openquantumsafe.org/performance.html

Feedback/improvement suggestions by @dstebila :

baentsch commented 4 years ago

@dstebila : I tend to reject your suggestion for "labeling close to the data" unless you can provide a good reason: (My) rationale: 1) We already have tooltips/callouts containing algorithm name and value -- which I think fulfills your wish very well (and IMO better than static vertical labels underneath a value column). 2) The current data structures allow us to easily create time-series graphs (instead of bar charts) showing each algorithm's performance over time (nightly runs) -- with the ability to hide/filter out complete series/algorithms. A per-algorithm (single dataset) column doesn't allow us to do this as easily (unless I still don't understand the Chart.js logic -- which is possible). Please let me have arguments to change my assessment/point of view.

dstebila commented 4 years ago

@dstebila : I tend to reject your suggestion for "labeling close to the data" unless you can provide a good reason: (My) rationale: 1) We already have tooltips/callouts containing algorithm name and value -- which I think fulfills your wish very well (and IMO better than static vertical labels underneath a value column). 2) The current data structures allow us to easily create time-series graphs (instead of bar charts) showing each algorithm's performance over time (nightly runs) -- with the ability to hide/filter out complete series/algorithms. A per-algorithm (single dataset) column doesn't allow us to do this as easily (unless I still don't understand the Chart.js logic -- which is possible). Please let me have arguments to change my assessment/point of view.

I found one website on the Internet that agrees with me. :) But not all visualization types accommodate that easily, or automatically. In the image for this paper I had to do it manually.

To do it for bar charts seems straightforward with Chart.js, so I'd be curious to see what it looks like, though admittedly it could look horrible. Doing it for line graphs line in the 'agrees with me' link doesn't seem possible in Chart.js natively, although maybe this plugin would do it.

However, let's get the main functionality working and then we could fiddle around with these details later.

baentsch commented 4 years ago

Have a look at https://test.openquantumsafe.org/speed_kem_series.html: Measurement types split out as per the last bullet above; two different ways to show time series. The huge performance difference between the algorithms makes the line graph nearly useless... Maybe I'll add a switch between bar chart and line chart. Any other idea(s) welcome.

baentsch commented 4 years ago

@dstebila Please check (again) https://test.openquantumsafe.org/speed_kem_series.html: It now implements pretty much all of your proposals/suggestions. I left in some old data to document my stupidity properly selecting AWS CPU resources (and to make the graphs look cute :-). If OK, I'll then proceed to refactor the code such that all visualizations get this new logic.

baentsch commented 4 years ago

@dstebila Now also including family filters and table views (when drilling down to/selecting single days/test runs) as discussed on Wed. Again, all first implemented in https://test.openquantumsafe.org/speed_kem_series.html -- if OK (?) I'd then like to transfer, possibly refactor this to all visualizations. Goal is to close issue and finalize PR for now and switch focus to collecting data from the other machines/architectures.

dstebila commented 4 years ago

Is it possible to do the algorithm families as a multiselect? So that I could easily choose to view several families? Right now you can only do one family, although you can click individual algorithms after that.

Also, can we add download links for the raw data?

I am wondering as well whether we should be cycle counts in the table.

baentsch commented 4 years ago

Is it possible to do the algorithm families as a multiselect? So that I could easily choose to view several families? Right now you can only do one family, although you can click individual algorithms after that.

Done+live. I find it looks ugly but I see the value.

Also, can we add download links for the raw data?

Done+live: Appears together with the table (at date selection).

I am wondering as well whether we should be cycle counts in the table.

I'm not sure I see the value in this feature: a) Can't this be inferred/computed from the CPU MHz output we already deliver as part of the output? b) As not all measurement types collect cycle count (only liboqs ones do), and if you really want this feature, what about we add yet another option transforming the presently displayed operations/s ("opss") to CPU cycles via the collected [M]Hz info: [CPU] cycles=[CPU M]Hz/opss? Final reason for hesitation: I personally don't like showing large(r) numbers to indicate less (performance).

dstebila commented 4 years ago

I am wondering as well whether we should be cycle counts in the table.

I'm not sure I see the value in this feature: a) Can't this be inferred/computed from the CPU MHz output we already deliver as part of the output? b) As not all measurement types collect cycle count (only liboqs ones do), and if you really want this feature, what about we add yet another option transforming the presently displayed operations/s ("opss") to CPU cycles via the collected [M]Hz info: [CPU] cycles=[CPU M]Hz/opss? Final reason for hesitation: I personally don't like showing large(r) numbers to indicate less (performance).

Well for algorithms like Classic McEliece keygen, which is on the order of 1-2 operations per second, CPU frequency / operations per second loses precision.

baentsch commented 4 years ago

Well, OK... But to be honest, even if there's more "cycle efficiency" I still believe "net" operations/sec are more tell-tale for effective utility of an algorithm -- and I thought that's what we meant to capture with the visualizations. The one rationale I can understand is to make numbers comparable to the Supercop way of showing things. So I'll add the cycle counts to the measurements and then add them to the tables shown. But I think I leave it to someone else to add them to the visuals.

dstebila commented 4 years ago

Ready to close?

baentsch commented 4 years ago

Not yet: The items are ticked off and implemented for the liboqs performance tests, but I still need to move the corresponding code to the other data sets (incl. those without cycle counts :-). Will do that via a "v0.2 PR" later this week. Then we can close this.

baentsch commented 4 years ago

Closed with PR #9