search

DK96-OS / MathTools

Mathematical Software Components. This library is actively maintained, and aims to stay updated. New feature proposals are welcome, but may not be included.
Apache License 2.0
2 stars 1 forks source link

Prime Number Cache Performance Testing Improvement #21

Closed DK96-OS closed 3 years ago

DK96-OS commented 3 years ago

Improve performance testing by recording small, independent time measurements, and then performing statistical analysis to determine the reliability of each measurement.

DK96-OS commented 3 years ago

GitHub Actions Test Results Accuracy Issue

Performance testing using Kotlin's built in measureNanoTime function provides highly accurate and reliable time measurements for blocks of code. This has been demonstrated offline using OpenJDK 11, however it appears that on GitHub, the measureNanoTime function returns values that are multiples of 100, or 0. TestTargetedAccessJuly6_2021_12_25 Table 1: TargetedAccess Test Measurements The most recent GitHub Actions Test report for PrimeCachePerformanceTest, showing the TargetedAccess Test output. This occurs in both Java 8 and Java 11 Test Reports.

Implications

Without a common reliable computing resource, reproducibility is a challenge. Comparing results between changes over time will require consideration of the system running the tests, which unfortunately will not be GitHub.

Potential Resolutions

One way that this could be resolved, is to investigate alternative OpenJDK distributions available in GitHub Actions. Another is to try to use a newer version of OpenJDK.

DK96-OS commented 3 years ago

Java Version Performance Comparison

While the precision of each measurement is still limited to 100 nanoseconds by GitHub actions, there are noteable differences between Java versions. PrimeCachePerformanceGraph Table 2 : Prime Cache Performance Across Java Versions This graph shows the output results for TargetedIndex Test, for versions 11, 14, 15, and 16. Version 8 is shown in the legend, but this data was not included in the graph.

Java version 14 is clearly the performance winner, with 15 in second place. Version 16 delivers the worst performance overall, but appears to be more stable than 11. With monotonically increasing measurements, there are no spikes in performance at specific target indices, which cannot be said for any other version.

DK96-OS commented 3 years ago

Consider that:

These statements, while true, justify removing support for 8, and adding support for 16.