DK96-OS
commented
3 years ago Unless there is another method besides measureNanoTime that can record time measurements to nanosecond accuracy, the only option remaining is to perform each operation 100x.
One issue with this approach is that the operation itself is not isolated, the cache must be cleared between each operation which may affect the result. To work around this, the Cache can be instantiated 100x, and each cache can be accessed in sequence during the measurement. This would isolate the operation, and enable the cache clearing to be performed post-measurement.
Performance testing using Kotlin's built in
Table 1: TargetedAccess Test Measurements The most recent GitHub Actions Test report for
measureNanoTimefunction provides highly accurate and reliable time measurements for blocks of code. This has been demonstrated offline using OpenJDK 11, however it appears that on GitHub, the measureNanoTime function returns values that are multiples of 100, or 0.PrimeCachePerformanceTest, showing the TargetedAccess Test output. This occurs in both Java 8 and Java 11 Test Reports.Implications
Without a common reliable computing resource, reproducibility is a challenge. Comparing results between changes over time will require consideration of the system running the tests, which unfortunately will not be GitHub.
Potential Resolutions
One way that this could be resolved, is to investigate alternative OpenJDK distributions available in GitHub Actions. Another is to try to use a newer version of OpenJDK.
Originally posted by @DK96-OS in https://github.com/DK96-OS/MathTools/issues/21#issuecomment-874916971