Closed awf closed 3 years ago
@dcrc2 after this, I am tidying the other types, e.g. int -> ks::Integer. I am inclined to move ks::tuple -> ks::Tuple; are there any potential gotchas?
@dcrc2 after this, I am tidying the other types, e.g. int -> ks::Integer. I am inclined to move ks::tuple -> ks::Tuple; are there any potential gotchas?
Sounds fine to me.
@cgravill, how do I look at the benchmark results?
@cgravill, how do I look at the benchmark results?
from the PR?
Ah.. I see now there are failures in the benchmark -- I guess we are being tolerant of fails, and yet we should know about new ones....
We could change it so someone has to go in and explicit mark the failures as ignore. We kinda want both.
Perhaps need a separation between correctness testing of running benchmarks and more optimistic running of benchmarks?
@cgravill, how do I look at the benchmark results?
from the PR?
- You'd open the checks
- then Benchmark / build
- select the artifact
- download
- inspect with pytest-benchmark
Can we have the build emit at least one svg/png/txt into the artefacts, in order to quickly see the results?
@awf What are you after? The output of the benchmark runs are there on the std out, and can see on the checks and quicker to get at than the artifacts:
------------------------------------------------------------------------- benchmark 'torch.Size([16, 16]) test_backwards': 2 tests -------------------------------------------------------------------------
Name (time in us) Median IQR Outliers Mean StdDev Min Max Iterations Rounds
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
backwards[sqrl_pytorch-PyTorch Nice-torch.Size([16, 16])] 76.4010 (1.0) 1.1000 (1.10) 2565;3530 77.1161 (1.0) 3.4151 (1.0) 73.9010 (1.00) 163.9020 (1.0) 1 42992
backwards[sqrl_pytorch-PyTorch-torch.Size([16, 16])] 76.4010 (1.0) 1.0000 (1.0) 4595;5810 77.8629 (1.01) 4.6130 (1.35) 73.8010 (1.0) 165.0030 (1.01) 1 39154
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------- benchmark 'torch.Size([16, 16]) test_forward': 3 tests ------------------------------------------------------------------------
Name (time in us) Median IQR Outliers Mean StdDev Min Max Iterations Rounds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
forward[sqrl_pytorch-Knossos-torch.Size([16, 16])] 13.2010 (1.0) 0.2000 (1.0) 887;1577 13.3057 (1.0) 0.7819 (1.0) 12.7000 (1.0) 64.8010 (1.0) 1 55679
forward[sqrl_pytorch-PyTorch Nice-torch.Size([16, 16])] 47.0010 (3.56) 0.9000 (4.50) 2377;8021 48.5442 (3.65) 4.0005 (5.12) 45.5010 (3.58) 124.3020 (1.92) 1 35663
forward[sqrl_pytorch-PyTorch-torch.Size([16, 16])] 47.0010 (3.56) 0.7000 (3.50) 4256;4941 48.0999 (3.61) 3.5670 (4.56) 45.6000 (3.59) 160.9020 (2.48) 1 33602
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------- benchmark 'torch.Size([16, 16]) test_inference': 3 tests ------------------------------------------------------------------------
Name (time in us) Median IQR Outliers Mean StdDev Min Max Iterations Rounds
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
inference[sqrl_pytorch-Knossos-torch.Size([16, 16])] 12.1000 (1.0) 0.1010 (1.0) 784;5511 12.1256 (1.0) 0.6893 (1.0) 11.7000 (1.0) 63.9010 (1.0) 1 53996
inference[sqrl_pytorch-PyTorch Nice-torch.Size([16, 16])] 35.2000 (2.91) 3.8000 (37.62) 2298;1469 36.6900 (3.03) 3.4147 (4.95) 34.4000 (2.94) 106.2020 (1.66) 1 46468
inference[sqrl_pytorch-PyTorch-torch.Size([16, 16])] 34.9000 (2.88) 0.2010 (1.99) 4840;5975 35.5582 (2.93) 2.5184 (3.65) 34.3000 (2.93) 148.1020 (2.32) 1 42373
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------- benchmark 'torch.Size([4, 4]) test_backwards': 2 tests --------------------------------------------------------------------------
Name (time in us) Median IQR Outliers Mean StdDev Min Max Iterations Rounds
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
backwards[sqrl_pytorch-PyTorch Nice-torch.Size([4, 4])] 56.8010 (1.0) 0.6900 (1.0) 118;3657 57.8622 (1.00) 20.0945 (1.79) 55.4000 (1.0) 1,321.1200 (1.63) 1 56117
backwards[sqrl_pytorch-PyTorch-torch.Size([4, 4])] 56.9010 (1.00) 0.7000 (1.01) 78;355 57.5752 (1.0) 11.2488 (1.0) 55.6000 (1.00) 808.4130 (1.0) 1 4650
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------- benchmark 'torch.Size([4, 4]) test_forward': 3 tests --------------------------------------------------------------------------
Name (time in us) Median IQR Outliers Mean StdDev Min Max Iterations Rounds
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
forward[sqrl_pytorch-Knossos-torch.Size([4, 4])] 11.2000 (1.0) 0.2000 (1.0) 122;185 11.2702 (1.0) 1.0411 (1.0) 10.7000 (1.0) 63.2010 (1.0) 1 7361
forward[sqrl_pytorch-PyTorch Nice-torch.Size([4, 4])] 38.1000 (3.40) 0.5000 (2.50) 98;2140 39.0105 (3.46) 21.5841 (20.73) 36.9010 (3.45) 1,345.5210 (21.29) 1 47081
forward[sqrl_pytorch-PyTorch-torch.Size([4, 4])] 38.1000 (3.40) 0.4010 (2.00) 94;1955 38.8816 (3.45) 16.6131 (15.96) 37.0010 (3.46) 639.4100 (10.12) 1 38910
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------- benchmark 'torch.Size([4, 4]) test_inference': 3 tests -------------------------------------------------------------------------
Name (time in us) Median IQR Outliers Mean StdDev Min Max Iterations Rounds
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
inference[sqrl_pytorch-Knossos-torch.Size([4, 4])] 10.0000 (1.0) 0.2000 (1.0) 102;567 10.1017 (1.0) 1.0447 (1.0) 9.6000 (1.0) 81.6010 (1.0) 1 7121
inference[sqrl_pytorch-PyTorch Nice-torch.Size([4, 4])] 29.7010 (2.97) 0.2990 (1.49) 128;2026 30.3784 (3.01) 14.1483 (13.54) 29.1000 (3.03) 637.9100 (7.82) 1 56179
inference[sqrl_pytorch-PyTorch-torch.Size([4, 4])] 29.7010 (2.97) 0.3000 (1.50) 370;2396 30.2962 (3.00) 9.9701 (9.54) 29.1000 (3.03) 614.1090 (7.53) 1 55005
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
doing a comparison against e.g. master is a little more involved and scheduled as AB#19477
@awf What are you after? The output of the benchmark runs are there on the std out, and can see on the checks and quicker to get at than the artifacts:
Ah yes, indeed, thanks.
Change default float to float32.
A few test tolerances needed to go from 1e-6/1e-8 to 1e-5 but this is expected.