search

microsoft / knossos-ksc

Compiler with automatic differentiation
Other
45 stars 10 forks source link

float64->float32 #928

Closed awf closed 3 years ago

awf commented 3 years ago

Change default float to float32.

A few test tolerances needed to go from 1e-6/1e-8 to 1e-5 but this is expected.

awf commented 3 years ago

@dcrc2 after this, I am tidying the other types, e.g. int -> ks::Integer. I am inclined to move ks::tuple -> ks::Tuple; are there any potential gotchas?

dcrc2 commented 3 years ago

@dcrc2 after this, I am tidying the other types, e.g. int -> ks::Integer. I am inclined to move ks::tuple -> ks::Tuple; are there any potential gotchas?

Sounds fine to me.

awf commented 3 years ago

@cgravill, how do I look at the benchmark results?

cgravill commented 3 years ago

@cgravill, how do I look at the benchmark results?

from the PR?

  1. You'd open the checks
  2. then Benchmark / build
  3. select the artifact
  4. download
  5. inspect with pytest-benchmark
awf commented 3 years ago

Ah.. I see now there are failures in the benchmark -- I guess we are being tolerant of fails, and yet we should know about new ones....

image

cgravill commented 3 years ago

We could change it so someone has to go in and explicit mark the failures as ignore. We kinda want both.

Perhaps need a separation between correctness testing of running benchmarks and more optimistic running of benchmarks?

awf commented 3 years ago

@cgravill, how do I look at the benchmark results?

from the PR?

  1. You'd open the checks
  2. then Benchmark / build
  3. select the artifact
  4. download
  5. inspect with pytest-benchmark

Can we have the build emit at least one svg/png/txt into the artefacts, in order to quickly see the results?

cgravill commented 3 years ago

@awf What are you after? The output of the benchmark runs are there on the std out, and can see on the checks and quicker to get at than the artifacts:

------------------------------------------------------------------------- benchmark 'torch.Size([16, 16]) test_backwards': 2 tests -------------------------------------------------------------------------
Name (time in us)                                              Median               IQR             Outliers     Mean            StdDev                Min                 Max            Iterations  Rounds
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
backwards[sqrl_pytorch-PyTorch Nice-torch.Size([16, 16])]     76.4010 (1.0)      1.1000 (1.10)     2565;3530  77.1161 (1.0)      3.4151 (1.0)      73.9010 (1.00)     163.9020 (1.0)               1   42992
backwards[sqrl_pytorch-PyTorch-torch.Size([16, 16])]          76.4010 (1.0)      1.0000 (1.0)      4595;5810  77.8629 (1.01)     4.6130 (1.35)     73.8010 (1.0)      165.0030 (1.01)              1   39154
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------- benchmark 'torch.Size([16, 16]) test_forward': 3 tests ------------------------------------------------------------------------
Name (time in us)                                            Median               IQR            Outliers     Mean            StdDev                Min                 Max            Iterations  Rounds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
forward[sqrl_pytorch-Knossos-torch.Size([16, 16])]          13.2010 (1.0)      0.2000 (1.0)      887;1577  13.3057 (1.0)      0.7819 (1.0)      12.7000 (1.0)       64.8010 (1.0)               1   55679
forward[sqrl_pytorch-PyTorch Nice-torch.Size([16, 16])]     47.0010 (3.56)     0.9000 (4.50)    2377;8021  48.5442 (3.65)     4.0005 (5.12)     45.5010 (3.58)     124.3020 (1.92)              1   35663
forward[sqrl_pytorch-PyTorch-torch.Size([16, 16])]          47.0010 (3.56)     0.7000 (3.50)    4256;4941  48.0999 (3.61)     3.5670 (4.56)     45.6000 (3.59)     160.9020 (2.48)              1   33602
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------- benchmark 'torch.Size([16, 16]) test_inference': 3 tests ------------------------------------------------------------------------
Name (time in us)                                              Median               IQR            Outliers     Mean            StdDev                Min                 Max            Iterations  Rounds
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
inference[sqrl_pytorch-Knossos-torch.Size([16, 16])]          12.1000 (1.0)      0.1010 (1.0)      784;5511  12.1256 (1.0)      0.6893 (1.0)      11.7000 (1.0)       63.9010 (1.0)               1   53996
inference[sqrl_pytorch-PyTorch Nice-torch.Size([16, 16])]     35.2000 (2.91)     3.8000 (37.62)   2298;1469  36.6900 (3.03)     3.4147 (4.95)     34.4000 (2.94)     106.2020 (1.66)              1   46468
inference[sqrl_pytorch-PyTorch-torch.Size([16, 16])]          34.9000 (2.88)     0.2010 (1.99)    4840;5975  35.5582 (2.93)     2.5184 (3.65)     34.3000 (2.93)     148.1020 (2.32)              1   42373
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------- benchmark 'torch.Size([4, 4]) test_backwards': 2 tests --------------------------------------------------------------------------
Name (time in us)                                            Median               IQR            Outliers     Mean             StdDev                Min                   Max            Iterations  Rounds
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
backwards[sqrl_pytorch-PyTorch Nice-torch.Size([4, 4])]     56.8010 (1.0)      0.6900 (1.0)      118;3657  57.8622 (1.00)     20.0945 (1.79)     55.4000 (1.0)      1,321.1200 (1.63)              1   56117
backwards[sqrl_pytorch-PyTorch-torch.Size([4, 4])]          56.9010 (1.00)     0.7000 (1.01)       78;355  57.5752 (1.0)      11.2488 (1.0)      55.6000 (1.00)       808.4130 (1.0)               1    4650
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------- benchmark 'torch.Size([4, 4]) test_forward': 3 tests --------------------------------------------------------------------------
Name (time in us)                                          Median               IQR            Outliers     Mean             StdDev                Min                   Max            Iterations  Rounds
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
forward[sqrl_pytorch-Knossos-torch.Size([4, 4])]          11.2000 (1.0)      0.2000 (1.0)       122;185  11.2702 (1.0)       1.0411 (1.0)      10.7000 (1.0)         63.2010 (1.0)               1    7361
forward[sqrl_pytorch-PyTorch Nice-torch.Size([4, 4])]     38.1000 (3.40)     0.5000 (2.50)      98;2140  39.0105 (3.46)     21.5841 (20.73)    36.9010 (3.45)     1,345.5210 (21.29)             1   47081
forward[sqrl_pytorch-PyTorch-torch.Size([4, 4])]          38.1000 (3.40)     0.4010 (2.00)      94;1955  38.8816 (3.45)     16.6131 (15.96)    37.0010 (3.46)       639.4100 (10.12)             1   38910
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------- benchmark 'torch.Size([4, 4]) test_inference': 3 tests -------------------------------------------------------------------------
Name (time in us)                                            Median               IQR            Outliers     Mean             StdDev                Min                 Max            Iterations  Rounds
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
inference[sqrl_pytorch-Knossos-torch.Size([4, 4])]          10.0000 (1.0)      0.2000 (1.0)       102;567  10.1017 (1.0)       1.0447 (1.0)       9.6000 (1.0)       81.6010 (1.0)               1    7121
inference[sqrl_pytorch-PyTorch Nice-torch.Size([4, 4])]     29.7010 (2.97)     0.2990 (1.49)     128;2026  30.3784 (3.01)     14.1483 (13.54)    29.1000 (3.03)     637.9100 (7.82)              1   56179
inference[sqrl_pytorch-PyTorch-torch.Size([4, 4])]          29.7010 (2.97)     0.3000 (1.50)     370;2396  30.2962 (3.00)      9.9701 (9.54)     29.1000 (3.03)     614.1090 (7.53)              1   55005
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

doing a comparison against e.g. master is a little more involved and scheduled as AB#19477

awf commented 3 years ago

@awf What are you after? The output of the benchmark runs are there on the std out, and can see on the checks and quicker to get at than the artifacts:

Ah yes, indeed, thanks.