Profiling for C++ backend

[RichardWarfield]

I implemented this functionality in order to be able to profile a function running on the C++ backend. Some notes on the implementation are in the commit message and usage should be the same as for the Python profiler, i.e. call cgt.profile.stop(), cgt.profile.start(), cgt.profile.print_stats().

Currently I have only implemented this on the sequential interpreter though it should be straightforward to make the parallel interpreter work as well.

To test this I made a few minor changes to the MNIST demo so the backend can be switched from the command line.

Examples:

Python backend (original functionality):

richard@seoul-mint ~/workspace/cgt $ python ./examples/demo_mnist.py --epochs=1 --profile --devtype=cpu --backend=python
     Epoch |  Train NLL |  Train Err |   Test NLL |   Test Err | Epoch Time
         0 |   0.245118 |     0.0781 |   0.225886 |     0.0706 |    14.5864
Total time elapsed: 15.7 seconds

************************************************************
*************************  By Op  **************************
************************************************************
Instruction               Count         Time         Frac    Frac cumsum
----------------------  -------  -----------  -----------  -------------
Mul22{N,N}                 1413  5.53256      0.352893          0.352893
Mul22{T,N}                 1407  4.41996      0.281926          0.63482
Mul22{N,T}                  938  3.91324      0.249605          0.884425
multiply                   7510  0.382699     0.0244104         0.908835
Alloc{dtype=f4,ndim=2}    21592  0.283814     0.018103          0.926938
...

C++ CPU backend:

richard@seoul-mint ~/workspace/cgt $ python ./examples/demo_mnist.py --epochs=1 --profile --devtype=cpu --backend=native
using python impl for Argmax{1}
     Epoch |  Train NLL |  Train Err |   Test NLL |   Test Err | Epoch Time
         0 |   0.244724 |     0.0778 |    0.22509 |     0.0696 |    3.05734
Total time elapsed: 3.17 seconds

************************************************************
*************************  By Op  **************************
************************************************************
Instruction               Count         Time         Frac    Frac cumsum
----------------------  -------  -----------  -----------  -------------
Mul22{N,N}                 1413  0.598506     0.188843          0.188843
sqrt                       1407  0.595224     0.187807          0.37665
Mul22{T,N}                 1407  0.468959     0.147968          0.524617
multiply                   7510  0.386939     0.122088          0.646706
divide                     4229  0.341507     0.107754          0.754459
...

C++ GPU backend:

richard@seoul-mint ~/workspace/cgt $ python ./examples/demo_mnist.py --epochs=1 --profile --devtype=gpu --backend=native
using python impl for Argmax{1}
     Epoch |  Train NLL |  Train Err |   Test NLL |   Test Err | Epoch Time
         0 |   0.244901 |     0.0779 |   0.225598 |     0.0703 |    1.89208
Total time elapsed: 1.89 seconds

************************************************************
*************************  By Op  **************************
************************************************************
Instruction               Count         Time         Frac    Frac cumsum
----------------------  -------  -----------  -----------  -------------
Transport                 13160  1.64103      0.866082          0.866082
Mul22{N,N}                 1413  0.107393     0.0566784         0.922761
multiply                   7510  0.033405     0.0176301         0.940391
add                        3754  0.015549     0.00820627        0.948597
divide                     4700  0.0137958    0.00728097        0.955878
...

[f0k]

You merged a bunch of foreign commits into your PR. You should be able to fix this by:

git fetch upstream
git reset --hard upstream/master
git cherry-pick 96b36ed
git push --force

(assuming you've got joschu/cgt added as a remote named "upstream")

[RichardWarfield]

Thanks... it should be fixed now.

[joschu]

Nice work -- I'll review it in the next day or two.