Real-life benchmark: Hotgym example using C++ algorithms

[breznak]

Implement a pipeline running full real-world HTM task.

Currently implemented using raw HTM classes (TM, SP,...), not NetworkAPI (needs TM/SPRegion), not as python code using c++ bindings (would be possible).

Pipeline:

• [x] compile as standalone executable (for profiling)
• [ ] load CSV from file
  • [ ] use our classical "hotgym" dataset
  • [ ] parse command-line for optional filename and num runs
• [x] encode CSV data
  • [ ] MultiEncoder for more fields than 1
• [x] run SpatialPooler to get SDR
  • [x] global
  • [x] local inhibition
• [x] run TP to get temporal predictions
  • [x] use more modern TM as alternative
  • [x] TP (old) obsoleted
  • [x] BacktrackingTM (TP based) obsoleted
  • [x] show SDR output computed by these TM flavours
  • [x] also checks deterministic algorithms' outputs #194
• [x] compute Anomaly score
  • [ ] test AnomalyLikelihood
• [ ] add SDR Classifier
  • [ ] needs encoder topDownCompute (SDR -> Real) decided WONTFIX
• [x] measure execution time
  • [x] more fine-grained separate timers for each part of pipeline (SP, Encoder, TM,..)
  • [ ] fine grained timer checks for each part
• [ ] implement as a class to make more reusable
• [x] use SDR for all layers
  • [ ] SDR Metrics
  • [ ] enforce common Compute/Serializable/... interface
• [ ] implement using core algorithms (SP, TM)
  • [x] encoder
  • [x] SP
  • [x] TM
  • [x] AN
  • [ ] classifier
  • [ ] predictor
  • [ ] CP (later, when implemented) #285
  • [ ] implement using NetworkAPI
• [ ] test parallelization #255
• [ ] test interfaces
  • [ ] serialization
• [ ] optimize parameters #433

---

We are looking for a real-life benchmark we can use as a base for our performance optimizations #3 . In Python there is a "Hotgym anomaly example" (stresses encoder, SP, TM, Anomaly) , implement similar example in C++ and add it to integration-tests with timing.

• suggested with NAPI

I have ported SPRegion and TMRegion and they run under windows but I was waiting until after PyBind was implemented to merge them in with a later PR.

Waiting for #54 SPRegion & TMRegion in C++

[breznak]

@ctrl-z-9000-times @dkeeney Please hold off merging PRs before we implement this (should be soon, by Mon)

Any of you good with NetworkAPI? I need a "hello world" example, where we create a Network with all the basic parts: encoder, SP, TM, Anomaly. And run it through hotgym dataset and measure time. It should be relatively simple, is there any example constructing network?

Then we can proceed merging: SDR, Random, Eigen PRs Thanks! :+1:

[dkeeney]

The CppRegionTests.cpp has some of the parts, but SP and TM cannot be implemented in C++ only until we complete the SPRegion and TMRegion classes in C++. Currently there is Python code that fills in the gap. I have ported SPRegion and TMRegion and they run under windows but I was waiting until after PyBind was implemented to merge them in with a later PR.

I think a hotgym example/tests would be a good addition to unit_tests.

[ctrl-z-9000-times]

Sorry, I can't help with the network API. I don't use the network API code, it doesn't compile so I comment it out from CMakeLists.txt (boost is playing hide and seek w/ cmake).

To benchmark the SpatialPooler and SDR-Classifier, MNIST is a good hello-world type dataset. It should take 10-20 minutes to run through the whole dataset, assuming you compiled for release mode. Debug mode takes approx 10 times longer. The repo "Numenta/nupic.vision" has an example of MNIST which works, but it will need to be updated to work with this fork. I wrote my own solution to the MNIST dataset but I think there are bugs in it still.

[dkeeney]

Our whole objective is to get the network API to build (with boost until we can use C++17) so that it is usable. It is the framework in which all of the algorithms can be coordinated.

• Did you install cmake and boost?
• Have you tried the latest version of the nupic_cpp repository?

The hot Gym example is Python code. In order to port that to a C++ example/test so we can use it as a performance tests we will need to port not only the hot Gym example but also complete the C++ code set by porting the Python code SPRegion and TMRegion modules to C++ so that the SP and TM can be executed as C++ plugins rather than as Python plugins.

[breznak]

have ported SPRegion and TMRegion and they run under windows but I was waiting until after PyBind was implemented to merge them in with a later PR.

Thanks David, so I have the options:

• implement regions in python (and test bindings, c++ core)
• implement LATER with your c++ *Region classes
• implement using raw classes (SP, TM), similar to what is done in HelloSPTMTest

To benchmark the SpatialPooler and SDR-Classifier, MNIST is a good hello-world type dataset. It should take 10-20 minutes to run through the whole dataset

That would be a good example, thanks! We can even add TM (would be useless, but we want to stress it). I will use MNIST as addition to hotgym

(please resolve the boost build issue, but in a new thread)

need to port not only the hot Gym example but also complete the C++ code set by porting the Python code SPRegion and TMRegion modules to C++ so that the SP and TM can be executed as C++ plugins

Still, I can now write example in Py that uses cpp impl, so our c++ code, right?

[ctrl-z-9000-times]

build (with boost until we can use C++17)

Thanks David! I updated to gcc-8 and c++17 and now it builds much cleaner.

[breznak]

@dkeeney @ctrl-z-9000-times should we keep both TP, TM in this example,benchmark, or go only with TM, as most code does?

[dkeeney]

Lets keep both because this tests the C++ to Python and Python to C++ interfaces for both algorithms.