Use a platform-independent hash function in Coordinate Encoder

[chetan51]

Investigate a faster, platform-independent hash function instead of numpy.random, and use it instead in Coordinate Encoder.

[cogmission]

I think the takeaway from Fergal's post was to use a cross-platform random number generator more than try and increase its efficiency since we are both faced with duplicating results in other environments. Also, it would be nice to change to Mersenne Twister for all random number generators in the modules coupled with a common seed used across all modules. The spacial pooler uses an elaborate seed that emphasizes uniqueness - which for someone trying to port to another language is a little frustrating. I think we might be shooting ourselves in the foot with this issue because the goal should be rapid and mass adoption emphasized over extreme optimization. Not to be a buzz kill or anything. You know I think your work is great! :-)

[fergalbyrne]

Hi David/Chetan,

Just to clarify, Python and numpy use the standard Mersenne Twister, as do Ruby and several languages (see the Wikipedia page for the full list). In order to pass tests, nupic.core must also be using the same RNG. So there's nothing needing to be done, we already have a cross-platform RNG which is also efficient "enough". (It is a somewhat expensive RNG but it has way better properties than the ones used in Java for example, which was designed for slower hardware and small memories).

Ignoring the Geospatial and Coordinate Encoders, the reason for using seeds for RNGs all over nupic (and nupic.core) is so that the exact state of encoders, synapses, input connection maps etc are reproducible for testing and comparison between implementations. If you use the same seeds everywhere, any deviation is due to bugs (or algorithmic changes), so you can isolate and fix them.

So, when porting, we simply need to ensure we have a decent MT generator which can be configured to reproduce the one in nupic. That way we can have input encodings and SDRs which can be shared between HTMs running on different platforms.

And I also hope everyone appreciates how impressive and important we all consider the work Chetan's been doing - to paraphrase Jeff, he's a catalyst within a catalyst!

Cheers, F

On Fri, Aug 15, 2014 at 3:01 AM, David Ray notifications@github.com wrote:

I think the takeaway from Fergal's post was to use a cross-platform random number generator more than try and increase its efficiency since we are both faced with duplicating results in other environments. Also, it would be nice to change to Mersenne Twister for all random number generators in the modules coupled with a common seed used across all modules. The spacial pooler uses an elaborate seed that emphasizes uniqueness - which for someone trying to port to another language is a little frustrating. I think we might be shooting ourselves in the foot with this issue because the goal should be rapid and mass adoption emphasized over extreme optimization. Not to be a buzz kill or anything. You know I think your work is great! :-)

— Reply to this email directly or view it on GitHub https://github.com/numenta/nupic/issues/1205#issuecomment-52267231.

Fergal Byrne, Brenter IT

Author, Real Machine Intelligence with Clortex and NuPIC https://leanpub.com/realsmartmachines

Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014: http://euroclojure.com/2014/ and at LambdaJam Chicago, July 2014: http://www.lambdajam.com

http://inbits.com - Better Living through Thoughtful Technology http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne

e:fergalbyrnedublin@gmail.com t:+353 83 4214179 Join the quest for Machine Intelligence at http://numenta.org Formerly of Adnet editor@adnet.ie http://www.adnet.ie

[cogmission]

Hi Fergal,

Agreed! :-) (especially that comment about Chetan's work)

However, there is still a problem in that the seed used in the spatial_pooler_unit_test is generated by a custom elaborate function that incorporates the current time into its calculation - thereby guaranteeing that no two passes over that code generate the same random number. This code would have to be changed in order to adhere to the convention of "commonality" across all modules, tests and eventually all platforms.

To be specific, the "tie breaker" function uses calls to nupic/bindings/math.py which has wrapper functions which call out to C++ functionality underneath. Also, the Random function, in the SpatialPooler is imported (as "NumpyRandom") from that math function and calls several "wrapped" functions inside the same library. I am not sure what finally gets used underneath the various methods that are getting called to generate random numbers and functionality, but I do know that whatever it is cannot be duplicated across multiple platforms - at least not without a lot of headaches.

Even if the "stock" python random number generator were the one being used (and just to be absolutely clear - we are not using the stock RNG - instead it comes from the nupic/bindings/math.py module); we would have to be mindful of even the methods called on that generator. I just spent a couple of hours trying to find the right combination of ways to initialize the MersenneTwister.java, and methods called which generate the same random number as that which is generated inside of nupic. We would have to restrict the methods called inside of nupic to one or two because finding the corollary for those methods is not a trivial task.

In summary, the uniformity of using MersenneTwister in java to match that same usage in Python or C++ is only "theoretically" feasible, and only if we are mindful of the ways in which it is used inside of nupic (i.e. no methods being called except for Random.random() ). At the present time, this is not the case. I have decided to circumvent these methods entirely within the nupic python code so that I can reproduce the same output during my porting work. In order to adhere to a convention of commonality across the nupic codebase there would have to be changes made - we are not simply "there" at the present time.

cheers, David

[cogmission]

P.S. My "success" with duplicating the random number functionality is limited only to the porting of the new temporal_memory.py because it doesn't call into math.py. The spatial pooler is again a "different animal" in that it is using nupic/bindings/math.py which has a bunch of methods which wrap C++ calls for special random number generation functionality and these are not cross-platform friendly.

[fergalbyrne]

Hi David,

My bad, I was only looking at a few examples of this. It looks like you're correct in your assessments, and this would mean this is far from the trivial task I pictured at first. Thankfully, Chetan chose to use the numpy RNG for his encoder, so it's "only" the entire NuPIC codebase which suffers from this problem!

It might be possible to simply replace the custom RNG class in nupic.core [1] with a well-behaved MT-based RNG. This would break backward compatibility, but that might not be a huge issue, and I believe it would be worth the transition.

[1] https://github.com/numenta/nupic.core/blob/master/src/main/utils/Random.hpp

On Fri, Aug 15, 2014 at 10:55 AM, David Ray notifications@github.com wrote:

P.S. My "success" with duplicating the random number functionality is limited only to the porting of the new temporal_memory.py because it doesn't call into math.py. The spatial pooler is again a "different animal" in that it is using nupic/bindings/math.py which has a bunch of methods which wrap C++ calls for special random number generation functionality and these are not cross-platform friendly.

— Reply to this email directly or view it on GitHub https://github.com/numenta/nupic/issues/1205#issuecomment-52291049.

Fergal Byrne, Brenter IT

Author, Real Machine Intelligence with Clortex and NuPIC https://leanpub.com/realsmartmachines

Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014: http://euroclojure.com/2014/ and at LambdaJam Chicago, July 2014: http://www.lambdajam.com

http://inbits.com - Better Living through Thoughtful Technology http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne

e:fergalbyrnedublin@gmail.com t:+353 83 4214179 Join the quest for Machine Intelligence at http://numenta.org Formerly of Adnet editor@adnet.ie http://www.adnet.ie

[cogmission]

For convenience let me recap:

1. The use of Python random
2. Uniform seed. (Just use "42")
3. Only use the random() method.

Without these, cross-platform identical results are impossible. No mistaking it.

[cogmission]

Oh btw, "Only the whole Nupic library..." - that was funny! :-) I don't think this should be a problem, this library has no cryptographic intent?

[oxtopus]

Feel free to re-open if anyone thinks this issue needs reconsideration, but coordinate encoder now uses the random implementation in nupic.core.

See https://github.com/numenta/nupic.core/blob/master/src/nupic/utils/Random.cpp and https://github.com/numenta/nupic/commit/feaec50e352e0765b07e5dcb656ad1c9deef9685