search

htm-community / NAB

The Numenta Anomaly Benchmark
GNU Affero General Public License v3.0
3 stars 3 forks source link

htm core detector with random seed? #35

Closed Zbysekz closed 4 years ago

Zbysekz commented 4 years ago

  1. I have noticed, that htmcore_detector.py use random seed (=0). In fact, it is just about the RDSE encoder, there is the seed param not supplied. I can add this param with PR.

  2. I am not sure how optimalization exactly works here in NAB (the one with docker?), but in my opinion for optimalization process, the detector should be deterministic. (i am imagining itself as optimizer and i am trying to find better params, but each call the result is different with no dependency of what params i have changed.) This potentially could help @steinroe.

For example, for art_daily_flatmiddle i get these scores each call: -0.787,-0.567,-0.677....

What you think about this @breznak ?

breznak commented 4 years ago

I have noticed, that htmcore_detector.py use random seed (=0). In fact, it is just about the RDSE encoder, there is the seed param not supplied

I have noticed that in some past PR and started working on that: https://github.com/htm-community/htm.core/pull/799

What should be a simple PR, but I got down the rabbit hole... SP (or something) behaves strange with the default random(=0) seed.

optimalization process, the detector should be deterministic. (i am imagining itself as optimizer and i am trying to find better params, but each call the result is different with no dependency of what params i have changed.)

not exactly. The algorithm behaves correctly on "high level" (HTM), so say there's correct sparsity, boosting works, cells get predicted etc. But when we say "choose randomly 5 out of these cells, and do something.." we want that random to be truly random.

Otherwise, you get overfitting and the param-optimizer could learn as a knowledge "that on 158000th step the cell 1345 is ON". Then you'd get falsely good results for one seed & data combination, but not for other.

Zbysekz commented 4 years ago

Ok i get it, thanks for explaining 👍