search

htm-community / htm.core

Actively developed Hierarchical Temporal Memory (HTM) community fork (continuation) of NuPIC. Implementation for C++ and Python
http://numenta.org
GNU Affero General Public License v3.0
148 stars 74 forks source link

Performance Comparison to nupic #792

Open psteinroe opened 4 years ago

psteinroe commented 4 years ago

Hi everyone,

thanks for the great work on htm.core! I am doing some research on leveraging HTM for anomaly detection and I am wondering wether I should use htm.core or nupic. Is there any comparison in terms of performance?

@breznak You described some issues in PR #15. What would you say - continue with htm.core or rather get nupic running? Its just a masters thesis so I won't be able to dive deep into htm to improve the implementation...

dkeeney commented 4 years ago

htm.core is a rework of NuPic so that it will work. I think you will find that nearly everything you want to do with NuPic should be covered by htm.core. NuPic on the otherhand is basically broken.

@breznak will have to address performance of htm.core but I expect it to be comparable (or perhaps even better) than NuPic.

breznak commented 4 years ago

@steinroe glad for your interest!

I am wondering wether I should use htm.core or nupic

In terms of feature-fullness and active support, htm.core has now much surpased its parent, numenta's nupic(.core).

Is there any comparison in terms of performance? [in anomaly detection]

yes, this is an open problem. Theoretically htm.core should be better/or same/or slightly worse to Numenta's nupic. We did a couple of improvements as well as a couple of "regressions" in the name of biological plausibility. You could start by looking at the Changelog.md, the git diff is already too wild.

Yet, the so-far performance on NAB of htm.core is much worse. (Note, such regression is not observed on "sine", or "hotgym" data.). My gut feeling is it's just "some parameter that is off".

I won't be able to dive deep into htm to improve the implementation...

you would'nt have to dive deep, or we'd be here to help you.. So my recommendation would be: Give htm.core a trial period (say 1-5 weeks) and try to find where the cuplit is. Doing so would help the community and would be a significant result for your thesis. I could help you in trialing down the process in NAB so we can locate the err.

psteinroe commented 4 years ago

@breznak and @dkeeney thanks for the quick replies! I started by analysing the difference in the API and the respective parameters.

Regarding the Spatial Pooler, htm.core differentiates from nupic by three parameters: localAreaDensity, potentialRadius and stimulusThreshold. I tried some Bayesian Optimization to get an idea of what setting for these 3 might work well but only got a score of around 31 with {"localAreaDensity": 0.02149599864266627, "potentialRadius": 4148629.7213459704, "stimulusThreshold": 3.640494932522697}. All other settings were the same as in nupic. Do you have a gut feeling on what ranges would make sense? From the logs I would say localAreaDensity should be <0.09 and stimulusThreshold <10, but still the score is very bad. Any idea on that?

The second thing I did was a direct comparison of the anomaly scores yielded by nupic and by htm.core on the same datasets. You can find the output in the attached pdf. There are two things that aren’t right:

What would you say, is it rather because of the params of the encoders or params of the algorithm itself? Also, there are differences in the parameters of the RSDE encoder. Only the resolution parameter is also used in nupic. Or did I miss something here?

htm_impl_comparison.pdf

breznak commented 4 years ago

Very nice analysis @steinroe !! :+1: Let me answer your ideas by parts.

started by analysing the difference in the API

first, let me correct my mistake, the file I wanted to point you to is API_CHANGELOG.md

About the API (and implementation differences): From top of my head...

tried some Bayesian Optimization to get an idea of what setting for these 3 might work well

cool. If you don't have own framework for optimization, I suggest looking at ./py/htm/optimization/

Do you have a gut feeling on what ranges would make sense?

This is tricky, but can be computed rather precisely. I'll have to look at this deeper again... You can see my trials at community/NAB on this very same problem: https://github.com/htm-community/NAB/pull/15

stimulusThreshold This is a number specifying the minimum number of synapses that must be active in order for a column to turn ON. The purpose of this is to prevent noisy input from activating columns.

So this depends on expected avg number of ON bits from encoder, number of synapses and range of input field each dendrite covers, how noisy is the problem. Anything >=1 is imho a good starting point.

potentialRadius This parameter deteremines the extent of the input that each column can potentially be connected to. This can be thought of as the input bits that are visible to each column, or a 'receptive field' of the field of vision. A large enough value will result in global coverage, meaning that each column can potentially be connected to every input bit. This parameter defines a square (or hyper square) area: a column will have a max square potential pool with sides of length (2 * potentialRadius + 1), rounded to fit into each dimension.

depends on size of the encoder/encoding. And whether you want the columns to act as local/global approximators.

= size aka Inf aka "act global".

"potentialRadius": 4148629.7213459704

Seems your optimizer prefered that variant. (note, it might thus also just got stuck in a local optima). I'd try something as "global" and "25% of input field" as reasonable defaults.

localAreaDensity The desired density of active columns within a local inhibition area (the size of which is set by the internally calculated inhibitionRadius, which is in turn determined from the average size of the connected potential pools of all columns). The inhibition logic will insure that at most N columns remain ON within a local inhibition area, where N = localAreaDensity * (total number of columns in inhibition area) Default: 0.05 (5%)

This is lower-bounded by SP's size aka numColumns. (density * numCols >= some meaningful min value). Its value determines TM's function and TM's variant of "stimulusThreshold" (it's called something different - activationThreshold and minThreshold). 2-15% seems a reasonable value to me.

The second thing I did was a direct comparison of the anomaly scores yielded by nupic and by htm.core on the same datasets.

great graphs!, some notes on that later.

This leads me to a decomposition. The chain is: Encoder -> SP -> TM -> Anomaly Ideally, we'd start from the top to verify all our implementations.

1) It seems like the date encoding is not right or not used at all. The output shows no peak in e.g. the no jump dataset.

actually, there's a small drop at that time. All the graphs are on the same params? Looks that only the "flatmiddle" dataset htmcore results can detect anything meaningful. So the decision value of the others is questionable. But it could be in settings of the datetime encoder, ratio of datetime / RDSE size, insensitivity of sp/tm.

(2) The Scalar Encoding seems off, as e.g. in jumpsup a significant rise is not reflected at all. This could also be due to some parameters of the models I guess.

The RDSE encoder is rather well tested, so I wouldn't expect a hidden bug there. Maybe unusable default params. It seems to me the the HTM didn't learn at all in most cases (except the "flatmiddle") 

 What would you say, is it rather because of the params of the encoders or params of the algorithm itself?

I think it'd be in the params. But cannot tell whether enc/sp/tm. It's all tied together.

Great job investigating so far. I'll be looking at it tonight as well. Please let us know if you find something else or if we can help explain something. Cheers,

breznak commented 4 years ago

@steinroe I've added a "dataset" with only the artificial data, the results look rather good.

Update: When running on only the artificial/synthetic labels, our results are quite good: python run.py -d htmcore --detect --score --optimize --normalize --windowsFile labels/synthetic.json -n 8

htmcore detector benchmark scores written to /mnt/store/devel/HTM/NAB/results/htmcore/htmcore_reward_low_FN_rate_scores.csv

Running score normalization step
Final score for 'htmcore' detector on 'standard' profile = 84.84
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 84.36
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 88.37
Final scores have been written to /mnt/store/devel/HTM/NAB/results/final_results.json.

PS: also I'd suggest using the following branch, it has some nice prints and comments to it. https://github.com/htm-community/NAB/pull/15

breznak commented 4 years ago

CC @Zbysekz as the author of HTMpandaVis, do you think you could help us debugging the issue? I'd really appreciate that!

TL;DR: minor parameters and changes were surely done to htm.core. Compared to Numenta's Nupic, our results on NAB really suck now. I'm guessing it should be matter of incorrect params.

There look into the representations with the visualizer would be really helpful. More info here (and linked posts) https://github.com/htm-community/NAB/pull/15

breznak commented 4 years ago

Btw, I've made fixes to Jupyter plotter, see NAB/scripts/README. These are some figures: HTMcore: htmcore_nab

For result file : ../results/htmcore/artificialWithAnomaly/htmcore_art_daily_flatmiddle.csv
True Positive (Detected anomalies) : 403
True Negative (Detected non anomalies) : 0
False Positive (False alarms) : 2679
False Negative (Anomaly not detected) : 0
Total data points : 3428
S(t)_standard score : -90.17164198169499

Numenta: numenta_nab

For result file : ../results/numenta/artificialWithAnomaly/numenta_art_daily_flatmiddle.csv
True Positive (Detected anomalies) : 1
True Negative (Detected non anomalies) : 4027
False Positive (False alarms) : 0
False Negative (Anomaly not detected) : 0
Total data points : 4032
S(t)_standard score : 0.4999963147227

EDIT: fixed uploaded imgs

Note: both images look very similar. (I don't know why numenta scores look that bad here,must be some bug)

psteinroe commented 4 years ago

@breznak Thanks for the detailed review!

You can turn that off too. HtmcoreDetector does not use that.

Sorry, I forgot to mention that I turned that off for both detectors - HTMCore and Nupic.

cool. If you don't have own framework for optimization, I suggest looking at ./py/htm/optimization/

I saw that right after I was done with my optimization. Going to try yours too very soon!

Thanks for the information on the parameters. I guess from your explanation this is very likely some parameters off. I will look into the default parameters of both to get complete comparison of what could be different.

I setup a repo with the stuff I used to plot both detectors against each other, as I didn't want to bother with all the NAB stuff for that. For the HTMCore detector I actually used yours from htm-community/NAB#15. For nupic, I setup a little server with a (very very) simple API to use the original nupic detector. I just removed the base class so I put the min/max stuff directly into the detector and removed the spatial anomaly detector in both. Feel free to play around: https://github.com/steinroe/htm.core-vs-nupic

When running on only the artificial/synthetic labels, our results are quite good:

That appears weird to me... I am going to look into the outputs of the scores in detail, thanks for that!!!!

There look into the representations with the visualizer would be really helpful.

True! I guess we could be able to find the differences in the param settings better.

breznak commented 4 years ago

When running on only the artificial/synthetic labels, our results are quite good:

That appears weird to me... I am going to look into the outputs of the scores in detail, thanks for that!!!!

I've confirmed that on the "nojump" data, the err still persists. HTMcore does not detect anything, numenta does have a peak.

This could be 2 things:

breznak commented 4 years ago

For nupic, I setup a little server with a (very very) simple API to use the original nupic detector.

This looks good! I might be interested for that for community/NAB to provide (old) numenta detectors. Numenta/NAB switched to docker for the old py2 support, and this seems a good way to interface that!

I'll try your repo, thanks :+1:

breznak commented 4 years ago

I don't understand the plotters/summary code: 

if Error is None:
    TP,TN,FP,FN = 0,0,0,0
    print(standard_score)
    for x in standard_score:
        if x > 0:
            TP +=1
        elif x == 0:
            TN +=1
        elif x == -0.11:
            FP +=1
        elif x == -1:
            FN +=1
    print("For result file : " + result_file)
    print("True Positive (Detected anomalies) : " + str(TP))
    print("True Negative (Detected non anomalies) : " + str(TN))
    print("False Positive (False alarms) : " + str(FP))
    print("False Negative (Anomaly not detected) : " + str(FN))
    print("Total data points : " + str(total_Count))
    print(detector_profile+" score : "+str(np.sum(standard_score)))
else:
    print(Error)
    print("Run from beginng to clear Error")

which gives

[-0.11       -0.11       -0.11       ... -0.10999362 -0.1099937
 -0.10999377]
For result file : ../results/htmcore/artificialWithAnomaly/htmcore_art_daily_nojump.csv
True Positive (Detected anomalies) : 403
True Negative (Detected non anomalies) : 0
False Positive (False alarms) : 2787
False Negative (Anomaly not detected) : 0
Total data points : 3428
S(t)_standard score : -90.17200961167651

But according to the img and raw_anomaly, there should be only a few FP! htm_nab_nojump

Bottom line, is our community/NAB (scorer) correct? We could just copy htmcore results into numenta/NAB and have them re-scored.

psteinroe commented 4 years ago

Sorry for the late reply. I am currently in progress of doing an in-depth comparison between the parameters and there are definitely some differences. You can find the table in the Readme here: https://github.com/steinroe/htm.core-vs-nupic

While most differences could be easily resolved, I need your input on the params of the RDSE Encoder. While HTMCore does have size and sparsity, nupic has w, n and offset set with default parameters. The descriptions seem similar, however I am not sure if e.g. size and w (which is probably short for width) mean the same. Do you have an idea here @breznak ?

This is the relevant section of the table, sorry for its size. The value columns show the values which are set by the respective detectors in NAB. If the cell is empty, the default value is used.

HTMCore Nupic CPP
Attribute Description Value Default Attribute Description Value Default
size Member "size" is the total number of bits in the encoded output SDR. 400 0
sparsity Member "sparsity" is the fraction of bits in the encoded output which this encoder will activate. This is an alternative way to specify the member "activeBits". 0.1 0
resolution Member "resolution" Two inputs separated by greater than, or equal to the resolution are guaranteed to have different representations. 0.9 0 resolution A floating point positive number denoting the resolution of the output representation. Numbers within [offset-resolution/2, offset+resolution/2] will fall into the same bucket and thus have an identical representation. Adjacent buckets will differ in one bit. resolution is a required parameter. max(0.001, (maxVal - minVal) / numBuckets) -
activeBits Member "activeBits" is the number of true bits in the encoded output SDR.
radius Member "radius" Two inputs separated by more than the radius have non-overlapping representations. Two inputs separated by less than the radius will in general overlap in at least some of their bits. You can think of this as the radius of the input.
Category Member "category" means that the inputs are enumerated categories. If true then this encoder will only encode unsigned integers, and all inputs will have unique / non-overlapping representations. FALSE
numBuckets 130
seed Member "seed" forces different encoders to produce different outputs, even if the inputs and all other parameters are the same. Two encoders with the same seed, parameters, and input will produce identical outputs. The seed 0 is special. Seed 0 is replaced with a random number. 0 (random) seed 42 42
w Number of bits to set in output. w must be odd to avoid centering problems. w must be large enough that spatial pooler columns will have a sufficiently large overlap to avoid false matches. A value of w=21 is typical. 21
n Number of bits in the representation (must be > w). n must be large enough such that there is enough room to select new representations as the range grows. With w=21 a value of n=400 is typical. The class enforces n > 6*w. 400
name None
offset A floating point offset used to map scalar inputs to bucket indices. The middle bucket will correspond to numbers in the range [offset - resolution/2, offset + resolution/2). If set to None, the very first input that is encoded will be used to determine the offset. None
verbosity 0
psteinroe commented 4 years ago

We could just copy htmcore results into numenta/NAB and have them re-scored.

Nice analysis!! Sounds like a good idea to try that out. I will do that after I am done with the parameter comparison.

breznak commented 4 years ago

That's a good idea to write such a comparison of params/API. I'd like the result to be published as a part of the repo here :+1:

[RDSE Encoder] While HTMCore does have size and sparsity, nupic has w, n and offset

I can help on those: 

activeBits = w
size = n

I'm not 100% sure about offset without looking, but it'd be used in resultion imho.

breznak commented 4 years ago

RDSE:

psteinroe commented 4 years ago

I can help on those:

Thanks!

Let me know if you think it'd help you and I can try to dig one up.

Let me try with the new parameter settings first. If that does not help, that might be another way to check wether its the encoder.

in the numenta detector, you see a "cheat" I've complained about:

Yes I saw that and I am calculating the resolution for htmcore encoder the same way to have a fair comparison.

The second parameter that is new for htm.core is the localAreaDensity. Nupic also has that one but uses another param numActiveColumnsPerInhArea instead to control the density of the active columns:

When using this method, as columns learn and grow their effective receptive fields, the inhibitionRadius will grow, and hence the net density of the active columns will decrease. This is in contrast to the localAreaDensity method, which keeps the density of active columns the same regardless of the size of their receptive fields.

Why was this removed from htm.core?

breznak commented 4 years ago

We could just copy htmcore results into numenta/NAB and have them re-scored.

Nice analysis!! Sounds like a good idea to try that out. I will do that after I am done with the parameter comparison.

I got the htmcore detector running with numenta/NAB.

...

psteinroe commented 4 years ago

That's a good idea to write such a comparison of params/API. I'd like the result to be published as a part of the repo here 👍

I will create a PR once I am done :) Is the table format readable or should I rather make it textual?

breznak commented 4 years ago

Is the table format readable or should I rather make it textual?

the table is good! Might decide to drop the unimportant ones (verbosity, name) for clarity. but that's just a detail.

breznak commented 4 years ago

The second parameter that is new for htm.core is the localAreaDensity. Nupic also has that one but uses another param numActiveColumnsPerInhArea instead to control the density of the active

yes, I proposed the removal. The reasons were nice, but not crucial, and now I'm suspecting this could be a lead..

https://github.com/htm-community/htm.core/pull/549/

The motivation for localAreaDensity is HTM's presumption that layers produce output (SDR) with a relatively const sparsity. (+ just code cleanup).

One the other hand, "a 'strong' column's receptive field grows" is also a good biological concept.

This would be a significant result, if one can be proven "better" (dominating) over the other.

psteinroe commented 4 years ago

This would be a significant result, if one can be proven "better" (dominating) over the other.

As the original sp also has localAreaDensity I would propose to first try out the original nupic detector with localAreaDensity instead of numActiveColumnsPerInhArea to see wether it has such an impact. I guess that would be faster than bringing it back.

psteinroe commented 4 years ago

Good News! Using the Numenta parameters with localAreaDensity of 0.1 I achieve a score of 49.9 on NAB. At least some improvement. Going to try out the swarm algorithm to optimise localAreaDensity now.

   "htmcore": {
        "reward_low_FN_rate": 54.3433173159184,
        "reward_low_FP_rate": 42.32359518695501,
        "standard": 49.96056087185252
    },

These are the params:

params_numenta_comparable = {
  "enc": {
    "value": {
        #"resolution": 0.9, calculate by max(0.001, (maxVal - minVal) / numBuckets) where numBuckets = 130
        "size": 400,
        "activeBits": 21
      },
    "time": {
        "timeOfDay": (21, 9.49),
      }
  },
  "sp": {
    # inputDimensions: use width of encoding
    "columnDimensions": 2048,
    # "potentialRadius": 999999, use width of encoding
    "potentialPct": 0.8,
    "globalInhibition": True,
    "localAreaDensity": 0.1,  # optimize this one
    "stimulusThreshold": 0,
    "synPermInactiveDec": 0.0005,
    "synPermActiveInc": 0.003,
    "synPermConnected": 0.2,
    "boostStrength": 0.0,
    "wrapAround": True,
    "minPctOverlapDutyCycle": 0.001,
    "dutyCyclePeriod": 1000,
  },
  "tm": {
    "columnDimensions": 2048,
    "cellsPerColumn": 32,
    "activationThreshold": 20,
    "initialPermanence": 0.24,
    "connectedPermanence": 0.5,
    "minThreshold": 13,
    "maxNewSynapseCount": 31,
    "permanenceIncrement": 0.04,
    "permanenceDecrement": 0.008,
    "predictedSegmentDecrement": 0.001,
    "maxSegmentsPerCell": 128,
    "maxSynapsesPerSegment": 128,
  },
  "anomaly": {
    "likelihood": {
      "probationaryPct": 0.1,
      "reestimationPeriod": 100
    }
  }
}
psteinroe commented 4 years ago

another thing that I wondered about:

In the detecor code there is a fixed param 999999999 defined when setting the infos for tm and sp. Shouldn't this be encodingWidth? Or is it the potentialRadius?

self.tm_info = Metrics([self.tm.numberOfCells()], 999999999)

breznak commented 4 years ago

param 999999999 defined when setting the infos for tm and sp. Shouldn't this be encodingWidth? Or is it the potentialRadius?

no, this is unimportant. the metric is only used of our info, does not affect the computation. It's not related to "width"/number of bits, but rather a time/steps used of EMA in the metric

psteinroe commented 4 years ago

As the original sp also has localAreaDensity I would propose to first try out the original nupic detector with localAreaDensity instead of numActiveColumnsPerInhArea to see wether it has such an impact. I guess that would be faster than bringing it back.

Alright, it seems to have a significant impact. Running the Numenta detectors with localAreaDensity of 0.1 instead of the numActiveColumnsPerInhArea results in basically the same results as with the htmcore detector.

The only question remaining is now if tuning localAreaDensity increases the score or if numActiveColumnsPerInhArea is superior in general. What do you propose how to proceed @breznak ?

Here is the code: https://github.com/steinroe/NAB/tree/test_numenta_localAreaDensity

   "numenta": {
        "reward_low_FN_rate": 52.56449422487971,
        "reward_low_FP_rate": 49.94586314087259,
        "standard": 50.82949995800923
    },
    "numentaTM": {
        "reward_low_FN_rate": 52.56449422487971,
        "reward_low_FP_rate": 49.94586314087259,
        "standard": 50.82949995800923
    },
psteinroe commented 4 years ago

Going to try out the swarm algorithm to optimise localAreaDensity now.

Used Bayesian optimization, but nevertheless these are the results: 

    "htmcore": {
        "reward_low_FN_rate": 60.852121191220256,
        "reward_low_FP_rate": 45.428862226866734,
        "standard": 55.50231971786488
    },

for the following params:

parameters_numenta_comparable = {
        "enc": {
            "value": {
                # "resolution": 0.9, calculate by max(0.001, (maxVal - minVal) / numBuckets) where numBuckets = 130
                "size": 400,
                "activeBits": 21,
                "seed": 5,  # ignored for the final run
            },
            "time": {
                "timeOfDay": (21, 9.49),
            }
        },
        "sp": {
            # inputDimensions: use width of encoding
            "columnDimensions": 2048,
            # "potentialRadius": use width of encoding
            "potentialPct": 0.8,
            "globalInhibition": True,
            "localAreaDensity": 0.025049634479368352,  # optimize this one
            "stimulusThreshold": 0,
            "synPermInactiveDec": 0.0005,
            "synPermActiveInc": 0.003,
            "synPermConnected": 0.2,
            "boostStrength": 0.0,
            "wrapAround": True,
            "minPctOverlapDutyCycle": 0.001,
            "dutyCyclePeriod": 1000,
            "seed": 5, # ignored for the final run
        },
        "tm": {
            "columnDimensions": 2048,
            "cellsPerColumn": 32,
            "activationThreshold": 20,
            "initialPermanence": 0.24,
            "connectedPermanence": 0.5,
            "minThreshold": 13,
            "maxNewSynapseCount": 31,
            "permanenceIncrement": 0.04,
            "permanenceDecrement": 0.008,
            "predictedSegmentDecrement": 0.001,
            "maxSegmentsPerCell": 128,
            "maxSynapsesPerSegment": 128,
            "seed": 5,  # ignored for the final run
        },
        "anomaly": {
            "likelihood": {
                "probationaryPct": 0.1,
                "reestimationPeriod": 100
            }
        }
    }

I created a PR htm-community/NAB/pull/25 for the updated params.

These are the logs for seed fixed to 5, where I achieved a standard score of 60 as maximum. optimization_logs

@breznak What would you suggest how to proceed from here?

breznak commented 4 years ago

Used Bayesian optimization, but nevertheless these are the results:

    "htmcore": {
        "reward_low_FN_rate": 60.852121191220256,
        "reward_low_FP_rate": 45.428862226866734,
        "standard": 55.50231971786488
    },

Wow, these are very nice results! I'm going to merge NAB.

"localAreaDensity": 0.025049634479368352, # optimize this one

Interestingly, this is what's claimed by the HTM theory (2%) as observed in the cortex.

What would you suggest how to proceed from here?

compared to the Numenta results:

I'd suggest:

breznak commented 4 years ago

I think we should review if NAB is conceptually correct (as the metric and methodology)!

See our current results (I made a tiny change to your recently updated params in NAB/fixing_anomaly branch) htmcore_nab_good

That is almost perfect! Yet in the plot: 

For result file : ../results/htmcore/artificialWithAnomaly/htmcore_art_daily_flatmiddle.csv
True Positive (Detected anomalies) : 403
True Negative (Detected non anomalies) : 0
False Positive (False alarms) : 2679
False Negative (Anomaly not detected) : 0
Total data points : 3428
S(t)_standard score : -90.17164198169499

nab_methodology_window

breznak commented 4 years ago

Q: do we want to keep comparable params & scores. Or just aim for the best score? Or both, separately?

psteinroe commented 4 years ago

this means we "beat" Numenta under the same conditions now, right?

Yes, but they still win with numActiveColumnsPerInhArea set.

your Bayesian opt is multi-parametric? or just considering the select (localArea) param?

For this test I just optimised the localAreaDensity keeping the others constant but it can be multi parametric.

you try to tune the current score wrt the other params (there's lots of local optima and parameters are interleaved)

Alright, I will work on that. Do you have a feeling about which params may be important / optimizable?

I'll revert and reintroduce the numActiveColsPerInhArea param (alternative to localArea)

Perfect, thanks for your work!

we should get NAB operational with our optimization framework for multi-param opt. So we can brute-force the parameter space.

The "problem" is that your framework calls the optimization function in parallel, so my current setup with bayesian optimisation where I simply write to and read from a params.json file won't work. I will think about how to set that up that and come back to you as soon as I have a solution that works.

either just the Plot summary FP/FN is wrong ?

My gut says this may be the problem, as the scoring results seems fine. We should debug this.

Q: do we want to keep comparable params & scores. Or just aim for the best score? Or both, separately?

I would say we just aim for the best score, as even the bug fixes in this fork may influence the best param setting. I don't know if its useful to keep a second set of params as a comparison between the two would be only fair if both use the best possible params.

breznak commented 4 years ago

Yes, but they still win with numActiveColumnsPerInhArea set.

ok, I'll be able to do that. I got stuck today playing with the params.

but it can be multi parametric.

great. Good if you can try adding more params (below for hints on which). The parameter space is with many local optima.

[sorry, I'll need to do some other work done, brb in cca 2-3hrs]

Just quickly:

Let me know what do you think should be a priority:

psteinroe commented 4 years ago

great. Good if you can try adding more params (below for hints on which). The parameter space is with many local optima.

Alright, I will work on that.

Let me know what do you think should be a priority:

I would say reintroducing the numActiveColumns as (1) the multi param optimisation can right now also be done with other tools such as the bayesian optimisation and (2) I could also have a look into getting the swarm optimisation to run with NAB, while reintroducing the param is something where you are considerably faster than me. Does that work for you?

breznak commented 4 years ago

while reintroducing the param is something where you are considerably faster

I'll do that tonight.

I could also have a look into getting the swarm optimisation to run with NAB

That would be great. Ideally if NAB can run with the current opt framework. The requirement is that "NAB" can be run as a single method (que opt's requirements) and returns a score.

breznak commented 4 years ago

@steinroe you can now use branch sp_reintroduce_numActiveColumnsPerInhArea #797 for comparison with the (re-)new param.

I'll try to stir again the waters in Forums on the topic if one is better than the other(?). I will have a look at param opt for NAB, because only than we can compare the params "by performance".

psteinroe commented 4 years ago

@breznak thanks for your work!

I am almost done with enabling swarm param opt with NAB. I just need 2 more hours.

breznak commented 4 years ago

enabling swarm param opt with NAB. I just need 2 more hours.

awesome! take your time.

Further on this front we have in progress #796 for easily running on clusters.

breznak commented 4 years ago

ok, if you can enable the param optimization for NAB, I'll now be able to start looking into "smart params" #536 aka

psteinroe commented 4 years ago

Okay this is what I got regarding param optimisation on NAB:

Idea and Setup

My goal was to not touch the nab codebase at all. Instead, I did a minor change to the htmcore detector (same can be done for any detector) where I read the params from a json file and set the bool params to True (as e.g. the swarm algorithm cannot deal with bool). You can see it in this branch: https://github.com/steinroe/NAB/tree/run_as_function

The second thing I added was a simple Dockerfile which installs NAB and, when started, runs NAB for the htmcore detector (again, this can be done with any detector)

FROM python:3.7

COPY . /NAB

WORKDIR /NAB

RUN pip install . --user --extra-index-url https://test.pypi.org/simple/

CMD python run.py -d htmcore --skipConfirmation --detect --optimize --score --normalize

I put this in the root of the NAB repo and build the image using docker build -t optimize-htmcore-nab:latest ..

Now we can use that image to run htmcore on NAB.

NOTE: If using Docker Desktop, don't forget to grant more resources to it.

Optimization

Swarm

For the optimization, e.g. with the swarm algorithm, one can use the following script. The idea is basically to create a container from the image, put the params.json file inside of the container, start the container and run the detector on NAB and then copy the results file from the container on the host, read it and return the score. If you can think of any better method please let me know, as this feels kinda hacky.

NOTE: On MacOS, you have to export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES for docker.from_env() not to call an error (check this link for details):

EDIT: Run this script with python -m htm.optimization.ae -n 3 --memory_limit 4 -v --swarming 100 optimize_anomaly_swarm.py (or whatever resources you want to grant)

import subprocess
import sys
import os
import json
import shutil
from pathlib import Path
import docker

default_parameters = {
    "enc": {
        "value": {
            # "resolution": 0.9, calculate by max(0.001, (maxVal - minVal) / numBuckets) where numBuckets = 130
            "size": 400,
            "activeBits": 21,  # results very sensitive to the size/active bits in the input encoders
            "seed": 5,
        },
        "time": {
            "timeOfDay": (21, 9.49),
        }
    },
    "sp": {
        # inputDimensions: use width of encoding
        "columnDimensions": 2048,
        # "potentialRadius": use width of encoding
        "potentialPct": 0.8,
        #"globalInhibition": True, always true (set in detector) as swarm cannot work with bool
        "localAreaDensity": 0.025,  # optimize this one
        "stimulusThreshold": 2,
        "synPermInactiveDec": 0.001,
        "synPermActiveInc": 0.006,
        "synPermConnected": 0.5,  # this shouldn't make any effect, keep as intended by Connections
        "boostStrength": 0.0,  # so far, boosting negatively affects results. Suggest leaving OFF (0.0)
        #"wrapAround": True, always true (set in detector) as swarm cannot work with bool
        "minPctOverlapDutyCycle": 0.001,
        "dutyCyclePeriod": 1000,
        "seed": 5,
    },
    "tm": {
        # "columnDimensions": 2048, #must match SP
        "cellsPerColumn": 32,
        "activationThreshold": 20,
        "initialPermanence": 0.24,
        "connectedPermanence": 0.5,
        "minThreshold": 13,
        "maxNewSynapseCount": 31,
        "permanenceIncrement": 0.04,
        "permanenceDecrement": 0.008,
        "predictedSegmentDecrement": 0.001,
        "maxSegmentsPerCell": 128,
        "maxSynapsesPerSegment": 128,
        "seed": 5,
    },
    "anomaly": {
        "likelihood": {
            "probationaryPct": 0.1,
            "reestimationPeriod": 100
        }
    }
}

def main(parameters=default_parameters, argv=None, verbose=True):
    # get client
    client = docker.from_env()
    print(client)

    # create container
    print('Creating Container')
    container = client.containers.create(image='optimize-htmcore-nab:latest')

    # make dir for container
    print('Making Dir')
    Path(os.path.join('temp', container.id)).mkdir(parents=True, exist_ok=True)

    # write params to json in that dir
    print('Writing Params')
    with open(os.path.join('temp', container.id, 'params.json'), 'w') as outfile:
        json.dump(default_parameters, outfile)

    # copy file to container
    print('Copy file to container')
    cmd = 'docker cp ./temp/' + container.id + '/params.json ' + container.id + ':NAB/nab/detectors/htmcore/params.json'
    subprocess.check_call(cmd, shell=True)

    # start container
    print('Starting container')
    container.start()
    container.wait()

    # copy results file into temp directory with container id in path
    cmd = 'docker cp ' + container.id + ':NAB/results/final_results.json ./temp/' + container.id + '/final_results.json'
    subprocess.check_call(cmd, shell=True)

    # read and return score from results file
    with open(os.path.join('temp', container.id, 'final_results.json')) as json_file:
        results = json.load(json_file)
        score = results.get('htmcore').get('standard')

    # remove container
    container.remove()

    # remove temp dir
    shutil.rmtree(os.path.join('temp', container.id))

    return score

if __name__ == '__main__':
    sys.exit(main() < 0.7)

Bayesian Optimization

The same thing can be done for any optimization function such as Bayesian Optimization, which I have had some good experiences with. As this one does not call the target function in parallel, this could be done without docker on the local file system, too. Here the script for using it with docker as above:

import subprocess
import os
import json
import docker
import shutil
from pathlib import Path
from bayes_opt import BayesianOptimization
from bayes_opt.logger import JSONLogger
from bayes_opt.event import Events
from bayes_opt.util import load_logs

default_parameters = {
    "enc": {
        "value": {
            # "resolution": 0.9, calculate by max(0.001, (maxVal - minVal) / numBuckets) where numBuckets = 130
            "size": 400,
            "activeBits": 21,  # results very sensitive to the size/active bits in the input encoders
            "seed": 5,
        },
        "time": {
            "timeOfDay": (21, 9.49),
        }
    },
    "sp": {
        # inputDimensions: use width of encoding
        "columnDimensions": 2048,
        # "potentialRadius": use width of encoding
        "potentialPct": 0.8,
        #"globalInhibition": True, always true (set in detector) as swarm cannot work with bool
        "localAreaDensity": 0.025,  # optimize this one
        "stimulusThreshold": 2,
        "synPermInactiveDec": 0.001,
        "synPermActiveInc": 0.006,
        "synPermConnected": 0.5,  # this shouldn't make any effect, keep as intended by Connections
        "boostStrength": 0.0,  # so far, boosting negatively affects results. Suggest leaving OFF (0.0)
        #"wrapAround": True, always true (set in detector) as swarm cannot work with bool
        "minPctOverlapDutyCycle": 0.001,
        "dutyCyclePeriod": 1000,
        "seed": 5,
    },
    "tm": {
        # "columnDimensions": 2048, #must match SP
        "cellsPerColumn": 32,
        "activationThreshold": 20,
        "initialPermanence": 0.24,
        "connectedPermanence": 0.5,
        "minThreshold": 13,
        "maxNewSynapseCount": 31,
        "permanenceIncrement": 0.04,
        "permanenceDecrement": 0.008,
        "predictedSegmentDecrement": 0.001,
        "maxSegmentsPerCell": 128,
        "maxSynapsesPerSegment": 128,
        "seed": 5,
    },
    "anomaly": {
        "likelihood": {
            "probationaryPct": 0.1,
            "reestimationPeriod": 100
        }
    }
}

def target_func(localAreaDensity):
    # get params
    my_params = default_parameters
    my_params['sp']['localAreaDensity'] = localAreaDensity

    # get client
    client = docker.from_env()

    # create container
    container = client.containers.create(image='optimize-htmcore-nab:latest')

    # make dir for container
    Path(os.path.join('temp', container.id)).mkdir(parents=True, exist_ok=True)

    # write params to json in that dir
    with open(os.path.join('temp', container.id, 'params.json'), 'w') as outfile:
        json.dump(my_params, outfile)

    # copy file to container
    cmd = 'docker cp ./temp/' + container.id + '/params.json ' + container.id + ':NAB/nab/detectors/htmcore/params.json'
    subprocess.check_call(cmd, shell=True)

    # start container
    container.start()
    container.wait()

    # copy results file into temp directory with container id in path
    cmd = 'docker cp ' + container.id + ':NAB/results/final_results.json ./temp/' + container.id + '/final_results.json'
    subprocess.check_call(cmd, shell=True)

    # read and return score from results file
    with open(os.path.join('temp', container.id, 'final_results.json')) as json_file:
        results = json.load(json_file)
        score = results.get('htmcore').get('standard')

    # remove container
    container.remove()

    # remove temp dir
    shutil.rmtree(os.path.join('temp', container.id))

    return score

def optimize_local_area_density():
    # define bounds for the params you want to optimize. Can be multivariate. Check https://github.com/fmfn/BayesianOptimization on how to
    bounds = {
        'localAreaDensity': (0.01, 0.15),
    }

    optimizer = BayesianOptimization(
        f=target_func,
        pbounds=bounds,
        random_state=1,
    )

    # We can start from saved logs
    if os.path.isfile('./local_area_density_optimization_logs_base.json'):
        print('Loading Logs...')
        load_logs(optimizer, logs=["./local_area_density_optimization_logs_base.json"]);

    # The new log file to write to
    logger = JSONLogger(path="./local_area_density_optimization_logs.json")
    optimizer.subscribe(Events.OPTIMIZATION_STEP, logger)

    # If you want to guide the optimization process
    val = 0.02
    while val <= 0.04:
        print('Adding', val)
        optimizer.probe(
            params={
                'localAreaDensity': val,
            },
            lazy=True,
        )
        val = round(val + 0.001, 3)

    print('Starting optimization...')

    optimizer.maximize(
        init_points=20,
        n_iter=50,
    )

    print(optimizer.max)

if __name__ == "__main__":
    optimize_local_area_density()
psteinroe commented 4 years ago

FYI: Running on my MacBook (with limited resources) I am at a max score of 60.907:

EDIT: Note: seed is not used.

New global best score 60.907.
{'anomaly': {'likelihood': {'probationaryPct': 0.10287247901390377,
                            'reestimationPeriod': 105}},
 'enc': {'time': {'timeOfDay': (19, 10.230927402319585)},
         'value': {'activeBits': 20, 'seed': 4, 'size': 433}},
 'sp': {'boostStrength': 0.0,
        'columnDimensions': 1992,
        'dutyCyclePeriod': 997,
        'localAreaDensity': 0.02315293328723999,
        'minPctOverlapDutyCycle': 0.0010876327134838708,
        'potentialPct': 0.7268162250552015,
        'seed': 6,
        'stimulusThreshold': 1,
        'synPermActiveInc': 0.005756214837461844,
        'synPermConnected': 0.5255368572159246,
        'synPermInactiveDec': 0.0010412298597037122},
 'tm': {'activationThreshold': 19,
        'cellsPerColumn': 33,
        'connectedPermanence': 0.4732123724546982,
        'initialPermanence': 0.23075451521239015,
        'maxNewSynapseCount': 31,
        'maxSegmentsPerCell': 135,
        'maxSynapsesPerSegment': 120,
        'minThreshold': 12,
        'permanenceDecrement': 0.007285708940843109,
        'permanenceIncrement': 0.04329160631961319,
        'predictedSegmentDecrement': 0.0010265038754633969,
        'seed': 5}}
breznak commented 4 years ago

Great work on the optimization! I'm going through it and will be trying it later.

A couple of questions, remarks

The second thing I added was a simple Dockerfile which installs NAB and, when started, runs NAB for the htmcore detector

do we need that? We can run htmcore detector on community/NAB as well as numenta/NAB.

For the optimization, e.g. with the swarm algorithm, one can use the following script. The idea is basically to create a container from the image, put the params.json file inside of the container, start the container and run the detector on NAB and then copy the results file from the container on the host, read it and return the score.

this seems smart, a least intruding solution.

The same thing can be done for any optimization function such as Bayesian Optimization, which I have had some good experiences with.

Later on, I'd like to achieve the following

I am at a max score of 60.907:

I'll try to run this setup as well.

psteinroe commented 4 years ago

do we need that? We can run htmcore detector on community/NAB as well as numenta/NAB.

The whole point of using docker is that NAB heavily depends on using the file system, e.g. for writing out the results. The optimization framework runs multiple scorings in parallel, so we would have conflicting results when running on our host system. With docker, we are able to prevent that. Or am I misinterpreting your question somehow?

merge your Bayes optimizer into the existing htm.optimization "framework" (currently has PSO, BruteForce)

I can have a look into that 👍

can the trick with NAB be used/automated to be used for the above framework?

I guess so, as it works with the PSO algorithm already. EDIT: Sorry, I forgot to add that the PSO script above runs on your framework, so you run python -m htm.optimization.ae -n 3 --memory_limit 4 -v --swarming 100 optimize_anomaly_swarm.py to start the optimization.

what I'm missing is how to run multi-parametric optimization. Could you please post an example of that?

Sure. With Bayesian Optimization you need to set bounds for each parameter, but you can use as many as you want, e.g. pbounds = {'x': (2, 4), 'y': (-3, 3)} to optimise x and y within (2, 4) and (-3, 3) respectively. See the repo README of the python implementation for details. If we can decide on useful bounds, I can setup a script for optimising all / a larger subset of the params.

EDIT: Added link to repo

psteinroe commented 4 years ago

About integrating Bayesian Opt into your optimization framework: I think the main challenge is that Bayes Opt requires bounds to be set while PSO as well as GridSearch only require a "starting point". I think such a change would require a change in the framework itself to be able to add bounds instead of the default_parameter dict that is currently used.

breznak commented 4 years ago

I like this, this is a really nice trick to get it done with NAB.

The optimization framework runs multiple scorings in parallel, so we would have conflicting results when running on our host system. With docker, we are able to prevent that.

good, bcs I thought it didn't run in parallel. We could do something like writing to /tmp/$PID/results.json but this already works good enough, so :+1:

Sorry, I forgot to add that the PSO script above runs on your framework, so you run python -m htm.optimization.ae `

great. I was thinking you're developing from scatch. It's better that you could be using our existing tools!

Sure. With Bayesian Optimization you need to set bounds for each parameter, but you can use as many as you want, e.g. pbounds = {'x': (2, 4), 'y': (-3, 3)} to optimise x and y

even better! this looks really convenient.

If we can decide on useful bounds, I can setup a script for optimising all / a larger subset of the params

I'll try working on constraining the params, and along with that we can come up with a subset and its ranges. For some params, we'd need some control over the granuity etc, eg. bounds for numColumns would be something as [1024, 8192], but we're not interested in 1023 etc, rather than stride by 1024. But we can hack such cases in the detector manually, as 

if x not in {1024, 2048, 4096, ...}:
  raise

I did a minor change to the htmcore detector (same can be done for any detector) where I read the params from a json file and set the bool params to True (as e.g. the swarm algorithm cannot deal with bool

psteinroe commented 4 years ago

good, bcs I thought it didn't run in parallel. We could do something like writing to /tmp/$PID/results.json but this already works good enough, so

We would probably still get interferences when the detector reads the params file. I think locking the parallel runs of NAB into a container is the cleanest and safest method. The only downside is probably performance, as we have to give docker host a fair share of our host resources.

For some params, we'd need some control over the granuity etc, eg. bounds for numColumns would be something as [1024, 8192], but we're not interested in 1023 etc, rather than stride by 1024. But we can hack such cases in the detector manually, as

Bayesian Optimization does that automatically by randomly choosing some parameters in the beginning and from time to time. Additionally, we can probe some settings (lets say 1024, 2048, ... 8192) so the algorithm knows of the effect these settings have. Its quite good in avoiding local maxima out of the box (at least to my experience)

could you make a PR with this to our NAB? Ie a toggle useOptimization and if set, it reads the params from the file. To eliminate one more step for using your idea

Sure, will do.

I'll get to testing this workflow/guide a bit later. Do you think it could be further automated? say a script, or a docker (with docker)?

I think we could put the image on docker hub so the user does not have to build the image. With a minor change to the script we could pull the image from docker hub. Then, The workflow would be as follows:

For using the htm.core optimization framework

Requirements: Docker Desktop, htmcore, requirements of script such as docker

For using the pure python script with bayesian opt

Requirements: Docker Desktop, requirements of script such as docker and bayesian opt

EDIT: Where to put the scripts? Another repo? Or just as branch?

breznak commented 4 years ago

We would probably still get interferences when the detector reads the params file. I think locking the parallel runs of NAB into a container is the cleanest and safest method

ok, I agree. We need to get there with minimal manhours, so this works as intended.

Additionally, we can probe some settings (lets say 1024, 2048, ... 8192) so the algorithm knows of the effect these settings have.

interesting. nice if we can "hint" to try these datapoints.

I think we could put the image on docker hub so the user does not have to build the image. With a minor change to the script we could pull the image from docker hub.

maybe not necessarily DockerHub, but a docker file in the repo (NAB) would be great :+1:

htmcore is already dependency of (our) NAB, so that's no problem. My (optimal) idea was a script/dockerfile that the user can run (modify which params should be probed, and provide data-as in NAB) and it runs the optimization and outputs the found set of params & score.

EDIT: Where to put the scripts? Another repo? Or just as branch?

I'd make this part of community/NAB. So submit as PR/branch, and we'll merge directly to master (this enhancement is unrelated to "fixing htmcore scores")

psteinroe commented 4 years ago

Sounds good to me! Do we want to enable optimisation on NAB in general or only for htmcore?

breznak commented 4 years ago

Do we want to enable optimisation on NAB in general or only for htmcore?

I'd say minimal changes first - we need to solve topic of this issue. So just HTMcore for now.

Because I'm still unsure how to proceed with https://github.com/htm-community/NAB/issues/21

but

psteinroe commented 4 years ago

I'd say minimal changes first - we need to solve topic of this issue. So just HTMcore for now.

Alright, I will update the optimisation PR with a proposal on how to make it as user friendly as possible.

Zbysekz commented 4 years ago

CC @Zbysekz as the author of HTMpandaVis, do you think you could help us debugging the issue? I'd really appreciate that!

TL;DR: minor parameters and changes were surely done to htm.core. Compared to Numenta's Nupic, our results on NAB really suck now. I'm guessing it should be matter of incorrect params.

There look into the representations with the visualizer would be really helpful. More info here (and linked posts) htm-community/NAB#15

Hello, sorry for the delay. Would it help, if i modify HTMdetector in steinroe repo to use PandaVis and see particular steps live ?

I am not sure if that is the script that you guys are using to run it.

breznak commented 4 years ago

hi Zbysekz!

HTMdetector in steinroe repo to use PandaVis and see particular steps live ?

that'd be great step! Actually we're merging and currently we run community/NAB. The file is in nab/detectors/htmcore/htmcore_detector.py

EDIT: