Open psteinroe opened 4 years ago
htm.core is a rework of NuPic so that it will work. I think you will find that nearly everything you want to do with NuPic should be covered by htm.core. NuPic on the otherhand is basically broken.
@breznak will have to address performance of htm.core but I expect it to be comparable (or perhaps even better) than NuPic.
@steinroe glad for your interest!
I am wondering wether I should use htm.core or nupic
In terms of feature-fullness and active support, htm.core has now much surpased its parent, numenta's nupic(.core).
Is there any comparison in terms of performance? [in anomaly detection]
yes, this is an open problem. Theoretically htm.core should be better/or same/or slightly worse to Numenta's nupic. We did a couple of improvements as well as a couple of "regressions" in the name of biological plausibility. You could start by looking at the Changelog.md, the git diff is already too wild.
Yet, the so-far performance on NAB of htm.core is much worse. (Note, such regression is not observed on "sine", or "hotgym" data.). My gut feeling is it's just "some parameter that is off".
I won't be able to dive deep into htm to improve the implementation...
you would'nt have to dive deep, or we'd be here to help you.. So my recommendation would be: Give htm.core a trial period (say 1-5 weeks) and try to find where the cuplit is. Doing so would help the community and would be a significant result for your thesis. I could help you in trialing down the process in NAB so we can locate the err.
@breznak and @dkeeney thanks for the quick replies! I started by analysing the difference in the API and the respective parameters.
Regarding the Spatial Pooler, htm.core differentiates from nupic by three parameters: localAreaDensity
, potentialRadius
and stimulusThreshold
. I tried some Bayesian Optimization to get an idea of what setting for these 3 might work well but only got a score of around 31 with {"localAreaDensity": 0.02149599864266627, "potentialRadius": 4148629.7213459704, "stimulusThreshold": 3.640494932522697}
. All other settings were the same as in nupic. Do you have a gut feeling on what ranges would make sense? From the logs I would say localAreaDensity should be <0.09 and stimulusThreshold <10, but still the score is very bad. Any idea on that?
The second thing I did was a direct comparison of the anomaly scores yielded by nupic and by htm.core on the same datasets. You can find the output in the attached pdf. There are two things that aren’t right:
What would you say, is it rather because of the params of the encoders or params of the algorithm itself? Also, there are differences in the parameters of the RSDE encoder. Only the resolution parameter is also used in nupic. Or did I miss something here?
Very nice analysis @steinroe !! :+1: Let me answer your ideas by parts.
started by analysing the difference in the API
first, let me correct my mistake, the file I wanted to point you to is API_CHANGELOG.md
About the API (and implementation differences): From top of my head...
numentaTM
detector. anomaly
transparently. AnomalyLikelihood
(I think you can change it to raw). I'm quite sure our c++ AnomalyLikelihood is broken (well, untested at best), there's also the ./py/
Likelihood, which should be also just py3 port of the Nupic's. tried some Bayesian Optimization to get an idea of what setting for these 3 might work well
cool. If you don't have own framework for optimization, I suggest looking at ./py/htm/optimization/
Do you have a gut feeling on what ranges would make sense?
This is tricky, but can be computed rather precisely. I'll have to look at this deeper again... You can see my trials at community/NAB on this very same problem: https://github.com/htm-community/NAB/pull/15
stimulusThreshold
This is a number specifying the minimum number of synapses that must be active in order for a column to turn ON. The purpose of this is to prevent noisy input from activating columns.
So this depends on expected avg number of ON bits from encoder, number of synapses and range of input field each dendrite covers, how noisy is the problem.
Anything >=1
is imho a good starting point.
potentialRadius
This parameter deteremines the extent of the input that each column can potentially be connected to. This can be thought of as the input bits that are visible to each column, or a 'receptive field' of the field of vision. A large enough value will result in global coverage, meaning that each column can potentially be connected to every input bit. This parameter defines a square (or hyper square) area: a column will have a max square potential pool with sides of length(2 * potentialRadius + 1)
, rounded to fit into each dimension.
depends on size
of the encoder/encoding. And whether you want the columns to act as local/global approximators.
= size aka Inf aka "act global".
"potentialRadius": 4148629.7213459704
Seems your optimizer prefered that variant. (note, it might thus also just got stuck in a local optima). I'd try something as "global" and "25% of input field" as reasonable defaults.
localAreaDensity
The desired density of active columns within a local inhibition area (the size of which is set by the internally calculated inhibitionRadius, which is in turn determined from the average size of the connected potential pools of all columns). The inhibition logic will insure that at most N columns remain ON within a local inhibition area, where N = localAreaDensity * (total number of columns in inhibition area) Default: 0.05 (5%)
This is lower-bounded by SP's size aka numColumns. (density * numCols >= some meaningful min value).
Its value determines TM's function and TM's variant of "stimulusThreshold" (it's called something different - activationThreshold
and minThreshold
).
2-15% seems a reasonable value to me.
The second thing I did was a direct comparison of the anomaly scores yielded by nupic and by htm.core on the same datasets.
great graphs!, some notes on that later.
This leads me to a decomposition. The chain is:
Encoder -> SP -> TM -> Anomaly
Ideally, we'd start from the top to verify all our implementations.
1) It seems like the date encoding is not right or not used at all. The output shows no peak in e.g. the no jump dataset.
actually, there's a small drop at that time. All the graphs are on the same params? Looks that only the "flatmiddle" dataset htmcore results can detect anything meaningful. So the decision value of the others is questionable. But it could be in settings of the datetime encoder, ratio of datetime / RDSE size, insensitivity of sp/tm.
(2) The Scalar Encoding seems off, as e.g. in jumpsup a significant rise is not reflected at all. This could also be due to some parameters of the models I guess.
The RDSE encoder is rather well tested, so I wouldn't expect a hidden bug there. Maybe unusable default params. It seems to me the the HTM didn't learn at all in most cases (except the "flatmiddle")
What would you say, is it rather because of the params of the encoders or params of the algorithm itself?
I think it'd be in the params. But cannot tell whether enc/sp/tm. It's all tied together.
Great job investigating so far. I'll be looking at it tonight as well. Please let us know if you find something else or if we can help explain something. Cheers,
@steinroe I've added a "dataset" with only the artificial data, the results look rather good.
Update:
When running on only the artificial/synthetic labels, our results are quite good:
python run.py -d htmcore --detect --score --optimize --normalize --windowsFile labels/synthetic.json -n 8
htmcore detector benchmark scores written to /mnt/store/devel/HTM/NAB/results/htmcore/htmcore_reward_low_FN_rate_scores.csv
Running score normalization step
Final score for 'htmcore' detector on 'standard' profile = 84.84
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 84.36
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 88.37
Final scores have been written to /mnt/store/devel/HTM/NAB/results/final_results.json.
PS: also I'd suggest using the following branch, it has some nice prints and comments to it. https://github.com/htm-community/NAB/pull/15
CC @Zbysekz as the author of HTMpandaVis, do you think you could help us debugging the issue? I'd really appreciate that!
TL;DR: minor parameters and changes were surely done to htm.core. Compared to Numenta's Nupic, our results on NAB really suck now. I'm guessing it should be matter of incorrect params.
There look into the representations with the visualizer would be really helpful. More info here (and linked posts) https://github.com/htm-community/NAB/pull/15
Btw, I've made fixes to Jupyter plotter, see NAB/scripts/README
.
These are some figures:
HTMcore:
For result file : ../results/htmcore/artificialWithAnomaly/htmcore_art_daily_flatmiddle.csv
True Positive (Detected anomalies) : 403
True Negative (Detected non anomalies) : 0
False Positive (False alarms) : 2679
False Negative (Anomaly not detected) : 0
Total data points : 3428
S(t)_standard score : -90.17164198169499
Numenta:
For result file : ../results/numenta/artificialWithAnomaly/numenta_art_daily_flatmiddle.csv
True Positive (Detected anomalies) : 1
True Negative (Detected non anomalies) : 4027
False Positive (False alarms) : 0
False Negative (Anomaly not detected) : 0
Total data points : 4032
S(t)_standard score : 0.4999963147227
EDIT: fixed uploaded imgs
Note: both images look very similar. (I don't know why numenta scores look that bad here,must be some bug)
@breznak Thanks for the detailed review!
You can turn that off too. HtmcoreDetector does not use that.
Sorry, I forgot to mention that I turned that off for both detectors - HTMCore and Nupic.
cool. If you don't have own framework for optimization, I suggest looking at ./py/htm/optimization/
I saw that right after I was done with my optimization. Going to try yours too very soon!
Thanks for the information on the parameters. I guess from your explanation this is very likely some parameters off. I will look into the default parameters of both to get complete comparison of what could be different.
I setup a repo with the stuff I used to plot both detectors against each other, as I didn't want to bother with all the NAB stuff for that. For the HTMCore detector I actually used yours from htm-community/NAB#15. For nupic, I setup a little server with a (very very) simple API to use the original nupic detector. I just removed the base class so I put the min/max stuff directly into the detector and removed the spatial anomaly detector in both. Feel free to play around: https://github.com/steinroe/htm.core-vs-nupic
When running on only the artificial/synthetic labels, our results are quite good:
That appears weird to me... I am going to look into the outputs of the scores in detail, thanks for that!!!!
There look into the representations with the visualizer would be really helpful.
True! I guess we could be able to find the differences in the param settings better.
When running on only the artificial/synthetic labels, our results are quite good:
That appears weird to me... I am going to look into the outputs of the scores in detail, thanks for that!!!!
I've confirmed that on the "nojump" data, the err still persists. HTMcore does not detect anything, numenta does have a peak.
This could be 2 things:
For nupic, I setup a little server with a (very very) simple API to use the original nupic detector.
This looks good! I might be interested for that for community/NAB to provide (old) numenta detectors. Numenta/NAB switched to docker for the old py2 support, and this seems a good way to interface that!
I'll try your repo, thanks :+1:
I don't understand the plotters/summary code:
if Error is None:
TP,TN,FP,FN = 0,0,0,0
print(standard_score)
for x in standard_score:
if x > 0:
TP +=1
elif x == 0:
TN +=1
elif x == -0.11:
FP +=1
elif x == -1:
FN +=1
print("For result file : " + result_file)
print("True Positive (Detected anomalies) : " + str(TP))
print("True Negative (Detected non anomalies) : " + str(TN))
print("False Positive (False alarms) : " + str(FP))
print("False Negative (Anomaly not detected) : " + str(FN))
print("Total data points : " + str(total_Count))
print(detector_profile+" score : "+str(np.sum(standard_score)))
else:
print(Error)
print("Run from beginng to clear Error")
which gives
[-0.11 -0.11 -0.11 ... -0.10999362 -0.1099937
-0.10999377]
For result file : ../results/htmcore/artificialWithAnomaly/htmcore_art_daily_nojump.csv
True Positive (Detected anomalies) : 403
True Negative (Detected non anomalies) : 0
False Positive (False alarms) : 2787
False Negative (Anomaly not detected) : 0
Total data points : 3428
S(t)_standard score : -90.17200961167651
But according to the img and raw_anomaly, there should be only a few FP!
Bottom line, is our community/NAB (scorer) correct? We could just copy htmcore results into numenta/NAB and have them re-scored.
Sorry for the late reply. I am currently in progress of doing an in-depth comparison between the parameters and there are definitely some differences. You can find the table in the Readme here: https://github.com/steinroe/htm.core-vs-nupic
While most differences could be easily resolved, I need your input on the params of the RDSE Encoder. While HTMCore does have size and sparsity, nupic has w, n and offset set with default parameters. The descriptions seem similar, however I am not sure if e.g. size and w (which is probably short for width) mean the same. Do you have an idea here @breznak ?
This is the relevant section of the table, sorry for its size. The value columns show the values which are set by the respective detectors in NAB. If the cell is empty, the default value is used.
HTMCore | Nupic CPP | ||||||
---|---|---|---|---|---|---|---|
Attribute | Description | Value | Default | Attribute | Description | Value | Default |
size | Member "size" is the total number of bits in the encoded output SDR. | 400 | 0 | ||||
sparsity | Member "sparsity" is the fraction of bits in the encoded output which this encoder will activate. This is an alternative way to specify the member "activeBits". | 0.1 | 0 | ||||
resolution | Member "resolution" Two inputs separated by greater than, or equal to the resolution are guaranteed to have different representations. | 0.9 | 0 | resolution | A floating point positive number denoting the resolution of the output representation. Numbers within [offset-resolution/2, offset+resolution/2] will fall into the same bucket and thus have an identical representation. Adjacent buckets will differ in one bit. resolution is a required parameter. | max(0.001, (maxVal - minVal) / numBuckets) | - |
activeBits | Member "activeBits" is the number of true bits in the encoded output SDR. | ||||||
radius | Member "radius" Two inputs separated by more than the radius have non-overlapping representations. Two inputs separated by less than the radius will in general overlap in at least some of their bits. You can think of this as the radius of the input. | ||||||
Category | Member "category" means that the inputs are enumerated categories. If true then this encoder will only encode unsigned integers, and all inputs will have unique / non-overlapping representations. | FALSE | |||||
numBuckets | 130 | ||||||
seed | Member "seed" forces different encoders to produce different outputs, even if the inputs and all other parameters are the same. Two encoders with the same seed, parameters, and input will produce identical outputs. The seed 0 is special. Seed 0 is replaced with a random number. | 0 (random) | seed | 42 | 42 | ||
w | Number of bits to set in output. w must be odd to avoid centering problems. w must be large enough that spatial pooler columns will have a sufficiently large overlap to avoid false matches. A value of w=21 is typical. | 21 | |||||
n | Number of bits in the representation (must be > w). n must be large enough such that there is enough room to select new representations as the range grows. With w=21 a value of n=400 is typical. The class enforces n > 6*w. | 400 | |||||
name | None | ||||||
offset | A floating point offset used to map scalar inputs to bucket indices. The middle bucket will correspond to numbers in the range [offset - resolution/2, offset + resolution/2). If set to None, the very first input that is encoded will be used to determine the offset. | None | |||||
verbosity | 0 |
We could just copy htmcore results into numenta/NAB and have them re-scored.
Nice analysis!! Sounds like a good idea to try that out. I will do that after I am done with the parameter comparison.
That's a good idea to write such a comparison of params/API. I'd like the result to be published as a part of the repo here :+1:
[RDSE Encoder] While HTMCore does have size and sparsity, nupic has w, n and offset
I can help on those:
activeBits = w
size = n
I'm not 100% sure about offset without looking, but it'd be used in resultion imho.
RDSE:
I can help on those:
Thanks!
Let me know if you think it'd help you and I can try to dig one up.
Let me try with the new parameter settings first. If that does not help, that might be another way to check wether its the encoder.
in the numenta detector, you see a "cheat" I've complained about:
Yes I saw that and I am calculating the resolution for htmcore encoder the same way to have a fair comparison.
The second parameter that is new for htm.core is the localAreaDensity
. Nupic also has that one but uses another param numActiveColumnsPerInhArea
instead to control the density of the active columns:
When using this method, as columns learn and grow their effective receptive fields, the inhibitionRadius will grow, and hence the net density of the active columns will decrease. This is in contrast to the localAreaDensity method, which keeps the density of active columns the same regardless of the size of their receptive fields.
Why was this removed from htm.core?
We could just copy htmcore results into numenta/NAB and have them re-scored.
Nice analysis!! Sounds like a good idea to try that out. I will do that after I am done with the parameter comparison.
I got the htmcore detector running with numenta/NAB.
...
That's a good idea to write such a comparison of params/API. I'd like the result to be published as a part of the repo here 👍
I will create a PR once I am done :) Is the table format readable or should I rather make it textual?
Is the table format readable or should I rather make it textual?
the table is good! Might decide to drop the unimportant ones (verbosity, name) for clarity. but that's just a detail.
The second parameter that is new for htm.core is the localAreaDensity. Nupic also has that one but uses another param numActiveColumnsPerInhArea instead to control the density of the active
yes, I proposed the removal. The reasons were nice, but not crucial, and now I'm suspecting this could be a lead..
https://github.com/htm-community/htm.core/pull/549/
The motivation for localAreaDensity
is HTM's presumption that layers produce output (SDR) with a relatively const sparsity. (+ just code cleanup).
One the other hand, "a 'strong' column's receptive field grows" is also a good biological concept.
numActiveColumnsPerInhArea
?
This would be a significant result, if one can be proven "better" (dominating) over the other.
This would be a significant result, if one can be proven "better" (dominating) over the other.
As the original sp also has localAreaDensity I would propose to first try out the original nupic detector with localAreaDensity instead of numActiveColumnsPerInhArea to see wether it has such an impact. I guess that would be faster than bringing it back.
Good News! Using the Numenta parameters with localAreaDensity of 0.1 I achieve a score of 49.9 on NAB. At least some improvement. Going to try out the swarm algorithm to optimise localAreaDensity now.
"htmcore": {
"reward_low_FN_rate": 54.3433173159184,
"reward_low_FP_rate": 42.32359518695501,
"standard": 49.96056087185252
},
These are the params:
params_numenta_comparable = {
"enc": {
"value": {
#"resolution": 0.9, calculate by max(0.001, (maxVal - minVal) / numBuckets) where numBuckets = 130
"size": 400,
"activeBits": 21
},
"time": {
"timeOfDay": (21, 9.49),
}
},
"sp": {
# inputDimensions: use width of encoding
"columnDimensions": 2048,
# "potentialRadius": 999999, use width of encoding
"potentialPct": 0.8,
"globalInhibition": True,
"localAreaDensity": 0.1, # optimize this one
"stimulusThreshold": 0,
"synPermInactiveDec": 0.0005,
"synPermActiveInc": 0.003,
"synPermConnected": 0.2,
"boostStrength": 0.0,
"wrapAround": True,
"minPctOverlapDutyCycle": 0.001,
"dutyCyclePeriod": 1000,
},
"tm": {
"columnDimensions": 2048,
"cellsPerColumn": 32,
"activationThreshold": 20,
"initialPermanence": 0.24,
"connectedPermanence": 0.5,
"minThreshold": 13,
"maxNewSynapseCount": 31,
"permanenceIncrement": 0.04,
"permanenceDecrement": 0.008,
"predictedSegmentDecrement": 0.001,
"maxSegmentsPerCell": 128,
"maxSynapsesPerSegment": 128,
},
"anomaly": {
"likelihood": {
"probationaryPct": 0.1,
"reestimationPeriod": 100
}
}
}
another thing that I wondered about:
In the detecor code there is a fixed param 999999999 defined when setting the infos for tm and sp. Shouldn't this be encodingWidth? Or is it the potentialRadius?
self.tm_info = Metrics([self.tm.numberOfCells()], 999999999)
param 999999999 defined when setting the infos for tm and sp. Shouldn't this be encodingWidth? Or is it the potentialRadius?
no, this is unimportant. the metric is only used of our info, does not affect the computation. It's not related to "width"/number of bits, but rather a time/steps used of EMA in the metric
As the original sp also has localAreaDensity I would propose to first try out the original nupic detector with localAreaDensity instead of numActiveColumnsPerInhArea to see wether it has such an impact. I guess that would be faster than bringing it back.
Alright, it seems to have a significant impact. Running the Numenta detectors with localAreaDensity
of 0.1 instead of the numActiveColumnsPerInhArea
results in basically the same results as with the htmcore detector.
The only question remaining is now if tuning localAreaDensity
increases the score or if numActiveColumnsPerInhArea
is superior in general. What do you propose how to proceed @breznak ?
Here is the code: https://github.com/steinroe/NAB/tree/test_numenta_localAreaDensity
"numenta": {
"reward_low_FN_rate": 52.56449422487971,
"reward_low_FP_rate": 49.94586314087259,
"standard": 50.82949995800923
},
"numentaTM": {
"reward_low_FN_rate": 52.56449422487971,
"reward_low_FP_rate": 49.94586314087259,
"standard": 50.82949995800923
},
Going to try out the swarm algorithm to optimise localAreaDensity now.
Used Bayesian optimization, but nevertheless these are the results:
"htmcore": {
"reward_low_FN_rate": 60.852121191220256,
"reward_low_FP_rate": 45.428862226866734,
"standard": 55.50231971786488
},
for the following params:
parameters_numenta_comparable = {
"enc": {
"value": {
# "resolution": 0.9, calculate by max(0.001, (maxVal - minVal) / numBuckets) where numBuckets = 130
"size": 400,
"activeBits": 21,
"seed": 5, # ignored for the final run
},
"time": {
"timeOfDay": (21, 9.49),
}
},
"sp": {
# inputDimensions: use width of encoding
"columnDimensions": 2048,
# "potentialRadius": use width of encoding
"potentialPct": 0.8,
"globalInhibition": True,
"localAreaDensity": 0.025049634479368352, # optimize this one
"stimulusThreshold": 0,
"synPermInactiveDec": 0.0005,
"synPermActiveInc": 0.003,
"synPermConnected": 0.2,
"boostStrength": 0.0,
"wrapAround": True,
"minPctOverlapDutyCycle": 0.001,
"dutyCyclePeriod": 1000,
"seed": 5, # ignored for the final run
},
"tm": {
"columnDimensions": 2048,
"cellsPerColumn": 32,
"activationThreshold": 20,
"initialPermanence": 0.24,
"connectedPermanence": 0.5,
"minThreshold": 13,
"maxNewSynapseCount": 31,
"permanenceIncrement": 0.04,
"permanenceDecrement": 0.008,
"predictedSegmentDecrement": 0.001,
"maxSegmentsPerCell": 128,
"maxSynapsesPerSegment": 128,
"seed": 5, # ignored for the final run
},
"anomaly": {
"likelihood": {
"probationaryPct": 0.1,
"reestimationPeriod": 100
}
}
}
I created a PR htm-community/NAB/pull/25 for the updated params.
These are the logs for seed fixed to 5, where I achieved a standard score of 60 as maximum.
@breznak What would you suggest how to proceed from here?
Used Bayesian optimization, but nevertheless these are the results:
"htmcore": {
"reward_low_FN_rate": 60.852121191220256,
"reward_low_FP_rate": 45.428862226866734,
"standard": 55.50231971786488
},
Wow, these are very nice results! I'm going to merge NAB.
"localAreaDensity": 0.025049634479368352, # optimize this one
Interestingly, this is what's claimed by the HTM theory (2%) as observed in the cortex.
What would you suggest how to proceed from here?
compared to the Numenta results:
I'd suggest:
I think we should review if NAB is conceptually correct (as the metric and methodology)!
See our current results (I made a tiny change to your recently updated params in NAB/fixing_anomaly branch)
That is almost perfect! Yet in the plot:
For result file : ../results/htmcore/artificialWithAnomaly/htmcore_art_daily_flatmiddle.csv
True Positive (Detected anomalies) : 403
True Negative (Detected non anomalies) : 0
False Positive (False alarms) : 2679
False Negative (Anomaly not detected) : 0
Total data points : 3428
S(t)_standard score : -90.17164198169499
Q: do we want to keep comparable params & scores. Or just aim for the best score? Or both, separately?
this means we "beat" Numenta under the same conditions now, right?
Yes, but they still win with numActiveColumnsPerInhArea
set.
your Bayesian opt is multi-parametric? or just considering the select (localArea) param?
For this test I just optimised the localAreaDensity keeping the others constant but it can be multi parametric.
you try to tune the current score wrt the other params (there's lots of local optima and parameters are interleaved)
Alright, I will work on that. Do you have a feeling about which params may be important / optimizable?
I'll revert and reintroduce the numActiveColsPerInhArea param (alternative to localArea)
Perfect, thanks for your work!
we should get NAB operational with our optimization framework for multi-param opt. So we can brute-force the parameter space.
The "problem" is that your framework calls the optimization function in parallel, so my current setup with bayesian optimisation where I simply write to and read from a params.json file won't work. I will think about how to set that up that and come back to you as soon as I have a solution that works.
either just the Plot summary FP/FN is wrong ?
My gut says this may be the problem, as the scoring results seems fine. We should debug this.
Q: do we want to keep comparable params & scores. Or just aim for the best score? Or both, separately?
I would say we just aim for the best score, as even the bug fixes in this fork may influence the best param setting. I don't know if its useful to keep a second set of params as a comparison between the two would be only fair if both use the best possible params.
Yes, but they still win with numActiveColumnsPerInhArea set.
ok, I'll be able to do that. I got stuck today playing with the params.
but it can be multi parametric.
great. Good if you can try adding more params (below for hints on which). The parameter space is with many local optima.
[sorry, I'll need to do some other work done, brb in cca 2-3hrs]
Just quickly:
Let me know what do you think should be a priority:
great. Good if you can try adding more params (below for hints on which). The parameter space is with many local optima.
Alright, I will work on that.
Let me know what do you think should be a priority:
I would say reintroducing the numActiveColumns as (1) the multi param optimisation can right now also be done with other tools such as the bayesian optimisation and (2) I could also have a look into getting the swarm optimisation to run with NAB, while reintroducing the param is something where you are considerably faster than me. Does that work for you?
while reintroducing the param is something where you are considerably faster
I'll do that tonight.
I could also have a look into getting the swarm optimisation to run with NAB
That would be great. Ideally if NAB can run with the current opt framework. The requirement is that "NAB" can be run as a single method (que opt's requirements) and returns a score.
@steinroe you can now use branch sp_reintroduce_numActiveColumnsPerInhArea
#797 for comparison with the (re-)new param.
I'll try to stir again the waters in Forums on the topic if one is better than the other(?). I will have a look at param opt for NAB, because only than we can compare the params "by performance".
@breznak thanks for your work!
I am almost done with enabling swarm param opt with NAB. I just need 2 more hours.
enabling swarm param opt with NAB. I just need 2 more hours.
awesome! take your time.
Further on this front we have in progress #796 for easily running on clusters.
ok, if you can enable the param optimization for NAB, I'll now be able to start looking into "smart params" #536 aka
Okay this is what I got regarding param optimisation on NAB:
My goal was to not touch the nab codebase at all. Instead, I did a minor change to the htmcore detector (same can be done for any detector) where I read the params from a json file and set the bool params to True (as e.g. the swarm algorithm cannot deal with bool). You can see it in this branch: https://github.com/steinroe/NAB/tree/run_as_function
The second thing I added was a simple Dockerfile which installs NAB and, when started, runs NAB for the htmcore detector (again, this can be done with any detector)
FROM python:3.7
COPY . /NAB
WORKDIR /NAB
RUN pip install . --user --extra-index-url https://test.pypi.org/simple/
CMD python run.py -d htmcore --skipConfirmation --detect --optimize --score --normalize
I put this in the root of the NAB repo and build the image using docker build -t optimize-htmcore-nab:latest .
.
Now we can use that image to run htmcore on NAB.
NOTE: If using Docker Desktop, don't forget to grant more resources to it.
For the optimization, e.g. with the swarm algorithm, one can use the following script. The idea is basically to create a container from the image, put the params.json file inside of the container, start the container and run the detector on NAB and then copy the results file from the container on the host, read it and return the score. If you can think of any better method please let me know, as this feels kinda hacky.
NOTE: On MacOS, you have to export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
for docker.from_env() not to call an error (check this link for details):
EDIT: Run this script with python -m htm.optimization.ae -n 3 --memory_limit 4 -v --swarming 100 optimize_anomaly_swarm.py
(or whatever resources you want to grant)
import subprocess
import sys
import os
import json
import shutil
from pathlib import Path
import docker
default_parameters = {
"enc": {
"value": {
# "resolution": 0.9, calculate by max(0.001, (maxVal - minVal) / numBuckets) where numBuckets = 130
"size": 400,
"activeBits": 21, # results very sensitive to the size/active bits in the input encoders
"seed": 5,
},
"time": {
"timeOfDay": (21, 9.49),
}
},
"sp": {
# inputDimensions: use width of encoding
"columnDimensions": 2048,
# "potentialRadius": use width of encoding
"potentialPct": 0.8,
#"globalInhibition": True, always true (set in detector) as swarm cannot work with bool
"localAreaDensity": 0.025, # optimize this one
"stimulusThreshold": 2,
"synPermInactiveDec": 0.001,
"synPermActiveInc": 0.006,
"synPermConnected": 0.5, # this shouldn't make any effect, keep as intended by Connections
"boostStrength": 0.0, # so far, boosting negatively affects results. Suggest leaving OFF (0.0)
#"wrapAround": True, always true (set in detector) as swarm cannot work with bool
"minPctOverlapDutyCycle": 0.001,
"dutyCyclePeriod": 1000,
"seed": 5,
},
"tm": {
# "columnDimensions": 2048, #must match SP
"cellsPerColumn": 32,
"activationThreshold": 20,
"initialPermanence": 0.24,
"connectedPermanence": 0.5,
"minThreshold": 13,
"maxNewSynapseCount": 31,
"permanenceIncrement": 0.04,
"permanenceDecrement": 0.008,
"predictedSegmentDecrement": 0.001,
"maxSegmentsPerCell": 128,
"maxSynapsesPerSegment": 128,
"seed": 5,
},
"anomaly": {
"likelihood": {
"probationaryPct": 0.1,
"reestimationPeriod": 100
}
}
}
def main(parameters=default_parameters, argv=None, verbose=True):
# get client
client = docker.from_env()
print(client)
# create container
print('Creating Container')
container = client.containers.create(image='optimize-htmcore-nab:latest')
# make dir for container
print('Making Dir')
Path(os.path.join('temp', container.id)).mkdir(parents=True, exist_ok=True)
# write params to json in that dir
print('Writing Params')
with open(os.path.join('temp', container.id, 'params.json'), 'w') as outfile:
json.dump(default_parameters, outfile)
# copy file to container
print('Copy file to container')
cmd = 'docker cp ./temp/' + container.id + '/params.json ' + container.id + ':NAB/nab/detectors/htmcore/params.json'
subprocess.check_call(cmd, shell=True)
# start container
print('Starting container')
container.start()
container.wait()
# copy results file into temp directory with container id in path
cmd = 'docker cp ' + container.id + ':NAB/results/final_results.json ./temp/' + container.id + '/final_results.json'
subprocess.check_call(cmd, shell=True)
# read and return score from results file
with open(os.path.join('temp', container.id, 'final_results.json')) as json_file:
results = json.load(json_file)
score = results.get('htmcore').get('standard')
# remove container
container.remove()
# remove temp dir
shutil.rmtree(os.path.join('temp', container.id))
return score
if __name__ == '__main__':
sys.exit(main() < 0.7)
The same thing can be done for any optimization function such as Bayesian Optimization, which I have had some good experiences with. As this one does not call the target function in parallel, this could be done without docker on the local file system, too. Here the script for using it with docker as above:
import subprocess
import os
import json
import docker
import shutil
from pathlib import Path
from bayes_opt import BayesianOptimization
from bayes_opt.logger import JSONLogger
from bayes_opt.event import Events
from bayes_opt.util import load_logs
default_parameters = {
"enc": {
"value": {
# "resolution": 0.9, calculate by max(0.001, (maxVal - minVal) / numBuckets) where numBuckets = 130
"size": 400,
"activeBits": 21, # results very sensitive to the size/active bits in the input encoders
"seed": 5,
},
"time": {
"timeOfDay": (21, 9.49),
}
},
"sp": {
# inputDimensions: use width of encoding
"columnDimensions": 2048,
# "potentialRadius": use width of encoding
"potentialPct": 0.8,
#"globalInhibition": True, always true (set in detector) as swarm cannot work with bool
"localAreaDensity": 0.025, # optimize this one
"stimulusThreshold": 2,
"synPermInactiveDec": 0.001,
"synPermActiveInc": 0.006,
"synPermConnected": 0.5, # this shouldn't make any effect, keep as intended by Connections
"boostStrength": 0.0, # so far, boosting negatively affects results. Suggest leaving OFF (0.0)
#"wrapAround": True, always true (set in detector) as swarm cannot work with bool
"minPctOverlapDutyCycle": 0.001,
"dutyCyclePeriod": 1000,
"seed": 5,
},
"tm": {
# "columnDimensions": 2048, #must match SP
"cellsPerColumn": 32,
"activationThreshold": 20,
"initialPermanence": 0.24,
"connectedPermanence": 0.5,
"minThreshold": 13,
"maxNewSynapseCount": 31,
"permanenceIncrement": 0.04,
"permanenceDecrement": 0.008,
"predictedSegmentDecrement": 0.001,
"maxSegmentsPerCell": 128,
"maxSynapsesPerSegment": 128,
"seed": 5,
},
"anomaly": {
"likelihood": {
"probationaryPct": 0.1,
"reestimationPeriod": 100
}
}
}
def target_func(localAreaDensity):
# get params
my_params = default_parameters
my_params['sp']['localAreaDensity'] = localAreaDensity
# get client
client = docker.from_env()
# create container
container = client.containers.create(image='optimize-htmcore-nab:latest')
# make dir for container
Path(os.path.join('temp', container.id)).mkdir(parents=True, exist_ok=True)
# write params to json in that dir
with open(os.path.join('temp', container.id, 'params.json'), 'w') as outfile:
json.dump(my_params, outfile)
# copy file to container
cmd = 'docker cp ./temp/' + container.id + '/params.json ' + container.id + ':NAB/nab/detectors/htmcore/params.json'
subprocess.check_call(cmd, shell=True)
# start container
container.start()
container.wait()
# copy results file into temp directory with container id in path
cmd = 'docker cp ' + container.id + ':NAB/results/final_results.json ./temp/' + container.id + '/final_results.json'
subprocess.check_call(cmd, shell=True)
# read and return score from results file
with open(os.path.join('temp', container.id, 'final_results.json')) as json_file:
results = json.load(json_file)
score = results.get('htmcore').get('standard')
# remove container
container.remove()
# remove temp dir
shutil.rmtree(os.path.join('temp', container.id))
return score
def optimize_local_area_density():
# define bounds for the params you want to optimize. Can be multivariate. Check https://github.com/fmfn/BayesianOptimization on how to
bounds = {
'localAreaDensity': (0.01, 0.15),
}
optimizer = BayesianOptimization(
f=target_func,
pbounds=bounds,
random_state=1,
)
# We can start from saved logs
if os.path.isfile('./local_area_density_optimization_logs_base.json'):
print('Loading Logs...')
load_logs(optimizer, logs=["./local_area_density_optimization_logs_base.json"]);
# The new log file to write to
logger = JSONLogger(path="./local_area_density_optimization_logs.json")
optimizer.subscribe(Events.OPTIMIZATION_STEP, logger)
# If you want to guide the optimization process
val = 0.02
while val <= 0.04:
print('Adding', val)
optimizer.probe(
params={
'localAreaDensity': val,
},
lazy=True,
)
val = round(val + 0.001, 3)
print('Starting optimization...')
optimizer.maximize(
init_points=20,
n_iter=50,
)
print(optimizer.max)
if __name__ == "__main__":
optimize_local_area_density()
FYI: Running on my MacBook (with limited resources) I am at a max score of 60.907:
EDIT: Note: seed is not used.
New global best score 60.907.
{'anomaly': {'likelihood': {'probationaryPct': 0.10287247901390377,
'reestimationPeriod': 105}},
'enc': {'time': {'timeOfDay': (19, 10.230927402319585)},
'value': {'activeBits': 20, 'seed': 4, 'size': 433}},
'sp': {'boostStrength': 0.0,
'columnDimensions': 1992,
'dutyCyclePeriod': 997,
'localAreaDensity': 0.02315293328723999,
'minPctOverlapDutyCycle': 0.0010876327134838708,
'potentialPct': 0.7268162250552015,
'seed': 6,
'stimulusThreshold': 1,
'synPermActiveInc': 0.005756214837461844,
'synPermConnected': 0.5255368572159246,
'synPermInactiveDec': 0.0010412298597037122},
'tm': {'activationThreshold': 19,
'cellsPerColumn': 33,
'connectedPermanence': 0.4732123724546982,
'initialPermanence': 0.23075451521239015,
'maxNewSynapseCount': 31,
'maxSegmentsPerCell': 135,
'maxSynapsesPerSegment': 120,
'minThreshold': 12,
'permanenceDecrement': 0.007285708940843109,
'permanenceIncrement': 0.04329160631961319,
'predictedSegmentDecrement': 0.0010265038754633969,
'seed': 5}}
Great work on the optimization! I'm going through it and will be trying it later.
A couple of questions, remarks
The second thing I added was a simple Dockerfile which installs NAB and, when started, runs NAB for the htmcore detector
do we need that? We can run htmcore
detector on community/NAB as well as numenta/NAB.
For the optimization, e.g. with the swarm algorithm, one can use the following script. The idea is basically to create a container from the image, put the params.json file inside of the container, start the container and run the detector on NAB and then copy the results file from the container on the host, read it and return the score.
this seems smart, a least intruding solution.
The same thing can be done for any optimization function such as Bayesian Optimization, which I have had some good experiences with.
Later on, I'd like to achieve the following
htm.optimization
"framework" (currently has PSO, BruteForce) I am at a max score of 60.907:
I'll try to run this setup as well.
do we need that? We can run htmcore detector on community/NAB as well as numenta/NAB.
The whole point of using docker is that NAB heavily depends on using the file system, e.g. for writing out the results. The optimization framework runs multiple scorings in parallel, so we would have conflicting results when running on our host system. With docker, we are able to prevent that. Or am I misinterpreting your question somehow?
merge your Bayes optimizer into the existing htm.optimization "framework" (currently has PSO, BruteForce)
I can have a look into that 👍
can the trick with NAB be used/automated to be used for the above framework?
I guess so, as it works with the PSO algorithm already. EDIT: Sorry, I forgot to add that the PSO script above runs on your framework, so you run python -m htm.optimization.ae -n 3 --memory_limit 4 -v --swarming 100 optimize_anomaly_swarm.py
to start the optimization.
what I'm missing is how to run multi-parametric optimization. Could you please post an example of that?
Sure. With Bayesian Optimization you need to set bounds for each parameter, but you can use as many as you want, e.g. pbounds = {'x': (2, 4), 'y': (-3, 3)}
to optimise x
and y
within (2, 4)
and (-3, 3)
respectively. See the repo README of the python implementation for details. If we can decide on useful bounds, I can setup a script for optimising all / a larger subset of the params.
EDIT: Added link to repo
About integrating Bayesian Opt into your optimization framework: I think the main challenge is that Bayes Opt requires bounds to be set while PSO as well as GridSearch only require a "starting point". I think such a change would require a change in the framework itself to be able to add bounds instead of the default_parameter dict that is currently used.
I like this, this is a really nice trick to get it done with NAB.
The optimization framework runs multiple scorings in parallel, so we would have conflicting results when running on our host system. With docker, we are able to prevent that.
good, bcs I thought it didn't run in parallel. We could do something like writing to /tmp/$PID/results.json
but this already works good enough, so :+1:
Sorry, I forgot to add that the PSO script above runs on your framework, so you run python -m htm.optimization.ae `
great. I was thinking you're developing from scatch. It's better that you could be using our existing tools!
Sure. With Bayesian Optimization you need to set bounds for each parameter, but you can use as many as you want, e.g. pbounds = {'x': (2, 4), 'y': (-3, 3)} to optimise x and y
even better! this looks really convenient.
If we can decide on useful bounds, I can setup a script for optimising all / a larger subset of the params
I'll try working on constraining the params, and along with that we can come up with a subset and its ranges.
For some params, we'd need some control over the granuity etc, eg. bounds for numColumns would be something as [1024, 8192], but we're not interested in 1023
etc, rather than stride by 1024. But we can hack such cases in the detector manually, as
if x not in {1024, 2048, 4096, ...}:
raise
I did a minor change to the htmcore detector (same can be done for any detector) where I read the params from a json file and set the bool params to True (as e.g. the swarm algorithm cannot deal with bool
useOptimization
and if set, it reads the params from the file. To eliminate one more step for using your idea good, bcs I thought it didn't run in parallel. We could do something like writing to /tmp/$PID/results.json but this already works good enough, so
We would probably still get interferences when the detector reads the params file. I think locking the parallel runs of NAB into a container is the cleanest and safest method. The only downside is probably performance, as we have to give docker host a fair share of our host resources.
For some params, we'd need some control over the granuity etc, eg. bounds for numColumns would be something as [1024, 8192], but we're not interested in 1023 etc, rather than stride by 1024. But we can hack such cases in the detector manually, as
Bayesian Optimization does that automatically by randomly choosing some parameters in the beginning and from time to time. Additionally, we can probe some settings (lets say 1024, 2048, ... 8192) so the algorithm knows of the effect these settings have. Its quite good in avoiding local maxima out of the box (at least to my experience)
could you make a PR with this to our NAB? Ie a toggle useOptimization and if set, it reads the params from the file. To eliminate one more step for using your idea
Sure, will do.
I'll get to testing this workflow/guide a bit later. Do you think it could be further automated? say a script, or a docker (with docker)?
I think we could put the image on docker hub so the user does not have to build the image. With a minor change to the script we could pull the image from docker hub. Then, The workflow would be as follows:
Requirements: Docker Desktop, htmcore, requirements of script such as docker
python -m htm.optimization.ae -n 3 --memory_limit 4 -v --swarming 100 optimize_anomaly_swarm.py
Requirements: Docker Desktop, requirements of script such as docker and bayesian opt
EDIT: Where to put the scripts? Another repo? Or just as branch?
We would probably still get interferences when the detector reads the params file. I think locking the parallel runs of NAB into a container is the cleanest and safest method
ok, I agree. We need to get there with minimal manhours, so this works as intended.
Additionally, we can probe some settings (lets say 1024, 2048, ... 8192) so the algorithm knows of the effect these settings have.
interesting. nice if we can "hint" to try these datapoints.
I think we could put the image on docker hub so the user does not have to build the image. With a minor change to the script we could pull the image from docker hub.
maybe not necessarily DockerHub, but a docker file in the repo (NAB) would be great :+1:
htmcore is already dependency of (our) NAB, so that's no problem. My (optimal) idea was a script/dockerfile that the user can run (modify which params should be probed, and provide data-as in NAB) and it runs the optimization and outputs the found set of params & score.
EDIT: Where to put the scripts? Another repo? Or just as branch?
I'd make this part of community/NAB. So submit as PR/branch, and we'll merge directly to master (this enhancement is unrelated to "fixing htmcore scores")
Sounds good to me! Do we want to enable optimisation on NAB in general or only for htmcore?
Do we want to enable optimisation on NAB in general or only for htmcore?
I'd say minimal changes first - we need to solve topic of this issue. So just HTMcore for now.
Because I'm still unsure how to proceed with https://github.com/htm-community/NAB/issues/21
but
I'd say minimal changes first - we need to solve topic of this issue. So just HTMcore for now.
Alright, I will update the optimisation PR with a proposal on how to make it as user friendly as possible.
CC @Zbysekz as the author of HTMpandaVis, do you think you could help us debugging the issue? I'd really appreciate that!
TL;DR: minor parameters and changes were surely done to htm.core. Compared to Numenta's Nupic, our results on NAB really suck now. I'm guessing it should be matter of incorrect params.
There look into the representations with the visualizer would be really helpful. More info here (and linked posts) htm-community/NAB#15
Hello, sorry for the delay. Would it help, if i modify HTMdetector in steinroe repo to use PandaVis and see particular steps live ?
I am not sure if that is the script that you guys are using to run it.
hi Zbysekz!
HTMdetector in steinroe repo to use PandaVis and see particular steps live ?
that'd be great step!
Actually we're merging and currently we run community/NAB
. The file is in nab/detectors/htmcore/htmcore_detector.py
EDIT:
Hi everyone,
thanks for the great work on htm.core! I am doing some research on leveraging HTM for anomaly detection and I am wondering wether I should use htm.core or nupic. Is there any comparison in terms of performance?
@breznak You described some issues in PR #15. What would you say - continue with htm.core or rather get nupic running? Its just a masters thesis so I won't be able to dive deep into htm to improve the implementation...