search

numenta / htmresearch

Experimental algorithms. Unsupported.
GNU Affero General Public License v3.0
223 stars 109 forks source link

NLP model API tests broken #552

Open BoltzmannBrain opened 8 years ago

BoltzmannBrain commented 8 years ago

The simple labels and hello classification tests for sensor_knn.json config fail -- 52.98% vs 66.2%, and 60% vs 80%, respectively.

@subutai would you please confirm you get the same?

subutai commented 8 years ago

@BoltzmannBrain

If I run: python hello_classification_model.py -c data/network_configs/sensor_knn.json

I get 90%. Do you not get the same?

I'm running simple_labels.py now (going slowly because I need to rebuild my cache)

BoltzmannBrain commented 8 years ago

No I get 70%. I ran again without using my cache and get the same results.

I notice there's a difference between hello_classification.py and the simple_labels.py implementation: doc 2 uses "lettuce" in the former but "kale" in the latter. I think it should be "kale" in both b/c they both use "kale" in doc 6.

subutai commented 8 years ago

I just ran:

python simple_labels.py --dataPath ~/nta/grok-projects/nlp_experiments/yale/buckets/yale_data_buckets_ints.csv -m htm --numLabels 10

and got 66.2%

With regards to kale vs lettuce, I don't think it matters too much. Results should be the same either way.

subutai commented 8 years ago

The tests w/ TM (sensor_tm_simple_tp.json config) fail on saving the model in executeModelLifecycle() of htmresearch/support/nlp_model_test_helpers.py.

@BoltzmannBrain What is the exact command you used?

BoltzmannBrain commented 8 years ago

The tests w/ TM (sensor_tm_simple_tp.json config) fail on saving the model in executeModelLifecycle() of htmresearch/support/nlp_model_test_helpers.py.

@BoltzmannBrain What is the exact command you used?

This config file is something I've used locally and isn't included in the repo / API tests (yet). It's verys imilar to imbu_sensor_tm_simple_tp.json and the error isn't reproduced with that config. I've removed that part of this issue. Sorry for the confusion.

BoltzmannBrain commented 8 years ago

and got 66.2%

Okay it looks like I have something locally that is to blame. Just to be sure, what nupic and nupic.core are you using? I'm on nupic 0.5.3dev0 and nupic.core 0.4.3dev0.