Giving very low score!!

[deepankar27]

I trained four classes with 4-5 sentences on each class & it's giving me very low scores, even I got low score when I ran the pre-trained model & gave same sentence which you had given as an example.

I am using as is config settings, no change in that.

[dilyararimovna]

Hello! Unfortunately, I could not reproduce this case. I added versions of packages to requirements.txt. You can try to reinstall packages and run again. If still such low score is shown, inform us here, please.

[deepankar27]

First, thanks for the response.

I am listing down my versions below: numpy==1.14.2 (higher version) nltk==3.2.5 (higher version) keras==2.1.4 (higher version) tensorflow==1.6.0 (higher version) gensim==3.4.0 (higher version) pandas==0.22.0 (higher version) sklearn h5py==2.7.1 (same version) tqdm==4.19.5 (lower version) -- > Upgraded to tqdm==4.19.6 fasttext==0.8.3 (same version)

I downloaded it & ran it again but no change in scoring.

And I am using Python 3.6.4

[dilyararimovna]

I also added cython==0.26.1 version. While testing I managed to reproduce your score. I run the command ./intent_classifier.py ./snips_pretrained/snips_config.json, while some warning information were being printed I pressed Enter. When all the info were printed I entered the phrase and obtained the following:

I want you to add 'I love you, baby' to my playlist
(0.378094, 'SearchCreativeWork')
(0.99986315, 'AddToPlaylist')

It is better to add some info that model is ready. That I have already done. Please, pull from repo.

[deepankar27]

Thanks!! but I just trying to figure out why is this happening with me?

I just added to check whether my query goes to the model or not so, made a change in intent_classifier.py -> infer -> line num 16

def infer(phrase): global preprocessor, classes, model try: print('phrase: '+phrase) predictions = model.infer(preprocessor.infer(phrase)) except Exception: print('Error', file=sys.stderr) return 0, 'error' return np.max(predictions), classes[np.argmax(predictions)]

I ran this command python intent_classifier.py ./snips_pretrained/snips_config.json from cmd it gives me this output and hit enter key just after the warning. And yes, I am trying to run it from Anaconda cmd as it has all libs.

So, coz of first enter key hit it gives me one score & another for my question.

Model is ready! You now can enter requests.

phrase:

(0.378094, 'SearchCreativeWork') I want you to add 'I love you, baby' to my playlist phrase: I want you to add 'I love you, baby' to my playlist

(0.378094, 'SearchCreativeWork')

I will also give a look into the multiclass.py it's bit wired..

[dilyararimovna]

Finally, I got your error. You are using Windows. Unfortunately, fasttext==0.8.3 does not support windows (actually, it says that it was successfully installed but for any word it returns zero embeddings), only Linux. Currently, this repo is not main in our library - it had been built before new fastText for python (https://github.com/facebookresearch/fastText) was released. Later (today or tomorrow) I will be able to check if new fastText works on Windows correctly.

Also it is better to try intents from DeepPavlov repo: https://github.com/deepmipt/DeepPavlov/tree/master/deeppavlov/models/classifiers/intents This is currently supported library. It uses new fastText version.

[deepankar27]

I am sorry I forgot to mention you what OS am I using.. even I saw it in feature no value is going:

Model is ready! You now can enter requests. I want you to add 'I love you, baby' to my playlist phrase: I want you to add 'I love you, baby' to my playlist

inside infer_on_batch.. sum of features.sum() 0.0 (0.378094, 'SearchCreativeWork')

[deepankar27]

Is this - https://github.com/deepmipt/DeepPavlov/tree/master/deeppavlov/models/classifiers/intents compatible on windows as well?

[deepankar27]

@dilyararimovna I guess it would be great if you just upgrade intent classifier. It would be great if you can upgrade the fastext for windows.

[dilyararimovna]

I checked fastText==0.8.22 (it can be installed from github, link is given above) but it is not compatible for Windows because of errors during installation. You can try classifier from DeepPavlov with GloVe embedder (https://github.com/deepmipt/DeepPavlov/blob/master/deeppavlov/models/embedders/glove_embedder.py).

[deepankar27]

Let me give a try with GloVe. I will keep you posted. Thanks again!!

[deepankar27]

One more can we create own GloVe model & use it on our corpus?

[dilyararimovna]

Do you mean train own GloVe model? Yes, you can find here instructions: https://github.com/stanfordnlp/GloVe

[deepankar27]

@dilyararimovna Thanks!! I will keep you posted about what happened after using GloVe embedding.

[deepankar27]

Changes made to train own model on train.csv data

1. Changes made in intent_model.py a. added from deeppavlov.models.embedders.glove_embedder import GloVeEmbedder b. and code change in def init(self, opt: Dict, embedder: GloVeEmbedder, #FasttextEmbedder, tokenizer: NLTKTokenizer, classes=None, vocabs=None, **kwargs):

2. In intents_snips.json config changed the embedder to "embedder": { "name": "glove", "save_path": "embeddings/glove.6B.100d_word2vec.txt", "load_path": "embeddings/glove.6B.100d_word2vec.txt", "emb_module": "gensim", "dim": 100 } I changed the glove.6B.100d to word2vec using gensim because, I was getting error when it was trying to load the Golve model. Error: ValueError: invalid literal for int() with base 10: 'the'

3. Set the train.csv data path in "dataset_reader": { "name": "basic_classification_reader", "x": "text", "y": "intents", "data_path": "deeppavlov/configs/intents", "url": "http://lnsigo.mipt.ru/export/datasets/snips_intents/train.csv" }

After above all changes I ran the command: python deep.py train configs/intents/intents_snips.json

2018-04-21 11:39:02.193 INFO in 'deeppavlov.core.models.keras_model'['keras_model'] at line 182: [loading weights from intent_cnn_snips_v2.h5] 2018-04-21 11:39:02.269 ERROR in 'deeppavlov.core.common.params'['params'] at line 92: Exception in <class 'deeppavlov.models.classifiers.intents.intent_model.KerasIntentModel'> Traceback (most recent call last): File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 686, in _call_cpp_shape_fn_impl input_tensors_as_shapes, status) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 516, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 1 in both shapes must be equal, but are 2 and 7. Shapes are [100,2] and [100,7]. for 'Assign_24' (op: 'Assign') with input shapes: [100,2], [100,7].

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\common\params.py", line 86, in from_params component = cls(dict(config_params, kwargs)) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\models\tf_backend.py", line 47, in call obj.init(*args, kwargs) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped return func(args, kwargs) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\models\classifiers\intents\intent_model.py", line 129, in init self.model = self.load(params) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped return func(args, kwargs) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\models\keras_model.py", line 183, in load model.load_weights(str(weights_path)) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\topology.py", line 2652, in load_weights f, self.layers, reshape=reshape) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\topology.py", line 3189, in load_weights_from_hdf5_group K.batch_set_value(weight_value_tuples) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 2365, in batch_set_value assign_op = x.assign(assign_placeholder) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\ops\variables.py", line 599, in assign return state_ops.assign(self._variable, value, use_locking=use_locking) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\ops\state_ops.py", line 280, in assign validate_shape=validate_shape) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 61, in assign use_locking=use_locking, name=name) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3273, in create_op compute_device=compute_device) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3313, in _create_op_helper set_shapes_for_outputs(op) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2501, in set_shapes_for_outputs return _set_shapes_for_outputs(op) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2474, in _set_shapes_for_outputs shapes = shape_func(op) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2404, in call_with_requiring return call_cpp_shape_fn(op, require_shape_fn=True) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 627, in call_cpp_shape_fn require_shape_fn) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 691, in _call_cpp_shape_fn_impl raise ValueError(err.message) ValueError: Dimension 1 in both shapes must be equal, but are 2 and 7. Shapes are [100,2] and [100,7]. for 'Assign_24' (op: 'Assign') with input shapes: [100,2], [100,7]. Traceback (most recent call last): File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 686, in _call_cpp_shape_fn_impl input_tensors_as_shapes, status) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 516, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 1 in both shapes must be equal, but are 2 and 7. Shapes are [100,2] and [100,7]. for 'Assign_24' (op: 'Assign') with input shapes: [100,2], [100,7].

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "deep.py", line 63, in main() File "deep.py", line 50, in main train_model_from_config(pipeline_config_path) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\commands\train.py", line 101, in train_model_from_config model = fit_chainer(config, iterator) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\commands\train.py", line 53, in fit_chainer component = from_params(component_config, vocabs=[], mode='train') File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\common\params.py", line 86, in from_params component = cls(dict(config_params, kwargs)) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\models\tf_backend.py", line 47, in call obj.init(*args, kwargs) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped return func(args, kwargs) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\models\classifiers\intents\intent_model.py", line 129, in init self.model = self.load(params) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped return func(args, kwargs) File "E:\username\NLP\CNN\DeepPavlov-master\deeppavlov\core\models\keras_model.py", line 183, in load model.load_weights(str(weights_path)) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\topology.py", line 2652, in load_weights f, self.layers, reshape=reshape) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\topology.py", line 3189, in load_weights_from_hdf5_group K.batch_set_value(weight_value_tuples) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 2365, in batch_set_value assign_op = x.assign(assign_placeholder) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\ops\variables.py", line 599, in assign return state_ops.assign(self._variable, value, use_locking=use_locking) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\ops\state_ops.py", line 280, in assign validate_shape=validate_shape) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 61, in assign use_locking=use_locking, name=name) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3273, in create_op compute_device=compute_device) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3313, in _create_op_helper set_shapes_for_outputs(op) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2501, in set_shapes_for_outputs return _set_shapes_for_outputs(op) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2474, in _set_shapes_for_outputs shapes = shape_func(op) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2404, in call_with_requiring return call_cpp_shape_fn(op, require_shape_fn=True) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 627, in call_cpp_shape_fn require_shape_fn) File "E:\Users\userid\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 691, in _call_cpp_shape_fn_impl raise ValueError(err.message) ValueError: Dimension 1 in both shapes must be equal, but are 2 and 7. Shapes are [100,2] and [100,7]. for 'Assign_24' (op: 'Assign') with input shapes: [100,2], [100,7].

Note: I haven't set the name of the file name like train.csv in intents_snips.json, I mentioned only the path in dataset_reader. Just a small feedback keeping the mind what a great thing you people are sharing with the world, it would be great if you have a training video or step by step tutorial which can show how to train model on our own corpus that too in GloVe/Word2Vec model, I know fasttext is better than these two but fasttext doesn't support windows and sometime it gives wired errors like insufficient memory & all.

Thanks!!

[dilyararimovna]

As SNIPS dataset contains 7 intents then I think this error was caused by difference between dataset structure. Try to use to load dataset for DeepPavlov (dot not use dataset that was downloaded for this repo).

[deepankar27]

I didn't use the downloaded dataset, I created mine & changed the SNIPS config file as per my requirement.

[dilyararimovna]

And your dataset has two classes? Then the problem is simple - as you can read in the error message - you are trying to load pre-trained model. If you want to train model from scratch you have to change load_path to model.

[deepankar27]

I changed the config file: Created "valid.csv" & "train.csv" both of them has same number of intents. Updating the own_snips_classes.dict (custom) classes/intents correctly in vocabs folder.

AddToPlaylist 13 RateBook 17

{ "in": ["x"], "in_y": ["y"], "out": ["y_predicted"], "main": true, "name": "intent_model", "save_path": "intents/own_intent_cnn_snips_v2", (new model name) "load_path": "intents/own_intent_cnn_snips_v2", (new model name) "classes": "#classes_vocab.keys()", "opt": { "kernel_sizes_cnn": [ 1, 2, 3 ], "filters_cnn": 256, "lear_metrics": [ "binary_accuracy", "fmeasure" ], "confident_threshold": 0.5, "optimizer": "Adam", "lear_rate": 0.01, "lear_rate_decay": 0.1, "loss": "binary_crossentropy", "text_size": 15, "coef_reg_cnn": 1e-4, "coef_reg_den": 1e-4, "dropout_rate": 0.5, "epochs": 1000, "dense_size": 100, "model_name": "cnn_model" }

But still getting this error: E:\Users\user\anaconda3\lib\site-packages\gensim\utils.py:1197: UserWarning: detected Windows; aliasing chunkize to chunkize_serial warnings.warn("detected Windows; aliasing chunkize to chunkize_serial") 2018-04-21 19:00:09.98 INFO in 'summarizer.preprocessing.cleaner'['textcleaner'] at line 37: 'pattern' package not found; tag filters are not available for English 2018-04-21 19:00:14.519 INFO in 'main'['deep'] at line 49: pipeline config_path configs/intents/intents_snips.json 2018-04-21 19:00:14.555 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 87: Cannot find E:\user\NLP\CNN\DeepPavlov-master\deeppavlov\configs\intents\test.csv file 2018-04-21 19:00:14.556 INFO in 'deeppavlov.dataset_iterators.basic_classification_iterator'['basic_classification_iterator'] at line 67: Splitting field <> to new fields <<['train', 'valid']>> 2018-04-21 19:00:14.560 INFO in 'deeppavlov.core.data.vocab'['vocab'] at line 158: [loading vocabulary from E:\user\NLP\CNN\DeepPavlov-master\download\vocabs\own_snips_classes.dict] 2018-04-21 19:00:14.561 INFO in 'deeppavlov.core.data.vocab'['vocab'] at line 146: [saving vocabulary to E:\user\NLP\CNN\DeepPavlov-master\download\vocabs\own_snips_classes.dict] 2018-04-21 19:00:14.565 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 49: [loading embeddings from E:\user\NLP\CNN\DeepPavlov-master\download\embeddings\glove.6B.100d_word2vec.txt] 2018-04-21 19:00:14.565 INFO in 'gensim.models.utils_any2vec'['utils_any2vec'] at line 172: loading projection weights from E:\user\NLP\CNN\DeepPavlov-master\download\embeddings\glove.6B.100d_word2vec.txt 2018-04-21 19:00:14.566 DEBUG in 'smart_open.smart_open_lib'['smart_open_lib'] at line 149: {'kw': {}, 'mode': 'rb', 'uri': 'E:\user\NLP\CNN\DeepPavlov-master\download\embeddings\glove.6B.100d_word2vec.txt'} 2018-04-21 19:00:14.567 DEBUG in 'smart_open.smart_open_lib'['smart_open_lib'] at line 621: encoding_wrapper: {'errors': 'strict', 'encoding': None, 'mode': 'rb', 'fileobj': <_io.BufferedReader name='E:\\user\\NLP\\CNN\\DeepPavlov-master\\download\\embeddings\\glove.6B.100d_word2vec.txt'>} 2018-04-21 19:00:58.339 INFO in 'gensim.models.utils_any2vec'['utils_any2vec'] at line 234: loaded (400000, 100) matrix from E:\user\NLP\CNN\DeepPavlov-master\download\embeddings\glove.6B.100d_word2vec.txt 2018-04-21 19:00:58.346325: I E:\tf_jenkins\workspace\rel-win\M\windows\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2018-04-21 19:00:58.352 INFO in 'deeppavlov.models.classifiers.intents.intent_model'['intent_model'] at line 91: self opt embedding_size 100 2018-04-21 19:00:58.352 INFO in 'deeppavlov.models.classifiers.intents.intent_model'['intent_model'] at line 92: fasttext_model load_path E:\user\NLP\CNN\DeepPavlov-master\download\embeddings\glove.6B.100d_word2vec.txt 2018-04-21 19:00:59.333 INFO in 'deeppavlov.core.models.keras_model'['keras_model'] at line 99: [initializing KerasIntentModel from scratch] WARNING:tensorflow:From E:\Users\user\anaconda3\lib\site-packages\tensorflow\python\util\deprecation.py:497: calling conv1d (from tensorflow.python.ops.nn_ops) with data_format=NHWC is deprecated and will be removed in a future version. Instructions for updating: NHWC for data_format is deprecated, use NWC instead 2018-04-21 19:01:02.587 WARNING in 'tensorflow'['tf_logging'] at line 126: From E:\Users\user\anaconda3\lib\site-packages\tensorflow\python\util\deprecation.py:497: calling conv1d (from tensorflow.python.ops.nn_ops) with data_format=NHWC is deprecated and will be removed in a future version. Instructions for updating: NHWC for data_format is deprecated, use NWC instead 2018-04-21 19:01:04.385 INFO in 'deeppavlov.core.commands.train'['train'] at line 117: train_config {'metrics': ['accuracy'], 'validate_best': True, 'test_best': True} 2018-04-21 19:01:04.386 INFO in 'deeppavlov.core.commands.train'['train'] at line 118: model config <deeppavlov.core.common.chainer.Chainer object at 0x000002CE403F4860> Traceback (most recent call last): File "deep.py", line 63, in main() File "deep.py", line 50, in main train_model_from_config(pipeline_config_path) File "E:\user\NLP\CNN\DeepPavlov-master\deeppavlov\core\commands\train.py", line 129, in train_model_from_config _train_batches(model, iterator, train_config, metrics_functions) File "E:\user\NLP\CNN\DeepPavlov-master\deeppavlov\core\commands\train.py", line 231, in _train_batches y_predicted = list(model(list(x))) File "E:\user\NLP\CNN\DeepPavlov-master\deeppavlov\core\common\chainer.py", line 88, in call res = component([mem[k] for k in in_params]) File "E:\user\NLP\CNN\DeepPavlov-master\deeppavlov\models\classifiers\intents\intent_model.py", line 219, in call preds = np.array(self.infer_on_batch(data)) File "E:\user\NLP\CNN\DeepPavlov-master\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped return func(args, kwargs) File "E:\user\NLP\CNN\DeepPavlov-master\deeppavlov\models\classifiers\intents\intent_model.py", line 203, in infer_on_batch predictions = self.model.predict(features) File "E:\Users\user\anaconda3\lib\site-packages\keras\engine\training.py", line 1824, in predict check_batch_axis=False) File "E:\Users\user\anaconda3\lib\site-packages\keras\engine\training.py", line 113, in _standardize_input_data 'with shape ' + str(data_shape)) ValueError: Error when checking : expected input_1 to have 3 dimensions, but got array with shape (20, 1)**

I really don't know where am I going wrong now?

[dilyararimovna]

This error means that you are trying to send training data in 2d format whereas required format is (batch_size, text_size, embedding_size). You have to check whether you are reading data correctly, whether you are giving data to model correctly. You can try debugging or printing.

[deepankar27]

First thanks for your continuous support & patience.

This is what my training dataset looks like.. it's just a copy from your existing dataset.

I haven't changed anything in the code except this config json file:

{ "dataset_reader": { "name": "basic_classification_reader", "x": "text", "y": "intents", "data_path": "E:/NLP/CNN/DeepPavlov-master/deeppavlov/configs/intents", "url": "http://lnsigo.mipt.ru/export/datasets/snips_intents/train.csv" }, "dataset_iterator": { "name": "basic_classification_iterator", "seed": 42, "field_to_split": "train", "split_fields": [ "train", "valid" ], "split_proportions": [ 0.9, 0.1 ] }, "chainer": { "in": ["x"], "in_y": ["y"], "pipe": [ { "id": "classes_vocab", "name": "default_vocab", "fit_on": ["y"], "level": "token", "save_path": "vocabs/own_snips_classes.dict", "load_path": "vocabs/own_snips_classes.dict" }, { "in": ["x"], "in_y": ["y"], "out": ["y_predicted"], "main": true, "name": "intent_model", "save_path": "intents/own_intent_cnn_snips_v2", "load_path": "intents/own_intent_cnn_snips_v2", "classes": "#classes_vocab.keys()", "opt": { "kernel_sizes_cnn": [ 1, 2, 3 ], "filters_cnn": 256, "lear_metrics": [ "binary_accuracy", "fmeasure" ], "confident_threshold": 0.5, "optimizer": "Adam", "lear_rate": 0.01, "lear_rate_decay": 0.1, "loss": "binary_crossentropy", "text_size": 15, "coef_reg_cnn": 1e-4, "coef_reg_den": 1e-4, "dropout_rate": 0.5, "epochs": 1000, "dense_size": 100, "model_name": "cnn_model" }, "embedder": { "name": "glove", "save_path": "embeddings/glove.6B.100d_word2vec.txt", "load_path": "embeddings/glove.6B.100d_word2vec.txt", "emb_module": "gensim", "dim": 100 }, "tokenizer": { "name": "nltk_tokenizer", "tokenizer": "wordpunct_tokenize" } } ], "out": ["y_predicted"] }, "train": { "epochs": 100, "batch_size": 20, "metrics": [ "sets_accuracy" ], "validation_patience": 5, "val_every_n_epochs": 1, "log_every_n_epochs": 1, "show_examples": false, "validate_best": true, "test_best": false } }

and executing the cmd: python deep.py train configs\intents\intents_snips.json

Then it creates own_snips_classes.dict in "DeepPavlov-master\download\vocabs" in that file as I mentioned you it shows correct counts of utterances for it's respective classes. Note: Neither I haven't done any change in code nor in dataset except the config file what I shared here with you.

[dilyararimovna]

For glove embedder now we are using gensim. Therefore, correct format for embedding file with 400k words and 50d embedding is following:

400000 50
the 0.418 0.24968 -0.41242 0.1217 0.34527 -0.044457 -0.49688 -0.17862 -0.00066023 -0.6566 0.27843 -0.14767 -0.55677 0.14658 -0.0095095 0.011658 0.10204 -0.12792 -0.8443 -0.12181 -0.016801 -0.33279 -0.1552 -0.23131 -0.19181 -1.8823 -0.76746 0.099051 -0.42125 -0.19526 4.0071 -0.18594 -0.52287 -0.31681 0.00059213 0.0074449 0.17778 -0.15897 0.012041 -0.054223 -0.29871 -0.15749 -0.34758 -0.045637 -0.44251 0.18785 0.0027849 -0.18411 -0.11514 -0.78581

Please, make sure you have the following format. Also please, note that glove embedder requires list of tokens whereas fasttext embedder requires string that will be splitted to tokens. For the next release all this points are going to be fixed. Now you can try to change call and encode methods for glove as for fasttext.

[deepankar27]

Thanks for the reply!! Do you want me to change the call & encode methods here deeppavlov/models/embedders/glove_embedder.py ? I found many files with same methods.

I know I am bothering a lot!! My intention is just to use it & see how it works!! And thanks again for the continuous support.

[dilyararimovna]

Hello! Yes. Firstly, you have to change your glove.txt file (add header line as above mentioned). Then change either deeppavlov/models/embedders/glove_embedder.py (make it taking string) or deeppavlov/models/classifiers/intents/intent_model.py (make method texts2vec giving list of tokens to embedder not sentence). For example, you can change row

embeddings_batch = self.fasttext_model([' '.join(sen.split()[:self.opt['text_size']]) for sen in sentences])

to

embeddings_batch = self.fasttext_model([sen.split()[:self.opt['text_size']] for sen in sentences])

[deepankar27]

Yahoo!! at last I am got the intent 😄 after the asked change in intent_model.py

I converted the glove model to word2vec coz it was giving me wired exception & it's gone when I converted it to word2vec & loaded it.

x::Rate this series a 5

['RateBook'] x::give four stars to What the Day ['RateBook'] x::add the piano bar to my Cindy Wilson ['AddToPlaylist'] x::hello ['AddToPlaylist'] x::

Now question is how we will get the scores of predicted classes, I have to set some threshold as well. For an example I don't have "Hello" in my trained dataset but it still show the intent i expect it should be having very low confidence score.

Edit 2 After a small change in Intent_model.py I am getting prob scores but word hello is not in my dataset & it still gives score of 0.6.

**x::add the piano bar to my Cindy Wilson predictions[[0.8980047 0.08839173]]

['AddToPlaylist'] x::hello predictions[[0.62429315 0.33787638]] ['AddToPlaylist']**

And really a BIG thanks again.

[dilyararimovna]

For the next release we will change output to much more comfortable. For the current version you can just change

if len(args) == 1:
    pred = model(args)
else:
    pred = model([args])

to

if len(args) == 1:
    pred = model(args, predict_proba=True)
else:
    pred = model([args], predict_proba=True)

in DeepPavlov/blob/master/deeppavlov/core/commands/infer.py

[deepankar27]

Thanks a lot!! I will do the change:

Now the concern is bit diff:

x::add the piano bar to my Cindy Wilson
>> [0.8980047  0.08839173] <-- This is correct
x::cindy wilson let's go to bar
>> [0.5182335  0.46872252] <-- This is also fine
x::cindy wilson let's go to my bar
>> [0.67825717 0.3152743 ] <-- This is bit high
x::Cindy Wilson let's go to my bar
>> [0.6949899 0.3103463]   <-- This is pretty high but not relevant 

If you see it gives good score for known sentence "add the piano bar to my Cindy Wilson" but if I change the sentence but keeping almost same words then also it gives high scores. I was expecting a very low score for such scenarios. I going to change the kernel sizes (kernel_sizes_cnn) and see whether it improves or not but do you have suggestions on this?

[dilyararimovna]

Because as you can see in json config file you did not put some preprocessor only embedder and tokenizer. If you want, you can add any preprocessor. See examples in deeppvlov.models/preprocessors

I am glad you have succeeded with usage of DeepPavlov. Looking forward for the next release (hopefully, one-two weeks)!

[deepankar27]

Cool!! I will wait for it.. and I guess we have to set the preprocessors like this right? "pipe": [ { "name": "stream_spacy_tokenizer", "lowercase": "true",

[dilyararimovna]

as an element of pipe list you can add something like this:

"pipe": [
{
"in": ["x"], 
"out": ["x_preproc"],
"name": "registered_preprocessor" # registered name of preprocessor
},
{ # here the model starts for example     
"in": ["x_preproc"],
"in_y": ["y"],
"out": ["y_predicted"],
...
} 

[deepankar27]

for this kind of dataset is there any out of the box preprocessor (except str_lower) is available which I can use or do I need to create one & registered it in DeepPavlov-master\deeppavlov\models\preprocessors\squad_preprocessor.py ? Edit 2 Also can I use SquadPreprocessor as a preprocessor but looks like it has many parameters to set in the config as well.

[dilyararimovna]

you can either use existing preprocessors or create and registered your own dataset (to register do not forgot to include importing from file to init.py).

[deepankar27]

Thanks a lot for the support!! I am using the model now & get back to you with my observations I guess that will help you to improve further :)

[dilyararimovna]

I am glad to help you!