Training aborts with GeneratorDataset iterator: Cancelled

[MPARANTHAMAN]

2020-03-04 21:47:31.760062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll Welcome to Rasa! 🤖

To get started quickly, an initial project will be created. If you need some help, check out the documentation at https://rasa.com/docs/rasa.

Created project directory at 'C:\Users\paranthaman\Desktop\paran'. Finished creating project structure. Training an initial model... Training Core model... Processed Story Blocks: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1671.70it/s, # trackers=1] Processed Story Blocks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 320.78it/s, # trackers=5] Processed Story Blocks: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 117.96it/s, # trackers=20] Processed Story Blocks: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 270.62it/s, # trackers=24] Processed trackers: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 899.72it/s, # actions=16] Processed actions: 16it [00:00, 132.02it/s, # examples=16] Processed trackers: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 231/231 [00:01<00:00, 91.16it/s, # actions=126] 2020-03-04 21:49:56.502934: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll 2020-03-04 21:49:56.511111: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error 2020-03-04 21:49:56.632605: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-JTAPB16 2020-03-04 21:49:56.637661: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-JTAPB16 2020-03-04 21:49:56.716448: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2020-03-04 21:49:57.824309: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled Epochs: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:13<00:00, 9.66it/s, t_loss=0.061, loss=0.002, acc=1.000] 2020-03-04 21:50:26 INFO rasa.utils.tensorflow.models - Finished training. 2020-03-04 21:50:27 INFO rasa.core.agent - Persisted model to 'C:\Users\PARANT~1\AppData\Local\Temp\tmp2looa6mc\core' Core model training completed. Training NLU model... 2020-03-04 21:50:28 INFO rasa.nlu.selectors.response_selector - Retrieval intent parameter was left to its default value. This response selector will be trained on training examples combining all retrieval intents. 2020-03-04 21:50:28 INFO rasa.nlu.training_data.training_data - Training data stats:

• intent examples: 43 (7 distinct intents)
• Found intents: 'mood_unhappy', 'greet', 'mood_great', 'goodbye', 'affirm', 'bot_challenge', 'deny'
• Number of response examples: 0 (0 distinct response)
• entity examples: 0 (0 distinct entities)
• found entities:

2020-03-04 21:50:28 INFO rasa.nlu.model - Starting to train component WhitespaceTokenizer 2020-03-04 21:50:28 INFO rasa.nlu.model - Finished training component. 2020-03-04 21:50:28 INFO rasa.nlu.model - Starting to train component RegexFeaturizer 2020-03-04 21:50:28 INFO rasa.nlu.model - Finished training component. 2020-03-04 21:50:28 INFO rasa.nlu.model - Starting to train component LexicalSyntacticFeaturizer 2020-03-04 21:50:28 INFO rasa.nlu.model - Finished training component. 2020-03-04 21:50:28 INFO rasa.nlu.model - Starting to train component CountVectorsFeaturizer 2020-03-04 21:50:28 INFO rasa.nlu.model - Finished training component. 2020-03-04 21:50:28 INFO rasa.nlu.model - Starting to train component CountVectorsFeaturizer 2020-03-04 21:50:28 INFO rasa.nlu.model - Finished training component. 2020-03-04 21:50:28 INFO rasa.nlu.model - Starting to train component DIETClassifier 2020-03-04 21:50:29.064813: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled Epochs: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:12<00:00, 9.17it/s, t_loss=1.334, i_loss=0.090, e_loss=0.000, i_acc=1.000, e_f1=0.000] 2020-03-04 21:50:54 INFO rasa.utils.tensorflow.models - Finished training. 2020-03-04 21:50:54 INFO rasa.nlu.model - Finished training component. 2020-03-04 21:50:54 INFO rasa.nlu.model - Starting to train component EntitySynonymMapper 2020-03-04 21:50:54 INFO rasa.nlu.model - Finished training component. 2020-03-04 21:50:54 INFO rasa.nlu.model - Starting to train component ResponseSelector 2020-03-04 21:50:54 INFO rasa.nlu.model - Finished training component. 2020-03-04 21:50:55 INFO rasa.nlu.model - Successfully saved model into 'C:\Users\PARANT~1\AppData\Local\Temp\tmp2looa6mc\nlu' NLU model training completed.

[sara-tagger]

Thanks for the issue, @JustinaPetr will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

[mxgordon]

I have the exact same issue and when I talk to it over the command line, it also is printing errors.

? Do you want to speak to the trained assistant on the command line? 🤖  Yes
2020-03-13 10:39:33 INFO     root  - Starting Rasa server on http://localhost:5005
2020-03-13 10:39:35.308328: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-03-13 10:39:37.714549: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
Bot loaded. Type a message and press enter (use '/stop' to exit):
Your input ->  how is it going                                                                                         2020-03-13 10:39:59.402914: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-03-13 10:40:00.235933: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-03-13 10:40:00.282991: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
Hey! How are you?
Your input ->     

[Will-Hardman]

I am also receiving this error. Using the latest version of RASA from pypy. It also appears that this could be preventing the from_entity() function from correctly extracting entities from user input.

[Will-Hardman]

@MPARANTHAMAN @JohnnyWobble are you both seeing this error when running one of the out-of-the-box examples, or have you build your own bots?

[JustinaPetr]

Hi @MPARANTHAMAN sorry for the late response. This looks like a tensorflow error. Could you please verify which tensorflow and Python versions are you using?

[MPARANTHAMAN]

Okk sir ,thanks for ur response

On Mon, Mar 16, 2020, 2:24 PM Justina Petraityte notifications@github.com wrote:

Hi @MPARANTHAMAN https://github.com/MPARANTHAMAN sorry for the late response. This looks like a tensorflow error. Could you please verify which tensorflow and Python versions are you using?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RasaHQ/rasa/issues/5372#issuecomment-599417056, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKHAJTK3DX3K7GZF6XXLJCTRHXSK7ANCNFSM4LBNHBAQ .

[Will-Hardman]

@JustinaPetr I have exactly the same issue as @MPARANTHAMAN. I am using Python 3.7 Anaconda with Rasa 1.8.1 and Tensorflow 2.1.1. The DIETClassifier successfully trains a model for intent classification, so the error seems to relate solely to the entity extraction model.

[chakra-ai]

Hi, @JustinaPetr I am getting the same issue as highlighted above. I am using Python 3.6.8 and Tensorflow 2.1.0. This is just after DIET Classifier. 2020-03-17 17:41:49 INFO rasa.nlu.model - Starting to train component DIETClassifier 2020-03-17 17:41:49.700799: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled

Could you please tell me what would be the issue.

Thanks.

[tmbo]

@dakshvar22 do you know what might be going on here?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to inactivity. Please create a new issue if you need more help.