search

RasaHQ / rasa

💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants
https://rasa.com/docs/rasa/
Apache License 2.0
18.94k stars 4.64k forks source link

Unclear error message if error in component/policy during training #5355

Closed erohmensing closed 4 years ago

erohmensing commented 4 years ago

If you try to train a model and there is a tensorflow error, the exception around "Not able to load a model" will print after this error and make it less intuitive. For example this error which I believe was to do with the tensorflow version (see the first error) was masked as a question of "Why can't it find the keras policy?" when the real issue was that something in the embedding policy's tf imports was broken. 

Traceback (most recent call last):
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/core/policies/ensemble.py", line 320, in from_dict
    constr_func = registry.policy_from_module_path(policy_name)
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/core/registry.py", line 21, in policy_from_module_path
    module_path, lookup_path="rasa.core.policies.registry"
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/utils/common.py", line 208, in class_from_module_path
    m = importlib.import_module(lookup_path)
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/core/policies/registry.py", line 5, in <module>
    from rasa.core.policies.embedding_policy import EmbeddingPolicy
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/core/policies/embedding_policy.py", line 22, in <module>
    from rasa.utils import train_utils
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/utils/train_utils.py", line 10, in <module>
    from tensor2tensor.models.transformer import (
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/tensor2tensor/models/__init__.py", line 25, in <module>
    from tensor2tensor.layers import modalities  # pylint: disable=g-import-not-at-top
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/tensor2tensor/layers/modalities.py", line 28, in <module>
    from tensor2tensor.layers import common_attention
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/tensor2tensor/layers/common_attention.py", line 952, in <module>
    def attention_bias_to_padding(attention_bias, cast_fn=tf.to_float):
AttributeError: module 'tensorflow' has no attribute 'to_float'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ella/miniconda3/envs/pre-poetry/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/__main__.py", line 76, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/cli/train.py", line 76, in train
    additional_arguments=extract_additional_arguments(args),
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/train.py", line 50, in train
    additional_arguments=additional_arguments,
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/train.py", line 101, in train_async
    additional_arguments,
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/train.py", line 188, in _train_async_internal
    additional_arguments=additional_arguments,
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/train.py", line 223, in _do_training
    additional_arguments=additional_arguments,
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/train.py", line 361, in _train_core_with_validated_data
    additional_arguments=additional_arguments,
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/core/train.py", line 44, in train
    policies = config.load(policy_config)
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/core/config.py", line 28, in load
    return PolicyEnsemble.from_dict(config_data)
  File "/Users/ella/miniconda3/envs/pre-poetry/lib/python3.7/site-packages/rasa/core/policies/ensemble.py", line 331, in from_dict
    "".format(policy_name)
rasa.core.policies.ensemble.InvalidPolicyConfig: Module for policy 'KerasPolicy' could not be loaded. Please make sure the name is a valid policy.

see also https://forum.rasa.com/t/memoizationpolicy-could-not-be-loaded/22936 https://forum.rasa.com/t/exception-failed-to-find-input-channel-class-for-socketio-connector-socketioinput/28730 https://rasaworkshop.slack.com/archives/C012QL4AV5H/p1589207054182500

tabergma commented 4 years ago

@erohmensing Is this issue still relevant as the linked forum post is already quite old? Do you know how we can reproduce it?

erohmensing commented 4 years ago

Hm i had gotten it when upgrading to 1.8 when the tensorflow-text version was wrong. Let me see if I can try to repro

tabergma commented 4 years ago

@erohmensing Tried to reproduce this error but failed. We also removed the dependency tensor2tensor which caused this issue. Also not sure what to do about it as I'm not sure how we would catch such an error.

erohmensing commented 4 years ago

Okay, then it may indeed not be replicable. It is probably just a coincidence that the AttributeError inside tensorflow just happens to match what we except here: https://github.com/RasaHQ/rasa/blob/master/rasa/core/policies/ensemble.py#L314

So unless there is a way to check that for specific arguments about the attribute or something, we can probably close this.

tabergma commented 4 years ago

The error in the description should not happen anymore as we removed the tensor2tensor dependency and it was related to that. Also not sure if it makes sense to add a check for a very specific error, that might not even occur again. I think we can close this issue. Also the stracktrace shows the origin of the error so we should be able to trace this if it happens again. Thanks for your support!

eziolotta commented 4 years ago

I have same issue when train a model with latest Rasa version I have istalled Rasa 1.10 in a virtual env with Python 3.6.5 in Windows

With tensorflow 2.1 I had an exception, documenting I read that a downgrade of tensorflow to 2.0 was needed. After downgrade I had other exception "InvalidPolicyConfig: Module for policy 'KerasPolicy' could not be loaded"

thanks, help me!!!...

########################################################## First Exception - with tensorflow 2.1

Traceback (most recent call last): File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_core\python\pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import * File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\imp.py", line 343, in load_dynamic return _load(spec) ImportError: DLL load failed: Impossibile trovare il modulo specificato.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\ezioDev\maven\git\bot\virtualenv_rasa_1.10\Scripts\rasa.exe__main.py", line 7, in File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa__main.py", line 82, in main set_log_level(log_level) File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\utils\common.py", line 71, in set_log_level update_tensorflow_log_level() File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\utils\common.py", line 112, in update_tensorflow_log_level import tensorflow as tf File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow__init.py", line 101, in from tensorflow_core import * File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_core__init__.py", line 40, in from tensorflow.python.tools import module_util as _module_util File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow\init.py", line 50, in getattr__ module = self._load() File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow\init__.py", line 44, in _load module = _importlib.import_module(self.name) File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\importlib__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_core\python\init__.py", line 49, in from tensorflow.python import pywrap_tensorflow File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_core\python\pywrap_tensorflow.py", line 74, in raise ImportError(msg) ImportError: Traceback (most recent call last): File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_core\python\pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import * File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\imp.py", line 343, in load_dynamic return _load(spec) ImportError: DLL load failed: Impossibile trovare il modulo specificato.

Failed to load the native TensorFlow runtime.

############################################################ Second Exception - after downgrade tensorflow 2.0

Training Core model... Traceback (most recent call last): File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\core\policies\ensemble.py", line 308, in from_dict constr_func = registry.policy_from_module_path(policy_name) File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\core\registry.py", line 21, in policy_from_module_path module_path, lookup_path="rasa.core.policies.registry" File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\utils\common.py", line 208, in class_from_module_path m = importlib.import_module(lookup_path) File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\importlib__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 955, in _find_and_load_unlocked File "", line 665, in _load_unlocked File "", line 678, in exec_module File "", line 219, in _call_with_frames_removed File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\core\policies\registry.py", line 5, in from rasa.core.policies.ted_policy import TEDPolicy File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\core\policies\ted_policy.py", line 8, in import tensorflow_addons as tfa File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_addons__init.py", line 21, in from tensorflow_addons import activations File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_addons\activations\init__.py", line 21, in from tensorflow_addons.activations.gelu import gelu File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\tensorflow_addons\activations\gelu.py", line 27, in @tf.keras.utils.register_keras_serializable(package='Addons') AttributeError: module 'tensorflow_core.keras.utils' has no attribute 'register_keras_serializable'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\ezioDev\maven\git\bot\virtualenv_rasa_1.10\Scripts\rasa.exe__main.py", line 7, in File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\main__.py", line 91, in main cmdline_arguments.func(cmdline_arguments) File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\cli\train.py", line 76, in train additional_arguments=extract_additional_arguments(args), File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\train.py", line 50, in train additional_arguments=additional_arguments, File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py", line 468, in run_until_complete return future.result() File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\train.py", line 101, in train_async additional_arguments, File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\train.py", line 188, in _train_async_internal additional_arguments=additional_arguments, File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\train.py", line 223, in _do_training additional_arguments=additional_arguments, File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\train.py", line 361, in _train_core_with_validated_data additional_arguments=additional_arguments, File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\core\train.py", line 43, in train policies = config.load(policy_config) File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\core\config.py", line 28, in load return PolicyEnsemble.from_dict(config_data) File "c:\eziodev\maven\git\bot\virtualenv_rasa_1.10\lib\site-packages\rasa\core\policies\ensemble.py", line 319, in from_dict "".format(policy_name) rasa.core.policies.ensemble.InvalidPolicyConfig: Module for policy 'KerasPolicy' could not be loaded. Please make sure the name is a valid policy.

################################################# Virtual env istalled packages

absl-py==0.9.0 aiofiles==0.5.0 aiohttp==3.6.2 APScheduler==3.6.3 astor==0.8.1 async-generator==1.10 async-timeout==3.0.1 attrs==19.3.0 blis==0.2.4 boto3==1.13.4 botocore==1.16.4 cachetools==4.1.0 certifi==2020.4.5.1 cffi==1.14.0 chardet==3.0.4 cloudpickle==1.3.0 colorama==0.4.3 colorclass==2.2.0 coloredlogs==10.0 colorhash==1.0.2 contextvars==2.4 cryptography==2.9.2 cycler==0.10.0 cymem==2.0.3 decorator==4.4.2 dnspython==1.16.0 docopt==0.6.2 docutils==0.15.2 fbmessenger==6.0.0 future==0.18.2 gast==0.2.2 gevent==1.5.0 google-auth==1.14.2 google-auth-oauthlib==0.4.1 google-pasta==0.2.0 greenlet==0.4.15 grpcio==1.28.1 h11==0.8.1 h2==3.2.0 h5py==2.10.0 hpack==3.0.0 hstspreload==2020.5.5 httplib2==0.17.3 httptools==0.1.1 httpx==0.9.3 humanfriendly==8.2 hyperframe==5.2.0 idna==2.9 idna-ssl==1.1.0 immutables==0.12 importlib-metadata==1.6.0 it-core-news-sm @ https://github.com/explosion/spacy-models/releases/download/it_core_news_sm-2.1.0/it_core_news_sm-2.1.0.tar.gz jmespath==0.9.5 joblib==0.14.1 jsonpickle==1.4.1 jsonschema==3.0.2 kafka-python==1.4.7 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.0 kiwisolver==1.2.0 Markdown==3.2.1 matplotlib==3.2.1 mattermostwrapper==2.2 multidict==4.7.5 murmurhash==1.0.2 networkx==2.4 numpy==1.18.4 oauth2client==4.1.3 oauthlib==3.1.0 opt-einsum==3.2.1 packaging==19.0 pika==1.1.0 plac==0.9.6 preshed==2.0.1 prompt-toolkit==2.0.10 protobuf==3.11.3 psycopg2-binary==2.8.5 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.20 pydot==1.4.1 PyJWT==1.7.1 pykwalify==1.7.0 pymongo==3.8.0 PyMySQL==0.9.3 pyparsing==2.4.7 pyreadline==2.1 pyrsistent==0.16.0 PySocks==1.7.1 python-crfsuite==0.9.7 python-dateutil==2.8.1 python-engineio==3.12.1 python-socketio==4.5.1 python-telegram-bot==12.7 pytz==2019.3 PyYAML==5.3.1 questionary==1.5.2 rasa==1.10.0 rasa-sdk==1.10.0 redis==3.5.0 requests==2.23.0 requests-oauthlib==1.3.0 requests-toolbelt==0.9.1 rfc3986==1.4.0 rocketchat-API==1.3.1 rsa==4.0 ruamel.yaml==0.16.10 ruamel.yaml.clib==0.2.0 s3transfer==0.3.3 sanic==19.12.2 Sanic-Cors==0.10.0.post3 sanic-jwt==1.4.1 Sanic-Plugins-Framework==0.9.2 scikit-learn==0.22.2.post1 scipy==1.4.1 six==1.14.0 sklearn-crfsuite==0.3.6 slackclient==2.5.0 sniffio==1.1.0 spacy==2.1.4 SQLAlchemy==1.3.16 srsly==1.0.2 tabulate==0.8.7 tensorboard==2.0.2 tensorflow==2.0.0 tensorflow-addons==0.7.1 tensorflow-estimator==2.0.1 tensorflow-hub==0.8.0 tensorflow-probability==0.9.0 termcolor==1.1.0 terminaltables==3.1.0 thinc==7.0.8 tornado==6.0.4 tqdm==4.45.0 twilio==6.26.3 typing-extensions==3.7.4.2 tzlocal==2.0.0 ujson==2.0.3 urllib3==1.25.9 wasabi==0.6.0 wcwidth==0.1.9 webexteamssdk==1.3 websockets==8.1 Werkzeug==1.0.1 wrapt==1.12.1 yarl==1.4.2 zipp==3.1.0

#################################