`rasa init` breaks on Linux when you try to train a model in 3.0.0rc01

koaning commented 3 years ago

I just installed Rasa on a new gitpod instance. You can see the versions below.

> python -m rasa --version
Rasa Version      :         3.0.0rc1
Minimum Compatible Version: 2.8.9
Rasa SDK Version  :         3.0.0rc1
Rasa X Version    :         None
Python Version    :         3.7.12
Operating System  :         Linux-5.4.0-1051-gke-x86_64-with-debian-11.0
Python Path       :         /usr/local/bin/python

When I now run rasa init here's what I see;

> python -m rasa init 
┌────────────────────────────────────────────────────────────────────────────────┐
│ Rasa Open Source reports anonymous usage telemetry to help improve the product │
│ for all its users.                                                             │
│                                                                                │
│ If you'd like to opt-out, you can use `rasa telemetry disable`.                │
│ To learn more, check out https://rasa.com/docs/rasa/telemetry/telemetry.       │
└────────────────────────────────────────────────────────────────────────────────┘
Welcome to Rasa! 🤖

To get started quickly, an initial project will be created.
If you need some help, check out the documentation at https://rasa.com/docs/rasa.
Now let's start! 👇🏽

? Please enter a path where the project will be created [default: current directory] rasa-ini                                                                                           
? Path 'rasa-ini' does not exist 🧐. Create path? Yes                                                                                                                                   
Created project directory at '/workspace/rasa-examples/rasa-ini/rasa-ini'.
Finished creating project structure.
? Do you want to train an initial model? 💪🏽 Yes                                                                                                                                       
Training an initial model...
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/__main__.py", line 139, in <module>
    main()
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/__main__.py", line 121, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/cli/scaffold.py", line 238, in run
    init_project(args, path)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/cli/scaffold.py", line 132, in init_project
    print_train_or_instructions(args)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/cli/scaffold.py", line 68, in print_train_or_instructions
    create_output_path(),
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/api.py", line 103, in train
    from rasa.model_training import train
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/model_training.py", line 16, in <module>
    import rasa.engine.validation
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/engine/validation.py", line 20, in <module>
    from rasa.core.policies.policy import PolicyPrediction
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/core/policies/policy.py", line 26, in <module>
    from rasa.core.featurizers.tracker_featurizers import TrackerFeaturizer
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/core/featurizers/tracker_featurizers.py", line 12, in <module>
    from rasa.core.featurizers.single_state_featurizer import SingleStateFeaturizer
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/core/featurizers/single_state_featurizer.py", line 7, in <module>
    from rasa.nlu.extractors.extractor import EntityTagSpec
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/nlu/extractors/extractor.py", line 30, in <module>
    import rasa.utils.train_utils
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/utils/train_utils.py", line 32, in <module>
    from rasa.utils.tensorflow.callback import RasaTrainingLogger, RasaModelCheckpoint
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/utils/tensorflow/callback.py", line 5, in <module>
    import tensorflow as tf
  File "/workspace/.pip-modules/lib/python3.7/site-packages/tensorflow/__init__.py", line 438, in <module>
    _ll.load_library(_main_dir)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 154, in load_library
    py_tf.TF_LoadLibrary(lib)
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb

Seems like a Tensorflow issue.

> python -m pip freeze | grep tensor
tensorflow==2.6.1
tensorflow-addons==0.14.0
tensorflow-estimator==2.6.0
tensorflow-text==2.6.0

I tried downgrading to tensorflow 2.6.0, but to no avail.

koaning commented 3 years ago

I can confirm that this error does not occur on my Intel Mac.

koaning commented 2 years ago

I can confirm that this error does not occur on gitpod when I use Rasa version 2.8.

koaning commented 2 years ago

Note that this error persists in RC2

> python -m rasa --version
Rasa Version      :         3.0.0rc2
Minimum Compatible Version: 3.0.0rc1
Rasa SDK Version  :         3.0.0rc1
Rasa X Version    :         None
Python Version    :         3.7.12
Operating System  :         Linux-5.4.0-1051-gke-x86_64-with-debian-11.0
Python Path       :         /usr/local/bin/python

Again:

> python -m rasa init 
┌────────────────────────────────────────────────────────────────────────────────┐
│ Rasa Open Source reports anonymous usage telemetry to help improve the product │
│ for all its users.                                                             │
│                                                                                │
│ If you'd like to opt-out, you can use `rasa telemetry disable`.                │
│ To learn more, check out https://rasa.com/docs/rasa/telemetry/telemetry.       │
└────────────────────────────────────────────────────────────────────────────────┘
Welcome to Rasa! 🤖

To get started quickly, an initial project will be created.
If you need some help, check out the documentation at https://rasa.com/docs/rasa.
Now let's start! 👇🏽

? Please enter a path where the project will be created [default: current directory] rasa-ini                                                                                           
? Path 'rasa-ini' does not exist 🧐. Create path? Yes                                                                                                                                   
Created project directory at '/workspace/rasa-examples/rasa-ini/rasa-ini'.
Finished creating project structure.
? Do you want to train an initial model? 💪🏽 Yes                                                                                                                                       
Training an initial model...
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/__main__.py", line 139, in <module>
    main()
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/__main__.py", line 121, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/cli/scaffold.py", line 238, in run
    init_project(args, path)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/cli/scaffold.py", line 132, in init_project
    print_train_or_instructions(args)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/cli/scaffold.py", line 68, in print_train_or_instructions
    DEFAULT_MODELS_PATH,
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/api.py", line 103, in train
    from rasa.model_training import train
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/model_training.py", line 16, in <module>
    import rasa.engine.validation
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/engine/validation.py", line 20, in <module>
    from rasa.core.policies.policy import PolicyPrediction
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/core/policies/policy.py", line 26, in <module>
    from rasa.core.featurizers.tracker_featurizers import TrackerFeaturizer
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/core/featurizers/tracker_featurizers.py", line 12, in <module>
    from rasa.core.featurizers.single_state_featurizer import SingleStateFeaturizer
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/core/featurizers/single_state_featurizer.py", line 7, in <module>
    from rasa.nlu.extractors.extractor import EntityTagSpec
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/nlu/extractors/extractor.py", line 30, in <module>
    import rasa.utils.train_utils
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/utils/train_utils.py", line 32, in <module>
    from rasa.utils.tensorflow.callback import RasaTrainingLogger, RasaModelCheckpoint
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/utils/tensorflow/callback.py", line 5, in <module>
    import tensorflow as tf
  File "/workspace/.pip-modules/lib/python3.7/site-packages/tensorflow/__init__.py", line 438, in <module>
    _ll.load_library(_main_dir)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 154, in load_library
    py_tf.TF_LoadLibrary(lib)
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb

usc-m commented 2 years ago

https://github.com/tensorflow/text/issues/385 This might be relevant - does reinstalling TF in the gitpod fix it?

koaning commented 2 years ago

I can confirm that pip install --upgrade tensorflow fixes things. Note that this installs tensorflow 2.7.

tensorflow==2.7.0
tensorflow-addons==0.14.0
tensorflow-estimator==2.7.0
tensorflow-io-gcs-filesystem==0.22.0
tensorflow-text==2.6.0

koaning commented 2 years ago

I just tried running it on gitpod with rc3. Still persists.

> gitpod /workspace/rasa-examples/rasa-ini $ python -m rasa train
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/__main__.py", line 139, in <module>
    main()
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/__main__.py", line 121, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/cli/train.py", line 59, in <lambda>
    train_parser.set_defaults(func=lambda args: run_training(args, can_exit=True))
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/cli/train.py", line 103, in run_training
    finetuning_epoch_fraction=args.epoch_fraction,
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/api.py", line 103, in train
    from rasa.model_training import train
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/model_training.py", line 16, in <module>
    import rasa.engine.validation
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/engine/validation.py", line 20, in <module>
    from rasa.core.policies.policy import PolicyPrediction
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/core/policies/policy.py", line 26, in <module>
    from rasa.core.featurizers.tracker_featurizers import TrackerFeaturizer
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/core/featurizers/tracker_featurizers.py", line 12, in <module>
    from rasa.core.featurizers.single_state_featurizer import SingleStateFeaturizer
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/core/featurizers/single_state_featurizer.py", line 7, in <module>
    from rasa.nlu.extractors.extractor import EntityTagSpec
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/nlu/extractors/extractor.py", line 30, in <module>
    import rasa.utils.train_utils
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/utils/train_utils.py", line 32, in <module>
    from rasa.utils.tensorflow.callback import RasaTrainingLogger, RasaModelCheckpoint
  File "/workspace/.pip-modules/lib/python3.7/site-packages/rasa/utils/tensorflow/callback.py", line 5, in <module>
    import tensorflow as tf
  File "/workspace/.pip-modules/lib/python3.7/site-packages/tensorflow/__init__.py", line 438, in <module>
    _ll.load_library(_main_dir)
  File "/workspace/.pip-modules/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 154, in load_library
    py_tf.TF_LoadLibrary(lib)
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb

joejuzl commented 2 years ago

Adding this slack comment for posterity:

So I think this problem is specifically related to the fact that we are updating rasa from inside and old docker container. It's also an error from tensorflow, as just running import tensorflow from within a python interpreter causes the error to be thrown.... I'm still not exactly sure what is causing it as it's hard to replicate outside of that one prebuilt gitpod container. I have created a fork of the repo where the docker file explicitly installs 3.0.0rc3 and everything works ok - so it's not an issue with Rasa 3.0.0 on Linux per say. I think my suggestion would be to rebuild the prebuild gitpod container once we release 3.0.0.

RasaHQ / rasa

`rasa init` breaks on Linux when you try to train a model in 3.0.0rc01 #10098