Tensorflow error while running benchmarking guide

[welly87]

Hi @koaning

I follow benchmarking guideline here https://rasahq.github.io/rasa-nlu-examples/benchmarking/

but found this error

(binus) Wellys-MacBook-Pro:rasa-demo wellytambunan$ rasa test nlu --config basic-bytepair.config.yml           --cross-validation --runs 1 --folds 2           --out gridresults/basic-bytepair-config
2020-08-14 10:22:31 INFO     rasa.cli.test  - Test model using cross validation.
Traceback (most recent call last):
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/rasa/__main__.py", line 92, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/rasa/cli/test.py", line 147, in run_nlu_test
    perform_nlu_cross_validation(config, nlu_data, output, vars(args))
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/rasa/test.py", line 243, in perform_nlu_cross_validation
    data, folds, nlu_config, output, **kwargs
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/rasa/nlu/test.py", line 1354, in cross_validate
    trainer = Trainer(nlu_config)
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/rasa/nlu/model.py", line 142, in __init__
    components.validate_requirements(cfg.component_names)
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/rasa/nlu/components.py", line 46, in validate_requirements
    from rasa.nlu import registry
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/rasa/nlu/registry.py", line 13, in <module>
    from rasa.nlu.classifiers.diet_classifier import DIETClassifier
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 9, in <module>
    import tensorflow_addons as tfa
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/tensorflow_addons/__init__.py", line 21, in <module>
    from tensorflow_addons import activations
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/tensorflow_addons/activations/__init__.py", line 21, in <module>
    from tensorflow_addons.activations.gelu import gelu
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/tensorflow_addons/activations/gelu.py", line 24, in <module>
    get_path_to_datafile("custom_ops/activations/_activation_ops.so"))
  File "/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 58, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
**tensorflow.python.framework.errors_impl.NotFoundError:** dlopen(/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/tensorflow_addons/custom_ops/activations/_activation_ops.so, 6): **Symbol not found:** __ZN10tensorflow11GetNodeAttrERKNS_9AttrSliceEN4absl11string_viewEPb
  Referenced from: /Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/tensorflow_addons/custom_ops/activations/_activation_ops.so
  Expected in: /Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.2.dylib
 in /Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/tensorflow_addons/custom_ops/activations/_activation_ops.so
(binus) Wellys-MacBook-Pro:rasa-demo wellytambunan$ 

[sara-tagger]

Thanks for raising this issue, @b-quachtran will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[koaning]

Ah. This might be because of the Tensorflow 2.3 update. I'll try to see if I can reproduce today. Until then; what version of Rasa/tensorflow are in your virtualenv?

[welly87]

i rebuild the experiment by using new conda environment.. still find this errors.. is that something related with tensorflow_text? dunno what is that one... quite new in TF world sorry ;)

    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: dlopen(/Users/wellytambunan/opt/anaconda3/envs/binus/lib/python3.6/site-packages/tensorflow_text/python/metrics/_text_similarity_metric_ops.dylib, 6): Symbol not found: __ZN10tensorflow15TensorShapeBaseINS_11TensorShapeEEC2EN4absl14lts_2020_02_254SpanIKxEE

[welly87]

i think it's 2.2.1...

[koaning]

Yeah I think it's an incompatible tensorflow version. I'll grab a coffee, have breakfast and I'll work on it.

[welly87]

sure.. thanks a lot.. really appreciated!

[koaning]

Mhm. This seems interesting. There's nothing tensorflow related that I am using in this library now that I check. I was briefly confused because this week I've been doing a lot of work on whatlies that does have the tensorflow dependencies.

Just to check. If you remove the rasa_nlu_examples component from your config.yml file. Does the error persist?

[koaning]

Also, could you locally run;

pip freeze | grep tensor

Feel free to list the results here in markdown.

[koaning]

I've got some extra tests running on some Rasa versions here. It looks green. Also just to check, how did you install Rasa in your virtualenv?

[welly87]

Also, could you locally run;

pip freeze | grep tensor

Feel free to list the results here in markdown.

tensorboard==2.1.1
tensorboard-plugin-wit==1.7.0
tensorflow==2.1.1
tensorflow-addons==0.7.1
tensorflow-estimator==2.1.0
tensorflow-hub==0.8.0
tensorflow-probability==0.9.0
tensorflow-text==2.3.0

[welly87]

Mhm. This seems interesting. There's nothing tensorflow related that I am using in this library now that I check. I was briefly confused because this week I've been doing a lot of work on whatlies that does have the tensorflow dependencies.

Just to check. If you remove the rasa_nlu_examples component from your config.yml file. Does the error persist?

do you mean this line as a whole? - name: rasa_nlu_examples.featurizers.dense.BytePairFeaturizer

[welly87]

I've got some extra tests running on some Rasa versions here. It looks green. Also just to check, how did you install Rasa in your virtualenv?

here's my cmd

conda create -n rasakata python=3.6

pip3 install rasa

pip install rasa-sdk

pip install nest_asyncio

pip3 install rasa-x --extra-index-url https://pypi.rasa.com/simple

git clone https://github.com/RasaHQ/rasa-nlu-examples

rasa test nlu --config basic-bytepair.config.yml --cross-validation --runs 1 --folds 2 --out gridresults/basic-bytepair-config

[koaning]

do you mean this line as a whole?

Yes. I want to make sure that if error is related to this repository or to Rasa.

Looking at your traceback it seems like the error is unrelated to this package. Looking at your installations it seems like tensorflow==2.1.1 and tensorflow-text==2.3.0 are not aligned. This difference in versions is possibly causing an error inside of core Rasa. Unrelated to this project.

If you could run the model with just Rasa components in config.yml (no components from this library) and if the error persists then we can rule out that the error is in this package. I think you can fix the issue by reinstalling tensorflow such that tensorflow and tensorflow-text have the same version.

[welly87]

Thank for your help!

Currently i'm trying to follow your instruction. Will let you know the result asap.

Cheers

[welly87]

Hi @koaning,

It works fine after removing the line. Here's basic-config.yml

language: en
pipeline:
- name: WhitespaceTokenizer
- name: CountVectorsFeaturizer
  OOV_token: oov.txt
  token_pattern: (?u)\b\w+\b
- name: CountVectorsFeaturizer
  analyzer: char_wb
  min_ngram: 1
  max_ngram: 4
- name: DIETClassifier
  epochs: 200

here's the command to run.

rasa test nlu --config basic-config.yml --cross-validation --runs 1 --folds 2 --out gridresults/basic-config

So any clue for this one? I'm not quite understand where the problem is...

Cheers

[welly87]

Hi @koaning, It works now! Not sure why. I use bytepair featurize.

1. I've created a clean conda environment
2. create new fresh project with rasa init --no-prompt
3. change the config.yml into these
4. edit the config to use indonesian (id)

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: id
pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: rasa_nlu_examples.featurizers.dense.BytePairFeaturizer
    lang: en
    vs: 1000
    dim: 25
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MemoizationPolicy
  - name: TEDPolicy
    max_history: 5
    epochs: 100
  - name: MappingPolicy

Basically i just do modification from the new project and adding this one

  - name: rasa_nlu_examples.featurizers.dense.BytePairFeaturizer
    lang: en
    vs: 1000
    dim: 25

I run rasa train. It works well.

So what's next? I think i should try changing the data/nlu.md into indonesian language dataset? Then run rasa train again?

Am i in the right direction?

Cheers

[koaning]

This is indeed in the right direction. For more information you can find the benchmarking guide on our documentation page.

There's also a video of me running a benchmark that might help as well https://www.youtube.com/watch?v=oj5oPGDlep4.

[koaning]

Note that there are many settings for the BytePair embeddings for the Indonesian language: https://nlp.h-its.org/bpemb/id/.

[koaning]

If you want to summarise the benchmarking results, you might enjoy rasalit.

[welly87]

thanks @koaning it's really great to see a dedication from rasa team. i would post some write up on this after i've got this working end to end..

[koaning]

Sure thing. Can I close down this issue for now?

[welly87]

sure.. thanks a lot