search

koenvervloesem / rasa-docker-arm

Rasa Docker image for ARMv7. Runs on a Raspberry Pi.
https://hub.docker.com/r/koenvervloesem/rasa
MIT License
19 stars 2 forks source link

Cannot convert a symbolic Tensor (dialogue_encoder/strided_slice_4:0) to a numpy array #3

Closed koenvervloesem closed 3 years ago

koenvervloesem commented 3 years ago

Starting from Rasa 1.10.21, training fails with the following error:

pi@pi-yellow:~/test_rasa $ docker-compose up -d
Recreating test_rasa_rasa_1 ... done
pi@pi-yellow:~/test_rasa $ docker-compose run rasa init
Welcome to Rasa! 🤖

To get started quickly, an initial project will be created.
If you need some help, check out the documentation at https://rasa.com/docs/rasa.
Now let's start! 👇🏽

? Please enter a path where the project will be crea
ted [default: current directory] .
? Directory '/app' is not empty. Continue?  Yes
Created project directory at '/app'.
Finished creating project structure.
? Do you want to train an initial model? 💪🏽  Yes
Training an initial model...
2021-02-08 19:15:03 INFO     rasa.model  - Data (version) for Core model section changed.
2021-02-08 19:15:03 INFO     rasa.model  - Data (version) for NLU model section changed.
Training Core model...
Processed Story Blocks: 100%|█| 5/5 [00:00<00:00, 59
Processed Story Blocks: 100%|█| 5/5 [00:00<00:00, 30
Processed Story Blocks: 100%|█| 5/5 [00:00<00:00, 71
Processed Story Blocks: 100%|█| 5/5 [00:00<00:00, 49
Processed trackers: 100%|█| 5/5 [00:00<00:00, 249.91
Processed actions: 16it [00:00, 1342.53it/s, # examples=16]
Processed trackers: 100%|█| 231/231 [00:02<00:00, 11
Traceback (most recent call last):
  File "/opt/venv/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/opt/venv/lib/python3.7/site-packages/rasa/__main__.py", line 92, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/opt/venv/lib/python3.7/site-packages/rasa/cli/scaffold.py", line 206, in run
    init_project(args, path)
  File "/opt/venv/lib/python3.7/site-packages/rasa/cli/scaffold.py", line 118, in init_project
    print_train_or_instructions(args, path)
  File "/opt/venv/lib/python3.7/site-packages/rasa/cli/scaffold.py", line 58, in print_train_or_instructions
    args.model = rasa.train(domain, config, training_files, output)
  File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 50, in train
    additional_arguments=additional_arguments,
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 101, in train_async
    additional_arguments,
  File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 188, in _train_async_internal
    additional_arguments=additional_arguments,
  File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 223, in _do_training
    additional_arguments=additional_arguments,
  File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 361, in _train_core_with_validated_data
    additional_arguments=additional_arguments,
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/train.py", line 66, in train
    agent.train(training_data, **additional_arguments)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/agent.py", line 752, in train
    self.policy_ensemble.train(training_trackers, self.domain, **kwargs)                                                                                                                                   [7/1116]
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/policies/ensemble.py", line 124, in train
    policy.train(training_trackers, domain, **kwargs)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/policies/ted_policy.py", line 325, in train
    batch_strategy=self.config[BATCH_STRATEGY],            
  File "/opt/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/models.py", line 131, in fit
    ) = self._get_tf_train_functions(eager, model_data, batch_strategy)
  File "/opt/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/models.py", line 348, in _get_tf_train_functions
    train_dataset_function, self.train_on_batch, eager, "train"
  File "/opt/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/models.py", line 330, in _get_tf_call_model_function
    tf_call_model_function(next(iter(init_dataset)))
  File "/opt/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
    result = self._call(*args, **kwds)
  File "/opt/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 615, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/opt/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 497, in _initialize
    *args, **kwds))                                                                     
  File "/opt/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2389, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/opt/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/opt/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
    capture_by_value=self._capture_by_value),       
  File "/opt/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/opt/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/opt/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 968, in wrapper
    raise e.ag_error_metadata.to_exception(e)                                    
NotImplementedError: in converted code:      

    /opt/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/models.py:197 train_on_batch  *
        prediction_loss = self.batch_loss(batch_in)                                           
    /opt/venv/lib/python3.7/site-packages/rasa/core/policies/ted_policy.py:610 batch_loss  *
        dialogue_embed, mask = self._emebed_dialogue(dialogue_in)                                           
    /opt/venv/lib/python3.7/site-packages/rasa/core/policies/ted_policy.py:578 _emebed_dialogue  *
        dialogue_transformed = self._tf_layers["transformer"](                 
    /opt/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py:778 __call__
        outputs = call_fn(cast_inputs, *args, **kwargs)                    
    /opt/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/transformer.py:619 call  *
        pad_mask = tf.minimum(
    /opt/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/transformer.py:586 _look_ahead_pad_mask  *
        pad_mask = 1 - tf.linalg.band_part(tf.ones((max_position, max_position)), -1, 0)
    /opt/venv/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py:2659 ones
        output = _constant_if_small(one, shape, dtype, name)
    /opt/venv/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py:2391 _constant_if_small 
        if np.prod(shape) < 1000:             
    <__array_function__ internals>:6 prod                                           

    /opt/venv/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3031 prod        
        keepdims=keepdims, initial=initial, where=where)
    /opt/venv/lib/python3.7/site-packages/numpy/core/fromnumeric.py:87 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)                                          
    /opt/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:728 __array__
        " array.".format(self.name))                                                

    NotImplementedError: Cannot convert a symbolic Tensor (dialogue_encoder/strided_slice_4:0) to a numpy array.

If anyone has a clue about this, please let me know.

indam23 commented 3 years ago

Installing numpy==1.19.5 fixed the issue when running Rasa==1.10.22 locally. If the numpy version is too high you should see errors like this during installation:

tensorflow 2.1.3 has requirement numpy<1.19.0,>=1.16.0, but you'll have numpy 1.20.1 which is incompatible.
koenvervloesem commented 3 years ago

Many thanks for the help, @melindaloubser1! Fixing the numpy version to 1.19.5 results in a working 1.20.21 container on my Raspberry Pi, now building 1.20.22.

koenvervloesem commented 3 years ago

1.20.22 works too, I published both images to Docker Hub.

Keep up the good work with Rasa, I hope to be able to catch up with the Rasa 2.x releases for ARM soon.