TyDunn
commented
2 years ago @mleimeister Given your work with #10391, any ideas of what might be going on here?
Closed indam23 closed 2 years ago
TyDunn
commented
2 years ago @mleimeister Given your work with #10391, any ideas of what might be going on here?
mleimeister
commented
2 years ago Hi @TyDunn @melindaloubser1, sorry I saw this notifcation just now. I ran the above config on the moodbot with chitchat responses example. With this setup, the error does not seem to be related to the use_text_as_label setting, since for me both
- name: ResponseSelector
epochs: 10
use_text_as_label: true
number_of_transformer_layers: 1and
- name: ResponseSelector
epochs: 10
use_text_as_label: false
number_of_transformer_layers: 1lead to the same issue. However, the error does not happen if additionally setting transfomer_size to a value >0.
I was not able to reproduce the issue using the second config
- name: ResponseSelector
epochs: 10
use_text_as_label: true
number_of_transformer_layers: 1
transformer_size: 256
hidden_layers_sizes:
text: []
label: []@melindaloubser1 Not sure why this isn't happening for me with the second config. Could this be dataset dependent? If so, the below analysis might be missing something. Stepping through the code, the root of the error message in the logs seems to come from the following:
ResponseSelector has as default number_of_transformer_layers: 0 and transformer_size: None. If the first is specified as >0, during training a parameter correction function sets the transformer size to a default value (see here and here). This is triggered in the DIETClassifier initializer, which is the parent class of ResponseSelector. Therefore during training, everything works fine.
During model loading, which happens when running rasa test nlu, before running the constructor at the end of DIET's _load function, the model weights are loaded here. This ends up in the transformer code, where the function prepare_transformer_layer is triggered. However, because this is before the constructor of ResponseSelector is run, the parameter correction step has not happened yet and the units input parameter, which comes from the transformer_size in the config, is None. This leads to the error as in the logs above
File "/Users/matthias/Workspace/rasa3/rasa/utils/tensorflow/rasa_layers.py", line 1077, in prepare_transformer_layer
units * 4,
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'A potential solution would be to also add a parameter correction step in the load function, before the model loading is happening. Alternatively, clarify in the docs that both those parameters have to be set?
jupyterjazz
commented
2 years ago Following @mleimeister's insights I created a draft solution on this branch.
@melindaloubser1 I was also unable to reproduce the error with the following configuration:
- name: ResponseSelector
epochs: 10
use_text_as_label: true
number_of_transformer_layers: 1
transformer_size: 256
hidden_layers_sizes:
text: []
label: []Can you please try reproducing the error with above-mentioned config file using this branch? In case it still exists, can you help me reproduce it?
Rasa Open Source version
3.0.2
Python version
3.8
What operating system are you using?
OSX
What happened?
Trained a model with the following config without error:
config.yml
But running
rasa test nlufails with the error below. I tried a few variations on transformer configuration with the same result e.g.Command / Request
Relevant log output
Definition of done