impossible for RasaNLUInterpreter to read the .pkl extension files

[antoinecomp]

Rasa Core version:

(MoodbotEnv) mike@mike-thinks:~/Programing/Rasa_tutorial/moodbot4$ pip list :
...
rasa-core (0.9.0a3)
rasa-nlu (0.12.3)

Python version:

(MoodbotEnv) mike@mike-thinks:~/Programing/Rasa_tutorial/moodbot4$ python -V 
Python 3.5.2

Operating system : Linux 16.04

Issue:

When trying to launch file train_online.py, the actual conversation chatbot able to send messages, there were issues

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals

import logging

from rasa_core.agent import Agent
from rasa_core.channels.console import ConsoleInputChannel
from rasa_core.interpreter import RegexInterpreter
from rasa_core.policies.keras_policy import KerasPolicy
from rasa_core.policies.memoization import MemoizationPolicy
from rasa_core.interpreter import RasaNLUInterpreter

logger = logging.getLogger(__name__)

def run_weather_online(input_channel, interpreter,
                          domain_file="weather_domain.yml",
                          training_data_file='data/stories.md'):
    agent = Agent(domain_file,
                  policies=[MemoizationPolicy(), KerasPolicy()],
                  interpreter=interpreter)

    agent.train_online(training_data_file,
                       input_channel=input_channel,
                       max_history=2,
                       batch_size=50,
                       epochs=200,
                       max_training_samples=300)

    return agent

if __name__ == '__main__':
    logging.basicConfig(level="INFO")
    nlu_interpreter = RasaNLUInterpreter('./models/nlu/default/weathernlu')
    run_weather_online(ConsoleInputChannel(), nlu_interpreter)

Yet the following error appeared :

(MoodbotEnv) mike@mike-thinks:~/Programing/Rasa_tutorial/moodbot4$ python train_online.py 
INFO:rasa_nlu.components:Added 'nlp_spacy' to component cache. Key 'nlp_spacy-en'.
Traceback (most recent call last):
  File "/home/mike/Programing/Rasa_tutorial/moodbot4/MoodbotEnv/lib/python3.5/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 508, in _unpickle
    obj = unpickler.load()
  File "/usr/lib/python3.5/pickle.py", line 1039, in load
    dispatch[key[0]](self)
  File "/usr/lib/python3.5/pickle.py", line 1177, in load_binstring
    self.append(self._decode_string(data))
  File "/usr/lib/python3.5/pickle.py", line 1159, in _decode_string
    return value.decode(self.encoding, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x89 in position 36: ordinal not in range(128)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train_online.py", line 37, in <module>
    nlu_interpreter = RasaNLUInterpreter('./models/nlu/default/weathernlu')
  File "/home/mike/Programing/Rasa_tutorial/moodbot4/MoodbotEnv/lib/python3.5/site-packages/rasa_core/interpreter.py", line 221, in __init__
    self._load_interpreter()
  File "/home/mike/Programing/Rasa_tutorial/moodbot4/MoodbotEnv/lib/python3.5/site-packages/rasa_core/interpreter.py", line 237, in _load_interpreter
    self.interpreter = Interpreter.load(self.model_directory)
  File "/home/mike/Programing/Rasa_tutorial/moodbot4/MoodbotEnv/lib/python3.5/site-packages/rasa_nlu/model.py", line 276, in load
    skip_validation)
  File "/home/mike/Programing/Rasa_tutorial/moodbot4/MoodbotEnv/lib/python3.5/site-packages/rasa_nlu/model.py", line 303, in create
    model_metadata, **context)
  File "/home/mike/Programing/Rasa_tutorial/moodbot4/MoodbotEnv/lib/python3.5/site-packages/rasa_nlu/components.py", line 398, in load_component
    cached_component, **context)
  File "/home/mike/Programing/Rasa_tutorial/moodbot4/MoodbotEnv/lib/python3.5/site-packages/rasa_nlu/registry.py", line 131, in load_component_by_name
    return component_clz.load(model_dir, metadata, cached_component, **kwargs)
  File "/home/mike/Programing/Rasa_tutorial/moodbot4/MoodbotEnv/lib/python3.5/site-packages/rasa_nlu/extractors/crf_entity_extractor.py", line 313, in load
    ent_tagger = joblib.load(model_file)
  File "/home/mike/Programing/Rasa_tutorial/moodbot4/MoodbotEnv/lib/python3.5/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 578, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "/home/mike/Programing/Rasa_tutorial/moodbot4/MoodbotEnv/lib/python3.5/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 523, in _unpickle
    raise new_exc
ValueError: You may be trying to read with python 3 a joblib pickle generated with python 2. This feature is not supported by joblib.

I know it means that compatibility across python versions is not fully supported but I don't understand why and what can I do to cope with it.

Just to inform myself I tried to launch the code using Python 2 explicitly : and it gave me :

(MoodbotEnv) mike@mike-thinks:~/Programing/Rasa_tutorial/moodbot4$ python2 train_online.py 
Traceback (most recent call last):
  File "train_online.py", line 37, in <module>
    nlu_interpreter = RasaNLUInterpreter('./models/nlu/default/weathernlu')
  File "/home/mike/.local/lib/python2.7/site-packages/rasa_core/interpreter.py", line 219, in __init__
    self._load_interpreter()
  File "/home/mike/.local/lib/python2.7/site-packages/rasa_core/interpreter.py", line 234, in _load_interpreter
    from rasa_nlu.config import RasaNLUConfig
ImportError: cannot import name RasaNLUConfig

Which is an issue they already dealt with here but which is unexpected as far as my rasa-core version is one of the latest.

I think the problem is either RasaNLUInterpreter module has changed for another here or it is impossible for RasaNLUInterpreter to read the .pkl extension files :

(envMoodbot2) mike@mike-thinks:~/Programing/Rasa_tutorial/moodbot4/models/nlu/default/weathernlu$ ls
crf_model.pkl  intent_classifier_sklearn.pkl  metadata.json  training_data.json

Content of domain file :

action_factory: null
action_names:
- utter_greet
- utter_goodbye
- utter_ask_location
- action_weather
actions:
- utter_greet
- utter_goodbye
- utter_ask_location
- actions.ActionWeather
config:
  store_entities_as_slots: true
entities:
- location
intents:
- greet
- goodbye
- inform
slots:
  location:
    initial_value: null
    type: rasa_core.slots.TextSlot
templates:
  utter_ask_location:
  - text: In what location?
  utter_goodbye:
  - text: Talk to you later.
  - text: Bye bye :(
  utter_greet:
  - text: Hello! How can I help?
topics: []

[antoinecomp]

I crafted a solution : as far as the problem was with RasaNLUInterpreter unable to read the .pkl files I just created the /models/nlu/default/weathernlu/test folder and copied all .json files into it and modified the main in train_online.py file with :

nlu_interpreter = RasaNLUInterpreter('./models/nlu/default/weathernlu/test')

And it worked as a charm.

[JustinaPetr]

Hey. Glad you found a solution. I am pretty sure the issue was due to the different version of Rasa NLU which I used for the video. Thanks for pointing that out, I will make sure to update the tutorial so that new readers wouldn't get confused :)

[antoinecomp]

Yes, I don't know if it is a good solution though, It was a bit of mac gayver :/

[antoinecomp]

I have seen that you still calls to the .pkl files in nlu_model.py whereas it creates an issue with interpreter=Interpreter.load('./models/nlu/default/weathernlu'):

Traceback (most recent call last):
  File "/home/mike/Programing/Rasa/Moodbot/MoodEnv/lib/python3.5/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 508, in _unpickle
    obj = unpickler.load()
  File "/usr/lib/python3.5/pickle.py", line 1039, in load
    dispatch[key[0]](self)
  File "/usr/lib/python3.5/pickle.py", line 1177, in load_binstring
    self.append(self._decode_string(data))
  File "/usr/lib/python3.5/pickle.py", line 1159, in _decode_string
    return value.decode(self.encoding, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x9f in position 36: ordinal not in range(128)
...
  File "/home/mike/Programing/Rasa/Moodbot/MoodEnv/lib/python3.5/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 523, in _unpickle
    raise new_exc
ValueError: You may be trying to read with python 3 a joblib pickle generated with python 2. This feature is not supported by joblib.

Indeed, there is some .pkl files in models/nlu/default/weathernlu :

(MoodEnv) mike@mike-thinks:~/Programing/Rasa/Moodbot/models/nlu/default/weathernlu$ ls
crf_model.pkl  intent_classifier_sklearn.pkl  metadata.json  training_data.json

I thought it was a good solution to get rid of these files but as far as I am still unable to get Rasa NLU understand my intents I am now unsure about it ...

[antoinecomp]

Actually I was wrong. My chatbot wasn't able to recognize intent anymore and this was due to the odd reason that I have to reinstalled the backend :

pip install rasa_nlu[spacy]
python -m spacy download en_core_web_md
python -m spacy link en_core_web_md en

The weird thing is that I think I will have to reinstall them every time I launch again the project ...