davidkartuzinski commented 6 years ago

RASA NLU Version: rasa-nlu (0.12.3) Operating System: Kubuntu 18.04, 64 bit Below is my PIP list:

absl-py (0.3.0) apixu-client (0.0.1) APScheduler (3.3.1) astor (0.7.1) atomicwrites (1.1.5) attrs (18.1.0) Automat (0.7.0) bleach (1.5.0) boto3 (1.4.4) botocore (1.5.95) certifi (2018.4.16) chardet (3.0.4) click (6.7) cloudpickle (0.5.3) coloredlogs (7.3) colorhash (1.0.2) ConfigArgParse (0.12.0) constantly (15.1.0) cycler (0.10.0) cymem (1.31.2) Cython (0.27.3) cytoolz (0.8.2) decorator (4.3.0) dill (0.2.8.2) docopt (0.6.2) docutils (0.14) en-core-web-sm (2.0.0) fakeredis (0.8.2) fbmessenger (4.3.1) Flask (0.12) Flask-Cors (3.0.6) ftfy (4.4.3) future (0.16.0) gast (0.2.0) gevent (1.2.1) graphviz (0.7.1) greenlet (0.4.14) grpcio (1.14.1) h5py (2.7.0) html5lib (1.0.1) humanfriendly (4.16.1) hyperlink (18.0.0) idna (2.7) incremental (17.5.0) itsdangerous (0.24) Jinja2 (2.10) jmespath (0.9.3) jsonpickle (0.9.4) jsonschema (2.6.0) Keras (2.1.6) klein (17.2.0) Markdown (2.6.11) MarkupSafe (1.0) matplotlib (1.5.3) mattermostwrapper (2.1) mock (2.0.0) more-itertools (4.3.0) msgpack (0.5.6) msgpack-numpy (0.4.3.1) murmurhash (0.26.4) networkx (1.11) numpy (1.15.0) packaging (17.1) pandoc (1.0.0b2) pathlib (1.0.1) pbr (4.2.0) pip (9.0.1) pkg-resources (0.0.0) plac (0.9.6) pluggy (0.7.1) ply (3.11) preshed (1.0.1) protobuf (3.6.0) py (1.5.4) PyHamcrest (1.9.0) PyJWT (1.6.4) pykwalify (1.6.0) pypandoc (1.4) pyparsing (2.2.0) PySocks (1.6.8) pytest (3.7.1) python-crfsuite (0.9.6) python-dateutil (2.7.3) python-telegram-bot (10.1.0) pytz (2018.5) PyYAML (3.13) rasa-core (0.9.4) rasa-nlu (0.12.3) redis (2.10.5) regex (2017.4.5) requests (2.14.2) ruamel.yaml (0.15.34) s3transfer (0.1.13) scikit-learn (0.19.2) scipy (1.1.0) setuptools (39.0.1) simplejson (3.11.1) six (1.11.0) sklearn-crfsuite (0.3.6) slackclient (1.2.1) spacy (1.8.2) tabulate (0.8.2) tensorboard (1.7.0) tensorflow (1.7.0) termcolor (1.1.0) thinc (6.5.2) toolz (0.9.0) tqdm (4.24.0) twilio (6.16.1) Twisted (18.7.0) typing (3.5.3.0) tzlocal (1.5.1) ujson (1.35) urllib3 (1.23) wcwidth (0.1.7) webencodings (0.5.1) websocket-client (0.48.0) Werkzeug (0.14.1) wheel (0.31.1) wrapt (1.10.11) zope.interface (4.5.0)

Issue: Installing Rasa Core and Rasa NLU is a nightmare.
I am trying to get a running Rasa NLU in a Python Virtual Environment. Below are the steps, I had to take to get to this error: "Exception: Not all required packages are installed. To use this pipeline, you need to install the missing dependencies. Please install spacy"

First, as a sudo user, nothing works with spacy or rasa, on Mint, or Ubuntu 16.04, or this Kubuntu. On my two computers or the computers of two others. All Linux. spaCy seems to be the real problem and most of these added steps, are due to spaCy simply not installing globally or/and my .env file.

I took these steps to get to the above error:

I had nothing but errors unless I was root, and installing spaCy was a nightmare. These are the basic steps. (I also had to install wheel using pip and pip3 - as I was getting failures).

I also had to follow these instructions: https://stackoverflow.com/a/48680354

install dev libs

sudo apt-get install python-dev -y && \ sudo apt-get install python3-dev -y && \ sudo apt-get install libevent-dev -y && \ and apt-get install python-dev -y && \ apt-get install python3-dev -y && \ apt-get install libevent-dev -y && \
[AS ROOT]

install new gcc

sudo apt-get update && \ sudo apt-get install build-essential software-properties-common -y && \ sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y && \ sudo apt-get update && \ sudo apt-get install gcc-snapshot -y && \ sudo apt-get update && \ sudo apt-get install gcc-6 g++-6 -y && \ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-6 && \ sudo apt-get install gcc-4.8 g++-4.8 -y && \ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.8;

Then I followed these instructions.

Check version of Python $ python3 -V

Verify and/or install build-essential python-dev git $ sudo apt-get install build-essential python-dev git

Then Create a Virtual Environment and install spaCy per https://spacy.io/usage/#source-ubuntu for a Virtual Environment :

cd into project folder (e.g. RasaBotRonan) $python3 -m venv .env $source .env/bin/activate

Install spaCy (as part of virtual Environment) (https://spacy.io/usage/#source-ubuntu for Linux instructions *other OS instructions available) $ git clone https://github.com/explosion/spaCy $cd spaCy $export PYTHONPATH=pwd $pip install -r requirements.txt $python setup.py build_ext --inplace $python -m spacy download en

Next, I installed RASA per these instructions:

$ cd .. <--- now in project folder root and spaCy is a folder inside this folder $ git clone https://github.com/dtaivpp/python-chatbot.git $ cd python-chatbot $ pip3 install -r requirements.txt

Train the initial run:

$ python3 train_rasa_core.py

I was getting all kinds of errors which precipitated the above sequence of steps. Finally the train_rase_core.py ran, and can run twice.

I tried $ python3 train_rasa_terminal.py and got the following error:

Exception: Not all required packages are installed. To use this pipeline, you need to install the missing dependencies. Please install spacy

I tried fixing it with this:

$ pip install -U scikit-learn scipy sklearn-crfsuite

But that does not fix it. I am running $python3 train_rasa_terminal.py from a directory "next to" spaCy

Below is my project directory structure: RasaBotRonan --.env --spaCy --python-chatbot

I can run python3 train_rasa_core from ../pythonchatbot/

Thank you. I really want to get Rasa up and running.

akelad commented 6 years ago

Thanks for creating this issue, @twhughes will get back to you about it soon.

twhughes commented 6 years ago

Hi @davidkartuzinski. It's a bit hard to debug from here but maybe I can ask a few questions first

what is the output of: which python3 does it point to a directory in your project directory? I just want to check that you've activated the virtual environment correctly.
what do you get when you try python3 -c "import spacy" with the virtual environment activated? If this throws an error then it's an issue with the spacy installation.
You mentioned you tried to fix it with pip install -U scikit-learn scipy sklearn-crfsuite Did you not instead try? pip install -U spacy

davidkartuzinski commented 6 years ago

Thank you for your very fast answer.

The output of which python3 is /home/david/Projects/RasaBotRonan/.env/bin/python3 I used that command within my .env as the venv was activated.
I don't remember installing spaCy that way exactly. But when I run the command now, from within the venv, I get:

/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility return f(*args, **kwds) /usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility return f(*args, **kwds)

Yes, I did try to pip install -U spacy globally and separately within various venvs, and multiple times, on multiple computers and after wiping and reinstalling ubuntu twice on one of my computers.. Also, my coding partner did the same. That command doesn't seem to work.

What i wrote above, was the only steps, on a clean install that worked for me.

==================== I tried to rerun: python3 train_rasa_core.py to verify that it was still working. And at count Epocj 400/400, I get the following error (which I didn't get before):

Epoch 400/400
240/240 [==============================] - 0s 131us/step - loss: 0.0464 - acc: 0.9792 - val_loss: 0.0743 - val_acc: 0.9667
INFO:rasa_core.policies.keras_policy:Done fitting keras policy model
Time to train: 0:00:16.246945
INFO:rasa_core.agent:Model directory ./models/dialogue exists and contains old model files. All files will be overwritten.
Traceback (most recent call last):
  File "train_rasa_core.py", line 54, in <module>
    agent.persist(model_path)
  File "/home/david/Projects/RasaBotRonan/.env/lib/python3.6/site-packages/rasa_core/agent.py", line 330, in persist
    self._clear_model_directory(model_path)
  File "/home/david/Projects/RasaBotRonan/.env/lib/python3.6/site-packages/rasa_core/agent.py", line 320, in _clear_model_directory
    shutil.rmtree(model_path)
  File "/usr/lib/python3.6/shutil.py", line 480, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.6/shutil.py", line 438, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 436, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'domain.yml'

This is the fourth time on this install that I ran it with the default training data. I don't know if this is a normal response.

I am willing to completely start again, but have found no tutorials, instructions, or step by step that actually is up to date or assumes a newer set-up. And a new programmer. The only reason I started learning Python 6 months ago (thanks @JoeyFaulkner and @mkennedy) was so that I could use Rasa. Luis and friends, didn't fit my requirements.

Again, thank you. Like I said, I can start all over again, note every step and take screencaps, but there does seem to be something wrong.

DK

davidkartuzinski commented 6 years ago

Per this video, https://www.youtube.com/watch?v=NBTeNcPhHJs spaCy has an issue wih the english language. Around minute 1:15 to 1:45. FYI.

twhughes commented 6 years ago

Ok

Based on the warning from step 2 it seems like that is not a spacy error but rather a numpy warning that can be safely ignored. Also, the command I pasted there is just to check whether spacy is installed and can be imported.

The error you are getting now has to do with the python-chatbot repository that you are using and not rasa core itself. Because we did not make this repository I do not know how to fix it. Perhaps you should create an issue on that repository to ask him about the error. Just paste that error message in and he should be able to fix it. Or try git pull from that directory to update if there were any recent changes that may have fixed any bugs. Did you try running the commands in the repo of that repository for installing scipy? (under Installation Linux (tested on Ubuntu 18.04.1 LTS Minimal install))

just to get started and make sure everything is working correctly, I'd highly recommend that you instead follow the tutorial here https://rasa.com/docs/core/quickstart/#id1. This is just a simple demonstration of rasa_core that we've made for beginners. It will walk you through the few bare minimum components that you need. If that is working then you can substitute your own training data and it should work. The docs are also a good resource for installation.

Let me know if that makes sense or you have any more issues.

Tyler

twhughes commented 6 years ago

Also, looking at that repo, it seems you need to train the NLU portion of rasa via python train_nlu.py before you can run train_rasa_core.py

davidkartuzinski commented 6 years ago

Thank you for your answers.

I understand about he Numpy error. Makes sense.

Regarding the Repo, the author of that repo has been trying to help us for about a week to get it working which I why I came here. I will make another effort to reinstall from that repo but I am certain that I ran train_nlu.py first which is where I got the error.

As to the start docs. I started there months ago and those instructions have never worked for myself or anyone else I know. In fact last night, I tried four more times. In two more venv and I also installed anaconda and tried from there as well.

The “trial” on the QuickStart where you run in browser doesn’t work and never has for anyone I know. The entire reason I am looking for other tutorials is because the docs don’t work as written. At least for me and about five others over the months who have tried them.

I will post the new errors shortly.

Not being able to run the tutorials or the start docs are one of the main reasons I switched from Windows to Linux. But that didn’t help at all.

I do appreciate it and I will post the errors next.

Thanks.

akelad commented 6 years ago

@davidkartuzinski PermissionError: [Errno 13] Permission denied: 'domain.yml' this looks more like an OS error -- what directory were you running this in? If it's the root directory you probably don't have permission to write there. try running it in your home directory instead

twhughes commented 6 years ago

Also, did you change any of the filenames in the python-chatbot repository? It looks like it's trying to clear a model directory but it's been given a domain file.

davidkartuzinski commented 6 years ago

Okay. It seems we cannot use a venv to make this happen: Following the instructions from Rasa Documentation: https://rasa.com/docs/core/installation/

mkdir RasaNoVenv
cd RasaNoVenv
pip3 install rasa_core Apart from getting " Cache entry deserialization failed, entry ignored", after each entry, it does seem to install correctly afterwards.
pip3 install -U rasa_core
pip3 install rasa_nlu[tensorflow] Now at Quickstart https://rasa.com/docs/core/quickstart/#quickstart
[In project top level directory] sudo nano stories.md, copy/paste: `## happy path
- greet
- utter_greet
- mood_great
- utter_happy

sad path 1

greet
- utter_greet
mood_unhappy
- utter_cheer_up
- utter_did_that_help
mood_affirm
- utter_happy

sad path 2

greet
- utter_greet
mood_unhappy
- utter_cheer_up
- utter_did_that_help
mood_deny
- utter_goodbye

say goodbye

goodbye
- utter_goodbye`
  1. In project directory: sudo nano domain.yml, copy paste: `intents:
- greet
- goodbye
- mood_affirm
- mood_deny
- mood_great
- mood_unhappy

actions:

utter_greet
utter_cheer_up
utter_did_that_help
utter_happy
utter_goodbye

templates: utter_greet:

text: "Hey! How are you?"

utter_cheer_up:
text: "Here is something to cheer you up:" image: "https://i.imgur.com/nGF1K8f.jpg"

utter_did_that_help:
text: "Did that help you?"

utter_happy:
text: "Great carry on!"

utter_goodbye:
text: "Bye"`
1. Now to train the dialogue model: $python3 -m rasa_core.train -d domain.yml -s stories.md -o models/dialogue [Seems to work]
2. Final test, talking to the bot [from command line]: $ python3 -m rasa_core.run -d models/dialogue It WORKS!!! NOT IN VENV.
  
  Now, I try the same thing in venv.

mkdir RasaInVenv
cd RasaInVenv
python3 -m venv .env
source .env/bin/activate [from Project Root] [now activated (.env)$}
pip3 install rasa_core [No Errors]
pip3 install -U rasa_core [No Errors]
pip3 install rasa_nlu[tensorflow] [no Errors] Now at Quickstart https://rasa.com/docs/core/quickstart/#quickstart
[In project top level directory] sudo nano stories.md, copy/paste: `## happy path
- greet
- utter_greet
- mood_great
- utter_happy

sad path 1

greet
- utter_greet
mood_unhappy
- utter_cheer_up
- utter_did_that_help
mood_affirm
- utter_happy

sad path 2

greet
- utter_greet
mood_unhappy
- utter_cheer_up
- utter_did_that_help
mood_deny
- utter_goodbye

say goodbye

goodbye
- utter_goodbye`
  1. In project directory: sudo nano domain.yml, copy paste: `intents:
- greet
- goodbye
- mood_affirm
- mood_deny
- mood_great
- mood_unhappy

actions:

utter_greet
utter_cheer_up
utter_did_that_help
utter_happy
utter_goodbye

templates: utter_greet:

text: "Hey! How are you?"

utter_cheer_up:
text: "Here is something to cheer you up:" image: "https://i.imgur.com/nGF1K8f.jpg"

utter_did_that_help:
text: "Did that help you?"

utter_happy:
text: "Great carry on!"

utter_goodbye:
text: "Bye"`
1. Now to train the dialogue model: $python3 -m rasa_core.train -d domain.yml -s stories.md -o models/dialogue [Still seems to work]
2. Final test IN A VENV, talking to the bot [from command line]: $ python3 -m rasa_core.run -d models/dialogue It WORKS!!! *Also works now.

I AM TOTALLY CONFUSED. I have tried this a million times to no satisfaction.

My only guess is that I copied the domain.yml and stories.md incorrectly. I may have been copying the entire parts instead of JUST what is in between the """.

But your docs work now for me, on my Ubuntu 18.04 machine. Both in and out of a VENV.

============================================================================

Thank you. Thank you

davidkartuzinski commented 6 years ago

I am now adding the nlu part. Keeping the venv from the previous comment above:

$ sudo nano nlu.md
[from within the top level project directory] `## intent:greet
- hey
- hello
- hi
- good morning
- good evening
- hey there

intent:goodbye

bye
goodbye
see you around
see you later

intent:mood_affirm

yes
indeed
of course
that sounds good
correct

intent:mood_deny

no
never
I don't think so
don't like that
no way
not really

intent:mood_great

perfect
very good
great
amazing
wonderful
I am feeling very good
I am great
I'm good

intent:mood_unhappy

sad
very sad
unhappy
bad
very bad
awful
terrible
not very good
extremly sad
so sad`
1. sudo nano nlu_config.yml [again from top level project directory] language: en pipeline: tensorflow_embedding
2. [I already have Rasa NLU installed] so $python3 -m rasa_nlu.train -c nlu_config.yml --data nlu.md -o models --fixed_model_name nlu --project current --verbose [ That worked. It finished training.]
3. verify directory is created: $ ls models/current/nlu/ [All okay - its there]
4. Final test with NLU $python3 -m rasa_core.run -d models/dialogue -u models/current/nlu
  
  It works!!!!! Still thoroughly confused. I don't know why it works now. But your docs work in venv and I am assuming outside of a virtual environment as well.

Again thank you for your patience and help. -DK

akelad commented 6 years ago

Cool, glad it worked!

twhughes commented 6 years ago

Great!

RasaHQ / rasa

spaCy already installed: "Exception: Not all required packages are installed. To use this pipeline, you need to install the missing dependencies. Please install spacy" #1309

RASA NLU Version: rasa-nlu (0.12.3) Operating System: Kubuntu 18.04, 64 bit Below is my PIP list:

install dev libs

install new gcc

Again, thank you. Like I said, I can start all over again, note every step and take screencaps, but there does seem to be something wrong.

sad path 1

sad path 2

say goodbye

Final test, talking to the bot [from command line]: $ python3 -m rasa_core.run -d models/dialogue It WORKS!!! NOT IN VENV.

sad path 1

sad path 2

say goodbye

intent:goodbye

intent:mood_affirm

intent:mood_deny

intent:mood_great

intent:mood_unhappy

Final test with NLU $python3 -m rasa_core.run -d models/dialogue -u models/current/nlu