consuming time for chat answers

[ttrd06]

hello at all,

i followed examples and tutorials for testing chatterbot but i have a 'bug' or there is something i don't understand: i used the terminal example and i have to wait until 10-15 seconds before the answers come from chatterbot. the conversation is not 'normal'... my config is: win10 64 - core I7 - 16gb of ram -- 1To SSD.

is there here a complete project to test it ?

[vkosuri]

I tired on my machine it is quite fast and working fine as expected.

Some times it might delay using JsonStorageAdapter because of if it is very huge size around 1MB.

Could you try un-comment this line . It will verbose more information on terminal how your request and response are selecting using different logic adapters.

For example

Type something to begin...
hello
INFO:chatterbot.adapters:Recieved input statement: hello
INFO:chatterbot.adapters:"hello" is not a known statement
INFO:chatterbot.adapters:Not processing the statement using <class 'chatterbot.logic.mathematical_evaluation.MathematicalEvaluation'>
INFO:chatterbot.adapters:<class 'chatterbot.logic.time_adapter.TimeLogicAdapter'> selected "The current time is 09:20 PM" as a response with a confidence of 0
INFO:chatterbot.adapters:Not processing the statement using <class 'chatterbot.logic.best_match.BestMatch'>
INFO:chatterbot.adapters:<class 'chatterbot.logic.no_knowledge_adapter.NoKnowledgeAdapter'> selected "hello" as a response with a confidence of 1
hello
how are you
INFO:chatterbot.adapters:Recieved input statement: how are you
INFO:chatterbot.adapters:"how are you" is not a known statement
INFO:chatterbot.adapters:Not processing the statement using <class 'chatterbot.logic.mathematical_evaluation.MathematicalEvaluation'>
INFO:chatterbot.adapters:<class 'chatterbot.logic.time_adapter.TimeLogicAdapter'> selected "The current time is 09:20 PM" as a response with a confidence of 1
INFO:chatterbot.adapters:No statements have known responses. Choosing a random response to return.
INFO:chatterbot.adapters:Using "how are you" as a close match to "hello"
INFO:chatterbot.adapters:No response to "hello" found. Selecting a random response.
INFO:chatterbot.adapters:<class 'chatterbot.logic.best_match.BestMatch'> selected "hello" as a response with a confidence of 0
INFO:chatterbot.adapters:<class 'chatterbot.logic.no_knowledge_adapter.NoKnowledgeAdapter'> selected "how are you" as a response with a confidence of 0
INFO:chatterbot.chatterbot:Adding "how are you" as a response to "hello"
The current time is 09:20 PM

[ttrd06]

thank's for reply,

i un-commented the line you asked then you have below:

Type something to begin... hello INFO:chatterbot.adapters:Recieved input statement: hello INFO:chatterbot.adapters:"hello" is a known statement INFO:chatterbot.adapters:Not processing the statement using MathematicalEvaluation INFO:chatterbot.adapters:TimeLogicAdapter selected "The current time is 06:10 PM" as a response with a confidence of 0 INFO:chatterbot.adapters:Using "hello" as a close match to "hello" INFO:chatterbot.adapters:Selecting response from 1 optimal responses. INFO:chatterbot.response_selection:Selecting first response from list of 1 options. INFO:chatterbot.adapters:Response selected. Using "Greetings!" INFO:chatterbot.adapters:BestMatch selected "Greetings!" as a response with a confidence of 1.0 INFO:chatterbot.adapters:NoKnowledgeAdapter selected "hello" as a response with a confidence of 0 Greetings!

i have to wait 30 seconds to have this answer... not glad about that as you can expect the size of database after training is 151 k not huge at all ! how many seconds you are waiting for ?

[vkosuri]

I don't see any issue with my machine.

helo
2017-04-02,22:22:59.725 INFO {input_adapter}[process_input_statement] Recieved input statement: helo
2017-04-02,22:22:59.815 INFO {input_adapter}[process_input_statement] "helo" is not a known statement
2017-04-02,22:22:59.820 INFO {multi_adapter}[process] Not processing the statement using MathematicalEvaluation
2017-04-02,22:22:59.822 INFO {multi_adapter}[process] TimeLogicAdapter selected "The current time is 10:22 PM" as a response with a confidence of 0
2017-04-02,22:22:59.825 INFO {best_match}[process] Using "helo" as a close match to "hello"
2017-04-02,22:22:59.826 INFO {best_match}[process] Selecting response from 1 optimal responses.
2017-04-02,22:22:59.830 INFO {response_selection}[get_first_response] Selecting first response from list of 1 options.
2017-04-02,22:22:59.830 INFO {best_match}[process] Response selected. Using "how are you"
2017-04-02,22:22:59.831 INFO {multi_adapter}[process] BestMatch selected "how are you" as a response with a confidence of 0.89
2017-04-02,22:22:59.831 INFO {multi_adapter}[process] NoKnowledgeAdapter selected "helo" as a response with a confidence of 0
how are you

[vkosuri]

You could add logger formatter after line number 8 to know exact time you have been waited

logging.basicConfig(format='%(asctime)s.%(msecs)03d %(levelname)s {%(module)s} [%(funcName)s] %(message)s', datefmt='%Y-%m-%d,%H:%M:%S', level=logging.INFO)

[ttrd06]

you'll see now with the new infos provided:

Type something to begin... hello 2017-04-02,19:23:44.506 INFO {input_adapter} [process_input_statement] Recieved input statement: hello 2017-04-02,19:23:44.568 INFO {input_adapter} [process_input_statement] "hello" is a known statement 2017-04-02,19:23:44.568 INFO {multi_adapter} [process] Not processing the statement using MathematicalEvaluation 2017-04-02,19:23:44.568 INFO {multi_adapter} [process] TimeLogicAdapter selected "The current time is 07:23 PM" as a response with a confidence of 0 2017-04-02,19:24:03.009 INFO {best_match} [process] Using "hello" as a close match to "Hello" 2017-04-02,19:24:21.181 INFO {best_match} [process] Selecting response from 2 optimal responses. 2017-04-02,19:24:21.181 INFO {response_selection} [get_first_response] Selecting first response from list of 2 options. 2017-04-02,19:24:21.181 INFO {best_match} [process] Response selected. Using "Greetings!" 2017-04-02,19:24:21.181 INFO {multi_adapter} [process] BestMatch selected "Greetings!" as a response with a confidence of 1.0 2017-04-02,19:24:21.197 INFO {multi_adapter} [process] NoKnowledgeAdapter selected "hello" as a response with a confidence of 0 Greetings!

Any new ideas with that ? Or do you have an example to provide more infos for debug this ?

[ttrd06]

and in the same time, i tried the twitter example and i had an another bug: File "C:\Python27\lib\site-packages\chatterbot\trainers.py", line 131, in init from twitter import Api as TwitterApi ImportError: cannot import name Api

did you experienced the same bug ?

[ttrd06]

last news, i uninstalled my python 2.7 (but working with others projects) and installed python 3 (clean install) i have both the sames issues ... if someone can explain me clearly what happens ?

[ttrd06]

another news: tested with Linux too and i have same issues with the twitter error: Python27\lib\site-packages\chatterbot\trainers.py", line 131, in init from twitter import Api as TwitterApi ImportError: cannot import name Api

So windows is not faulty because Linux has the same issue ... Anyone have this bug ?

[vkosuri]

For slowness issue try this example

# -*- coding: utf-8 -*-
from chatterbot import ChatBot
import logging

# Uncomment the following line to enable verbose logging
logging.basicConfig(format='%(asctime)s.%(msecs)03d %(levelname)s {%(module)s} [%(funcName)s] %(message)s', datefmt='%Y-%m-%d,%H:%M:%S', level=logging.INFO)

# Create a new instance of a ChatBot
bot = ChatBot("Terminal",
    storage_adapter="chatterbot.storage.JsonFileStorageAdapter",
    logic_adapters=[
        "chatterbot.logic.BestMatch"
    ],
    input_adapter="chatterbot.input.TerminalAdapter",
    output_adapter="chatterbot.output.TerminalAdapter",
    database="../database.db"
)

print("Type something to begin...")

# The following loop will execute each time the user enters input
while True:
    try:
        # We pass None to this method because the parameter
        # is not used by the TerminalAdapter
        bot_input = bot.get_response(None)

    # Press ctrl-c or ctrl-d on the keyboard to exit
    except (KeyboardInterrupt, EOFError, SystemExit):
        break

For twitter issue try to install twitter

pip install --upgrade twitter

[ttrd06]

good morning,

i followed what you said @vkosuri and no changes appears as you can check below:

Type something to begin...
hello
2017-04-03,08:40:24.539 INFO {input_adapter} [process_input_statement] Recieved input statement: hello
2017-04-03,08:40:24.618 INFO {input_adapter} [process_input_statement] "hello" is a known statement
2017-04-03,08:40:42.808 INFO {best_match} [process] Using "hello" as a close match to "hello"
2017-04-03,08:41:00.575 INFO {best_match} [process] Selecting response from 1 optimal responses.
2017-04-03,08:41:00.575 INFO {response_selection} [get_first_response] Selecting first response from list of 1 options.
2017-04-03,08:41:00.575 INFO {best_match} [process] Response selected. Using "Greetings!"
2017-04-03,08:41:00.575 INFO {multi_adapter} [process] BestMatch selected "Greetings!" as a response with a confidence of 1.0
2017-04-03,08:41:00.592 INFO {multi_adapter} [process] NoKnowledgeAdapter selected "hello" as a response with a confidence of 0
Greetings!

36 sec to wait from the beginning ... twitter, it's the same but what about pip install --upgrade twitter in the requirements file from chatterbot, you have: chatterbot-corpus>=0.0.1,<1.0.0 jsondatabase>=0.1.7,<1.0.0 nltk>=3.2.0,<4.0.0 pymongo>=3.3.0,<4.0.0 python-twitter>=3.0.0,<4.0.0

the twitter package is not mentioned, but i installed as you told me but as explained before, the bug is always here and tested both linux and windows and under python 2.7.13 and 3.6. now i don't think my config is faulty but bugs in chatterbot. Am i wrong ?

My config: win10 64 & linux 64 - 16gb ram - 1To SSD - intel I7

[vkosuri]

@ttrd06 As per my knowledge, I think there is no specified time limit to get response for all requests, it is dependent on different factors for example memory, Operating system, etc.

Did you problem was resolved after installing twitter.

@gunthercox Master, could you please comment on this topic?

[ttrd06]

@vkosuri twiter pb has been resolved since 2 minutes and others bugs inside with: INFO:chatterbot.trainers:Requesting 50 random tweets containing the word Google INFO:chatterbot.trainers:Adding 1 tweets with responses Traceback (most recent call last): File "C:/AI/Python/chat/twitter-test.py", line 40, in <module> chatbot.train().encode("utf-8") File "C:\Python27\lib\site-packages\chatterbot\trainers.py", line 213, in train self.storage.update(statement) File "C:\Python27\lib\site-packages\chatterbot\storage\jsonfile.py", line 161, in update self.database.data(key=statement.text, value=data) File "C:\Python27\lib\site-packages\jsondb\db.py", line 102, in data self._set_content(key, value) File "C:\Python27\lib\site-packages\jsondb\db.py", line 63, in _set_content obj = self._get_content() File "C:\Python27\lib\site-packages\jsondb\db.py", line 52, in _get_content obj = self.read_data(self.path) File "C:\Python27\lib\site-packages\jsondb\file_writer.py", line 15, in read_data obj = decode(content) File "C:\Python27\lib\site-packages\jsondb\compat.py", line 32, in decode return json_decode(value, encoding='utf-8', object_hook=json_util.object_hook) File "C:\Python27\lib\json\__init__.py", line 352, in loads return cls(encoding=encoding, **kw).decode(s) File "C:\Python27\lib\json\decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Python27\lib\json\decoder.py", line 380, in raw_decode obj, end = self.scan_once(s, idx) UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 115: invalid start byte so i think this example is completly faulty and not ready to use. i decided to abandon it you told me about factors, but did you seen my specs ? 16 gb of ram, SSD, intel I7 ... do you think it's a bad computer ? i tried CNTK, tensorflow with no issues ... you know, if you install something it must work or not. if it don't work: recycle and try another one

if @gunthercox has a solution about that, i will do everything he wants for

[ttrd06]

hi again,

nobody have explaination about this too big consuming time ?

[gunthercox]

@ttrd06 I will look into your issue as soon as I can.

:coffee: :coffee: :coffee:

It looks like you have encountered several possible issues. Scanning over all your previous comments it also looks like you possibly have the wrong twitter package installed (it should be python-twitter).

pip uninstall twitter
pip install python-twitter

[ttrd06]

thank's for reply @gunthercox,

at this time, twitter is important but secondary. my goal is use chatterbox with a raisonable time to answer. As i said before, i followed your tuto:

pip install chatterbot

and

`from chatterbot import ChatBot

chatbot = ChatBot( 'Ron Obvious', trainer='chatterbot.trainers.ChatterBotCorpusTrainer' )

Train based on the english corpus

chatbot.train("chatterbot.corpus.english")

Get a response to an input statement

chatbot.get_response("Hello, how are you today?")`

and finally:

`# -- coding: utf-8 -- from chatterbot import ChatBot import logging

bot = ChatBot("Terminal")

bot.storage.drop()

exit()

Uncomment the following line to enable verbose logging

logging.basicConfig(format='%(asctime)s.%(msecs)03d %(levelname)s {%(module)s} [%(funcName)s] %(message)s', datefmt='%Y-%m-%d,%H:%M:%S', level=logging.INFO)

Create a new instance of a ChatBot

bot = ChatBot("Terminal", storage_adapter="chatterbot.storage.JsonFileStorageAdapter", logic_adapters=[ { "import_path": "chatterbot.logic.BestMatch", "statement_comparison_function": "chatterbot.comparisons.levenshtein_distance", "response_selection_method": "chatterbot.response_selection.get_first_response" }, { 'import_path': 'chatterbot.logic.LowConfidenceAdapter', 'threshold': 0.65, 'default_response': 'I am sorry, but I do not understand.' } ], input_adapter="chatterbot.input.TerminalAdapter", output_adapter="chatterbot.output.TerminalAdapter", database="./database.db" )

print("Type something to begin...")

The following loop will execute each time the user enters input

while True: try:

We pass None to this method because the parameter

    # is not used by the TerminalAdapter
    bot_input = bot.get_response(None)

# Press ctrl-c or ctrl-d on the keyboard to exit
except (KeyboardInterrupt, EOFError, SystemExit):
    break`

And i have 50 secs of execution with this code or code pasted above from @vkosuri

[ttrd06]

hello at all, @gunthercox did you had time to check my issue ? i tried again today and i always have the same "bug" but not with CNTK or Tensorflow. i wait for your answer please. Thank's in advance

[gunthercox]

@ttrd06 I did not have time to check your issue. I will look into it as soon as I can.

[somerandomdudeontheinternet]

Hi, I am also facing the same issue, I also tried creating a completely new virtualenv with new installs.

Hello
2017-04-09,22:41:02.238 INFO {input_adapter} [process_input_statement] Recieved input statement: Hello
2017-04-09,22:41:02.383 INFO {input_adapter} [process_input_statement] "Hello" is a known statement
2017-04-09,22:41:02.385 INFO {multi_adapter} [process] Not processing the statement using MathematicalEvaluation
2017-04-09,22:41:02.386 INFO {multi_adapter} [process] TimeLogicAdapter selected "The current time is 10:41 PM" as a response with a confidence of 0
2017-04-09,22:41:16.632 INFO {best_match} [process] Using "Hello" as a close match to "Hello"
2017-04-09,22:41:31.025 INFO {best_match} [process] Selecting response from 2 optimal responses.
2017-04-09,22:41:31.025 INFO {response_selection} [get_first_response] Selecting first response from list of 2 options.
2017-04-09,22:41:31.025 INFO {best_match} [process] Response selected. Using "Greetings!"
2017-04-09,22:41:31.025 INFO {multi_adapter} [process] BestMatch selected "Greetings!" as a response with a confidence of 1.0
2017-04-09,22:41:31.038 INFO {multi_adapter} [process] NoKnowledgeAdapter selected "Hello" as a response with a confidence of 0
Greetings!

As you can see it is taking 30 seconds for me with just around 500 statements in the json db file. I tried debugging it, in "best_match.py" module, fetching the data takes 15 seconds and filtering another 15 seconds. statement_list = self.chatbot.storage.get_response_statements() takes 15 seconds response_list = self.chatbot.storage.filter( in_response_to__contains=closest_match.text ) takes 15 seconds.

This is for JsonFileStorage which might be the culprit, I haven't tried using MongoDB. I will try using MongoDB and update.

[somerandomdudeontheinternet]

Alright, with the MongoDB it seems to work fine without any excessive delays. So the issue seems to be related to JsonFileStorage. @ttrd06 Try using MongoDB instead, that should solve your problem for now.

[ttrd06]

thank's for you help @somerandomdudeontheinternet and sorry for the late, i took some days for hollidays. i will try it asap and back to close this issue if ok.

Nice day at all

[ttrd06]

hello at all,

i confirm what @somerandomdudeontheinternet has said before, use only mongodb adaptor to feel a good experience. json db is just worst with training example include in tutorials.

thank's for all, this issue is closed for me

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.