Error when trying to save data from 'learnf' elements to MongoDB

Expected Behavior

Given the following config.yaml file:

bot:
  storage:
    entities:
      learnf: mongo
      ...
    stores:
      mongo:
        type: mongo
        config:
          url: mongodb://localhost:27017/
          database: programy
          drop_all_first: false

And the following AIML content (from a file or from MongoDB it doesn't matter)

<?xml version="1.0" encoding="UTF-8" ?>
<aiml version="2.0">
    <category>
        <pattern>MY NAME IS *</pattern>
        <template>
            <think>
                <set name="name">
                    <star/>
                </set>
            </think>
            <learnf>
                <category>
                    <pattern>WHAT IS MY NAME</pattern>
                    <template>YOUR NAME IS
                        <eval>
                            <get name="name"/>
                        </eval>
                    </template>
                </category>
            </learnf>
            Nice to meet you <get name="name"/>
        </template>
    </category>
</aiml>

When the user says:

You: My name is Toto
Bot: Nice to meet you Toto

It should add a new entry in the MongoDB database, in the 'categories' collection with the following content:

_id: ObjectId("some_generated_value")
groupid: "LEARNF"
userid: "the_provided_user_id"
topic: "*"
that: "*"
pattern: "WHAT IS MY NAME"
template: "YOUR NAME IS Toto"

And when the bot is restarted, when the user says:

You: What is my name ?

Given the same userid, the bot should respond:

YOUR NAME IS Toto

Current Behavior

When the user says:

My name is Toto

An error is raised with the following message:

bson.errors.InvalidDocument: cannot encode object: <Element 'topic' at 0x270834ddc10>, of type: <class 'xml.etree.ElementTree.Element'>

And it does the same for elements 'topic', 'that', 'pattern' and 'template'.

Steps to Reproduce

Have MongoDB Server installed and running.
Download the template-y bot as described in the Wiki.
Open the file config.yaml in config/windows or config/xnix following your OS.
Modify the 'learnf' value from 'file' to 'mongo' in 'storage' > 'entities' section.

In the 'store' section add:

stores:
mongo:
type: mongo
  config:
    url: mongodb://localhost:27017/
    database: programy
    drop_all_first: false

In storage/categories directory, add a .aiml file with the following content:

<?xml version="1.0" encoding="UTF-8" ?>
<aiml version="2.0">
<category>
    <pattern>MY NAME IS *</pattern>
    <template>
        <think>
            <set name="name">
                <star/>
            </set>
        </think>
        <learnf>
            <category>
                <pattern>WHAT IS MY NAME</pattern>
                <template>YOUR NAME IS
                    <eval>
                        <get name="name"/>
                    </eval>
                </template>
            </category>
        </learnf>
        Nice to meet you <get name="name"/>
    </template>
</category>
</aiml>

Start your bot and enter the following sentence:
```
my name is toto
```
The bot will answer correctly but if you open your database and explore the 'categories' collection, it will be empty. And if you restart the bot it will not remember your name.
Open the 'template.log' file in scripts/windows/template.log to check the logs and see the error described in the section above.

Note: You can also reproduce this with an IDE like PyCharm to have a debugger, simply by extending or using any BotClient subclass (like FlaskRestBotClient) and running the bot with the same config.

Context (Environment)

Version of Program-y: 4.3
Version of Python: 3.8.9
Operating System ( inc Version ): Windows 10 - Build 19042
MongoDB Edition: MongoDB 4.4.6 Community

Detailed Description

The error occurs when data from 'learnf' elements are beeing inserted into the database.

But data from a 'learnf' element is in first place, converted into a 'Category' object (which is the schema of an object from the 'categories' collection in the database), and then inserted to the MongoDb collection.

When we look at the Category object created, attributes 'topic', 'that', 'pattern' and 'template' are objects instead of beeing (if I'm correct) attributes of type 'text'.

So, objects of type xml.etree.ElementTree.Element cannot be parsed to be saved in the MondoDB database, but if I'm right, the only attribute of this object that we are interested in is the 'text' attribute.

So, when a 'Category' object is created from data of a 'learnf' element, we can pass the text attribute of the 'topic', 'that' and 'pattern' objects to the 'Category' constructor, and convert the 'template' object to a text representation.

(See the code in the next section, it should be clearer)

Possible Implementation

Modify the save_learnf() method in class MongoLearnfStore

From:

    def save_learnf(self, client_context, category):
        YLogger.debug(self, "Storing learnf category in Mongo [%s] [%s] [%s] [%s] [%s] [%s]",
                      client_context.client.id,
                      client_context.userid,
                      category.pattern,
                      category.topic,
                      category.that,
                      category.template)

        pattern = category.pattern
        topic = category.topic
        that = category.that
        template = category.template

        groupid = "LEARNF"
        userid = client_context.userid

        category = Category(groupid=groupid, userid=userid, pattern=pattern, topic=topic, that=that, template=template)
        return self.add_document(category)

To:

    def save_learnf(self, client_context, category):
        YLogger.debug(self, "Storing learnf category in Mongo [%s] [%s] [%s] [%s] [%s] [%s]",
                      client_context.client.id,
                      client_context.userid,
                      category.pattern,
                      category.topic,
                      category.that,
                      category.template)

        # Changes
        pattern = category.pattern.text
        topic = category.topic.text
        that = category.that.text
        template = category.template.resolve(client_context)
        # End Changes

        groupid = "LEARNF"
        userid = client_context.userid

        category = Category(groupid=groupid, userid=userid, pattern=pattern, topic=topic, that=that, template=template)
        return self.add_document(category)

I did it locally and it works fine, but it may not be the correct way to do it, so it's simply a suggestion of what worked for me.

Thanks, excellent find, your change worked exactly as you stated and I have included the changes in the next release. Its also been added to the SQL version of the storage engine also

Keith

On 12 May 2021, at 23:11, Kddle @.***> wrote:

Expected Behavior

Given the following config.yaml file:

bot: storage: entities: learnf: mongo ... stores: mongo: type: mongo config: url: mongodb://localhost:27017/ database: programy drop_all_first: false And the following AIML content (from a file or from MongoDB it doesn't matter)

<?xml version="1.0" encoding="UTF-8" ?>
MY NAME IS *
When the user says:

You: My name is Toto Bot: Nice to meet you Toto It should add a new entry in the MongoDB database, in the 'categories' collection with the following content:

_id: ObjectId("some_generated_value") groupid: "LEARNF" userid: "the_provided_user_id" topic: "" that: "" pattern: "WHAT IS MY NAME" template: "YOUR NAME IS Toto" And when the bot is restarted, when the user says:

You: What is my name ? Given the same userid, the bot should respond:

YOUR NAME IS Toto Current Behavior

When the user says:

My name is Toto An error is raised with the following message:

bson.errors.InvalidDocument: cannot encode object: <Element 'topic' at 0x270834ddc10>, of type: <class 'xml.etree.ElementTree.Element'> And it does the same for elements 'topic', 'that', 'pattern' and 'template'.

Steps to Reproduce

Have MongoDB Server installed and running. Download the template-y bot as described in the Wiki. Open the file config.yaml in config/windows or config/xnix following your OS. Modify the 'learnf' value from 'file' to 'mongo' in 'storage' > 'entities' section. In the 'store' section add: stores: mongo: type: mongo config: url: mongodb://localhost:27017/ database: programy drop_all_first: false In storage/categories directory, add a .aiml file with the following content: <?xml version="1.0" encoding="UTF-8" ?>
MY NAME IS *
Start your bot and enter the following sentence: my name is toto The bot will answer correctly but if you open your database and explore the 'categories' collection, it will be empty. And if you restart the bot it will not remember your name. Open the 'template.log' file in scripts/windows/template.log to check the logs and see the error described in the section above. Note: You can also reproduce this with an IDE like PyCharm to have a debugger, simply by extending or using any BotClient subclass (like FlaskRestBotClient) and running the bot with the same config.

Context (Environment)

Version of Program-y: 4.3 Version of Python: 3.8.9 Operating System ( inc Version ): Windows 10 - Build 19042 MongoDB Edition: MongoDB 4.4.6 Community Detailed Description

The error occurs when data from 'learnf' elements are beeing inserted into the database.

But data from a 'learnf' element is in first place, converted into a 'Category' object (which is the schema of an object from the 'categories' collection in the database), and then inserted to the MongoDb collection.

When we look at the Category object created, attributes 'topic', 'that', 'pattern' and 'template' are objects instead of beeing (if I'm correct) attributes of type 'text'.

So, objects of type xml.etree.ElementTree.Element cannot be parsed to be saved in the MondoDB database, but if I'm right, the only attribute of this object that we are interested in is the 'text' attribute.

So, when a 'Category' object is created from data of a 'learnf' element, we can pass the text attribute of the 'topic', 'that' and 'pattern' objects to the 'Category' constructor, and convert the 'template' object to a text representation.

(See the code in the next section, it should be clearer)

Possible Implementation

Modify the save_learnf() method in class MongoLearnfStore

From:
def save_learnf(self, client_context, category):
    YLogger.debug(self, "Storing learnf category in Mongo [%s] [%s] [%s] [%s] [%s] [%s]",
                  client_context.client.id,
                  client_context.userid,
                  category.pattern,
                  category.topic,
                  category.that,
                  category.template)

    pattern = category.pattern
    topic = category.topic
    that = category.that
    template = category.template

    groupid = "LEARNF"
    userid = client_context.userid

    category = Category(groupid=groupid, userid=userid, pattern=pattern, topic=topic, that=that, template=template)
    return self.add_document(category)
To:
def save_learnf(self, client_context, category):
    YLogger.debug(self, "Storing learnf category in Mongo [%s] [%s] [%s] [%s] [%s] [%s]",
                  client_context.client.id,
                  client_context.userid,
                  category.pattern,
                  category.topic,
                  category.that,
                  category.template)

    # Changes
    pattern = category.pattern.text
    topic = category.topic.text
    that = category.that.text
    template = category.template.resolve(client_context)
    # End Changes

    groupid = "LEARNF"
    userid = client_context.userid

    category = Category(groupid=groupid, userid=userid, pattern=pattern, topic=topic, that=that, template=template)
    return self.add_document(category)
I did it locally and it works fine, but it may not be the correct way to do it, so it's simply a suggestion of what worked for me.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/keiffster/program-y/issues/290, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCUFMYLSTRG4Y7FQUKJMI3TNL4KXANCNFSM44ZOLSXA.

Great ! Thank you very much 🙂

Kddle

keiffster / program-y