"The CompositeEntityExtractor could not load the train file" using rasa.train

[BrianYing]

I was trying to directly use rasa.train instead of using "rasa train" command in terminal.

rasa_stack
    |__ data
       |__ nlu.json
    |__ models
    |__ rasa_run
       |__ rasa_run.py
    |__ config.yml

I set the training file parameter of train as "../data". But it gave me an error of "The CompositeEntityExtractor could not load the train file". /rasa1/lib/python3.7/site-packages/rasa_composite_entities/composite_entity_extractor.py:75: UserWarning: The CompositeEntityExtractor could not load the train file. "The CompositeEntityExtractor could not load "

Any idea on this issue? Thanks!

[BeWe11]

Can you share your "rasa_run.py" script with me for debugging?

[BrianYing]

from __future__ import division

import rasa
import json
import os

from typing import Text
from rasa.nlu.convert import convert_training_data
from datetime import datetime

def retrain_model(
        domain: Text = '../domain.yml',
        config: Text = '../config.yml',
        training_files: Text = '../data',
        output: Text = '../models',
        force_training=False):
    """
    This function takes all training parameters and start retraining pipeline
    and returns whether the training is success or not

    :param domain:
    :param config:
    :param training_files:
    :param output:
    :param force_training:
    :return: is_good: True if new model result a good result(nlu correct >= 95%)
                      False other wise;
             model_name: the new model name
    """
    # create model name
    model_name = datetime.now().strftime('%Y%m%d-%H%M%S')
    # call train
    rasa.train(domain, config, training_files, output, force_training, model_name)
    # call evaluate
    is_good = evaluate(training_files, model_name)

    return is_good, model_name

def evaluate(training_files, model_name):
    """
    This function takes in json training file and convert it into markdown for evaluation
    and use rasa.test to evaluate the model

    :param training_files: json training file
    :param model_name: trained model name
    :return: error.json for nlu and evaluate_result/ for core
             True if new model result a good result(nlu correct >= 95%)
             False other wise
    """
    # convert json to md for evaluation
    convert_training_data(data_file=training_files + '/nlu.json',
                          out_file='nlu_eval.md',
                          output_format='md',
                          language='')
    # evaluation
    model_path = '../models/' + model_name + '.tar.gz'
    rasa.test(model=model_path,
              stories='stories_eval.md',
              nlu_data='nlu_eval.md',
              output='evaluate_result')

    # count for number of all examples
    with open("../data/nlu.json", "r") as trainFile:
        data = json.load(trainFile)
        examples = data["rasa_nlu_data"]["common_examples"]
        num_data = len(examples)
        trainFile.close()

    # count for number of all errors
    with open("errors.json", "r") as errorFile:
        errors = json.load(errorFile)
        num_error = len(errors)
        errorFile.close()

    success_rate = (num_data - num_error) / num_data
    print("Number of data: ", num_data)
    print("Number of error: ", num_error)
    print("Success rate: ", success_rate)
    return True if success_rate >= 0.95 else False

if __name__ == '__main__':
    retrain_model(domain='../domain.yml',
                  config='../config.yml',
                  training_files='../data',
                  output='../models',
                  force_training=False)

[BrianYing]

Also, in order for the test function to work, I manually remove composite entity extractor by adding code below in /ENV/lib/python3.7/site-packages/rasa/nlu/test.py:

def remove_pretrained_extractors(pipeline: List[Component]) -> List[Component]:    

    """Removes pretrained extractors from the pipeline so that entities           
       from pre-trained extractors are not predicted upon parsing"""    

     pipeline = [c for c in pipeline if c.name not in PRETRAINED_EXTRACTORS and c.name.__str__() != 'CompositeEntityExtractor']  

     return pipeline

[BeWe11]

Unfortunately, I cannot access the raw train.json file when using the python API for training. How important is this ability to your workflow @BrianYing? Because of rasa's limited ability access to additional training data definitions, I'm thinking of changing the way the composite entity definitions are loaded.

Instead of defining the composite patterns in the same file as your other training data, you would create a new file exclusively for the patterns, and define the path in the config.yml file. Example:

pipeline:
- name: "rasa_composite_entities.CompositeEntityExtractor"
  path: "/path/to/patterns.json:

@BrianYing can you give me feedback if this would be feasible in your workflow?

[BrianYing]

I think it would be awesome @BeWe11 . This will also solve the issue that rasa x ui keep removing composite entity during editing. We were thinking of having a separate file anyways so that it would be easier to manipulate, then injecting it into the training file before training. This would bypass that step as well.

[nbeuchat]

@BeWe11 I also think this would be much better and would remove possible breaking changes that Rasa might introduce.

[BeWe11]

hey @nbeuchat,

you are right that the proposed change would help avoiding conflicts between changes in the rasa code base and this component. For example, it would probably have avoided https://github.com/BeWe11/rasa_composite_entities/issues/12 alltogether.

Unfortunately, I have no time right now to work on this. I've fixed https://github.com/BeWe11/rasa_composite_entities/issues/12, but obviously the underlying problem persists.

If you feel like creating a pull request for this, I'd be very grateful of course.

[ttlekich]

I can take a look at this if no one else gets around to it. Right now I have a fork in a (messy) state so I can specify the particular path of the nlu.json file:

- name: "rasa_composite_entities.CompositeEntityExtractor"
  training_file: "/path/to/training.json:

I had to make _get_train_files_cmd a non-static method to access the component_config. I am not sure if there is a better way to do this or if it breaks something I did not see.

Also, would it be more beneficial if a directory is allowed? Such that the extractor looks for composite entity patterns in the entire directory?

- name: "rasa_composite_entities.CompositeEntityExtractor"
  data: "/path/to/training_data"

Thanks!

[BeWe11]

Hey @ttlekich, thanks for considering helping me here :)

I think my preferred option would be to have a separate file just for composite definitions. Adding custom fields to the main JSON file was messy to begin with.

So in addition to the normal rasa training file, a user would create a separate composite_entities.json file and point to this file in the config:

- name: "rasa_composite_entities.CompositeEntityExtractor"
  data: "/path/to/composite_entities.json

This would also make pointing to directories unnecessary, as composite entities can be bundled in that single file.

Regarding _get_train_files_cmd: there is no problem making this method non-static.

What do you think? Feel free to open a pull request with your changes.

By the way I think there should be a fallback behavior in case no path is given in the yaml file, so currently working setups don't break.

[ttlekich]

@BeWe11 That sounds great and probably slightly easier than the directory approach. I'll take a look at this over the next week or two and send over a PR if I get something polished up.

[BeWe11]

Since version 0.5.0 composite entities can now be loaded by giving a path to a separate file (see the updated README). This issue should be resolved. Thanks to all and especially @ttlekich for the pull request that added this feature!