Unable to send some utterances to the DialogFlow agent during k-fold cross validation

[Davidelanz]

Bug description

I am working with DialogFlow and I have a Standard Edition plan. I am validating the agent in the following way. First, I prepare the dataset made by the utterances which trained the agent:

export GOOGLE_APPLICATION_CREDENTIALS=client_key.json
mkdir dataset_botium
node_modules/.bin/botium-cli nlpextract --config botium.json --convos dataset_botium --verbose

Then, I execute the 5-fold cross validation:

export GOOGLE_APPLICATION_CREDENTIALS=kfold_key.json
node_modules/.bin/botium-cli nlpanalytics k-fold -k 5 --config botium.json --convos dataset_botium

While validating, I encounter the following error for many utterances:

-Fold Round 1: 
    Failed sending utterance "#################" 
- Cannot send message to dialogflow container: 
    {
        "code":9,
        "message":"Intent with id #####################' not found among intents in environment ''
                           for agent with id '########################'.\ncom.google.apps.framework.request.StatusException:
                           <eye3 title='FAILED_PRECONDITION'/> generic::FAILED_PRECONDITION: 
                           Intent with id '########################### not found among intents in environment '
                           ' for agent with id '#################################'.",
        "details":[]
    }

Moreover, most of the validation results end with a "None" result, while this does not happen for testing manually the client. The validation results are the following:

############# Summary #############
K-Fold Round 1: Precision=0.5833 Recall=0.0990 F1-Score=0.1692 Tech.OK=286 Tech.Failures=47
K-Fold Round 2: Precision=0.7278 Recall=0.1181 F1-Score=0.2032 Tech.OK=331 Tech.Failures=0
K-Fold Round 3: Precision=0.8234 Recall=0.1348 F1-Score=0.2317 Tech.OK=327 Tech.Failures=0
K-Fold Round 4: Precision=0.7633 Recall=0.1504 F1-Score=0.2513 Tech.OK=285 Tech.Failures=37
K-Fold Round 5: Precision=0.6250 Recall=0.1279 F1-Score=0.2124 Tech.OK=277 Tech.Failures=38
K-Fold Avg: Precision=0.7046 Recall=0.1260 F1-Score=0.2138
Wrote output file k-fold.csv
Wrote predictions output file k-fold-predictions.csv

To Reproduce The code and the corpus is available at https://github.com/Davidelanz/nlp-contextual-meaning

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.