Dialogflow - Exception: "Call to a member function getQueryText() on null" when using OutputAudioConfig()

[00j]

Hello, Please could I ask for some help?

When I attempt to 'Detect Intent' on an example utterance and also request synthesised TTS using OutputAudioConfig() the DialogFlow API throws the following exception:

PHP Fatal error: Uncaught Error: Call to a member function getQueryText() on null in ..........\detect-intent-stream.php:191

It appears 'query_result' in StreamingDetectIntentResponse.php isn't populated when getting the response.

Is this expected?

Environment details

• OS: Linux Debian or Windows 10
• PHP version: 7.1.27 & 7.1.20
• Package name and version: DialogFlow: 0.8.3 ( Google Cloud : 0.103.0 )

Steps to reproduce

Please see examples.

Code example

To demonstrate the problem I've used the Meeting demo Agent and the standard detect intent stream example script as follows.

With example 1 you can see the example outputs the intermediate transcription and finally the detected intent and fulfilment text.

However with example 2 we see the exception after the intermediate transcription.

# 1. Example Without Output TTS:

require 'vendor/autoload.php';

use Google\Cloud\Dialogflow\V2\SessionsClient;
use Google\Cloud\Dialogflow\V2\AudioEncoding;
use Google\Cloud\Dialogflow\V2\InputAudioConfig;
use Google\Cloud\Dialogflow\V2\QueryInput;
use Google\Cloud\Dialogflow\V2\StreamingDetectIntentRequest;

$file_name = 'bookmeeting.raw';

detect_intent_stream(
    '****agent-id****',
    $file_name,
    'test-user-123',
    'en-GB'
);

/**
 * Returns the result of detect intent with streaming audio as input.
 * Using the same `session_id` between requests allows continuation
 * of the conversation.
 */
function detect_intent_stream($projectId, $path, $sessionId, $languageCode = 'en-GB')
{

    $serviceAccountPath = '/path/to/credentials.json';

    $config = [
        // the key 'credentials' was not documented!
        'credentials' => $serviceAccountPath,
    ];

    // need to use gRPC
    if (!defined('Grpc\STATUS_OK')) {
        throw new \Exception('Install the grpc extension ' .
            '(pecl install grpc)');
    }

    // new session
    $sessionsClient = new SessionsClient($config);
    $session = $sessionsClient->sessionName($projectId, $sessionId ?: uniqid());
    printf('Session path: %s' . PHP_EOL, $session);
    // hard coding audio_encoding and sample_rate_hertz for simplicity

    $audioConfig = new InputAudioConfig();
    $audioConfig->setAudioEncoding(AudioEncoding::AUDIO_ENCODING_LINEAR_16);
    $audioConfig->setLanguageCode($languageCode);
    $audioConfig->setSampleRateHertz(8000);

    // create query input
    $queryInput = new QueryInput();
    $queryInput->setAudioConfig($audioConfig);

    // first request contains the configuration
    $request = new StreamingDetectIntentRequest();

    $request->setSession($session);
    $request->setQueryInput($queryInput);
    $requests = [$request];
    // we are going to read small chunks of audio data from
    // a local audio file. in practice, these chunks should
    // come from an audio input device.
    $audioStream = fopen($path, 'rb');
    while (true) {
        $chunk = stream_get_contents($audioStream, 4096);
        if (!$chunk) {
            break;
        }
        $request = new StreamingDetectIntentRequest();
        $request->setInputAudio($chunk);
        $requests[] = $request;
    }

    // intermediate transcript info
    print(PHP_EOL . str_repeat("=", 20) . PHP_EOL);
    $stream = $sessionsClient->streamingDetectIntent();
    foreach ($requests as $request) {
        $stream->write($request);
    }

    foreach ($stream->closeWriteAndReadAll() as $i => $response) {
        $recognitionResult = $response->getRecognitionResult();
        if ($recognitionResult) {
            $transcript = $recognitionResult->getTranscript();
            printf('Intermediate transcript: %s' . PHP_EOL, $transcript);
        }
    }

    print(str_repeat("=", 20) . PHP_EOL);
    // get final response and relevant info
    $queryResult = $response->getQueryResult();

    $queryText = $queryResult->getQueryText();
    $intent = $queryResult->getIntent();
    if (isset($intent) && $intent) {
        $displayName = $intent->getDisplayName();
    }
    //$displayName = $intent->getDisplayName();
    $confidence = $queryResult->getIntentDetectionConfidence();
    $fulfilmentText = $queryResult->getFulfillmentText();
    // output relevant info
    printf('Query text: %s' . PHP_EOL, $queryText);
    printf('Detected intent: %s (confidence: %f)' . PHP_EOL, $displayName,
        $confidence);
    print(PHP_EOL);
    printf('Fulfilment text: %s' . PHP_EOL, $fulfilmentText);
    $sessionsClient->close();

}

# 2. Example With Output TTS:

require 'vendor/autoload.php';

use Google\Cloud\Dialogflow\V2\SessionsClient;
use Google\Cloud\Dialogflow\V2\AudioEncoding;
use Google\Cloud\Dialogflow\V2\InputAudioConfig;
use Google\Cloud\Dialogflow\V2\QueryInput;
use Google\Cloud\Dialogflow\V2\StreamingDetectIntentRequest;

use Google\Cloud\Dialogflow\V2\OutputAudioConfig;
use Google\Cloud\Dialogflow\V2\SynthesizeSpeechConfig;
use Google\Cloud\Dialogflow\V2\VoiceSelectionParams;

$file_name = 'bookmeeting.raw';

detect_intent_stream(
    '****agent-id****',
    $file_name,
    'test-user-123',
    'en-GB'
);

/**
 * Returns the result of detect intent with streaming audio as input.
 * Using the same `session_id` between requests allows continuation
 * of the conversation.
 */
function detect_intent_stream($projectId, $path, $sessionId, $languageCode = 'en-GB')
{

    $serviceAccountPath = '/path/to/credentials.json';

    $config = [
        // the key 'credentials' was not documented!
        'credentials' => $serviceAccountPath,
    ];

    // need to use gRPC
    if (!defined('Grpc\STATUS_OK')) {
        throw new \Exception('Install the grpc extension ' .
            '(pecl install grpc)');
    }

    // new session
    $sessionsClient = new SessionsClient($config);
    $session = $sessionsClient->sessionName($projectId, $sessionId ?: uniqid());
    printf('Session path: %s' . PHP_EOL, $session);
    // hard coding audio_encoding and sample_rate_hertz for simplicity

    $audioConfig = new InputAudioConfig();
    $audioConfig->setAudioEncoding(AudioEncoding::AUDIO_ENCODING_LINEAR_16);
    $audioConfig->setLanguageCode($languageCode);
    $audioConfig->setSampleRateHertz(8000);

    #=================== OUTPUT TTS FORMAT
    $audioOutputConfig = new OutputAudioConfig();
    $audioOutputConfig->setAudioEncoding(AudioEncoding::AUDIO_ENCODING_LINEAR_16);
    $audioOutputConfig->setSampleRateHertz(8000);

    $synthesizeSpeechConfig = new SynthesizeSpeechConfig();
    $synthesizeVoice = new VoiceSelectionParams();
    $synthesizeVoice->setName('en-GB-Wavenet-C');
    $synthesizeSpeechConfig->setVoice($synthesizeVoice);

    $audioOutputConfig->setSynthesizeSpeechConfig($synthesizeSpeechConfig);
    #=================== OUTPUT TTS FORMAT

    // create query input
    $queryInput = new QueryInput();
    $queryInput->setAudioConfig($audioConfig);

    // first request contains the configuration
    $request = new StreamingDetectIntentRequest();
    #=================== SET OUTPUT AUDIO CONFIG
    $request->setOutputAudioConfig($audioOutputConfig);
    #=================== SET OUTPUT AUDIO CONFIG

    $request->setSession($session);
    $request->setQueryInput($queryInput);
    $requests = [$request];
    // we are going to read small chunks of audio data from
    // a local audio file. in practice, these chunks should
    // come from an audio input device.
    $audioStream = fopen($path, 'rb');
    while (true) {
        $chunk = stream_get_contents($audioStream, 4096);
        if (!$chunk) {
            break;
        }
        $request = new StreamingDetectIntentRequest();
        $request->setInputAudio($chunk);
        $requests[] = $request;
    }

    // intermediate transcript info
    print(PHP_EOL . str_repeat("=", 20) . PHP_EOL);
    $stream = $sessionsClient->streamingDetectIntent();
    foreach ($requests as $request) {
        $stream->write($request);
    }

    print(str_repeat("+", 20) . PHP_EOL);

    foreach ($stream->closeWriteAndReadAll() as $i => $response) {
        $recognitionResult = $response->getRecognitionResult();
        if ($recognitionResult) {
            $transcript = $recognitionResult->getTranscript();
            printf('Intermediate transcript: %s' . PHP_EOL, $transcript);
        }
    }

    print(str_repeat("=", 20) . PHP_EOL);
    // get final response and relevant info
    $queryResult = $response->getQueryResult();

    #=================== OUTPUT AUDIO
    $audio = $response->getOutputAudio();
    file_put_contents('tts_audio.raw', $audio);
    #=================== OUTPUT AUDIO

    $queryText = $queryResult->getQueryText();
    $intent = $queryResult->getIntent();
    if (isset($intent) && $intent) {
        $displayName = $intent->getDisplayName();
    }
    //$displayName = $intent->getDisplayName();
    $confidence = $queryResult->getIntentDetectionConfidence();
    $fulfilmentText = $queryResult->getFulfillmentText();
    // output relevant info
    printf('Query text: %s' . PHP_EOL, $queryText);
    printf('Detected intent: %s (confidence: %f)' . PHP_EOL, $displayName,
        $confidence);
    print(PHP_EOL);
    printf('Fulfilment text: %s' . PHP_EOL, $fulfilmentText);
    $sessionsClient->close();

}

Example Audio

bookmeeting.zip

Composer Show for google/cloud

name     : google/cloud
descrip. : Google Cloud Client Library
keywords : Tasks, big query, bigquery, bigtable, cloud, datastore, gcs, google, google api, google api client, google apis, google apis client, google cloud, google cloud platform, kms, language, natural language, pub sub, pubsub, spanner, speech, stackdriver logging, storage, translate, translation, vision
versions : * v0.103.0
type     : library
license  : Apache License 2.0 (Apache-2.0) (OSI approved) https://spdx.org/licenses/Apache-2.0.html#licenseText
source   : [git] https://github.com/googleapis/google-cloud-php.git b4e559c2d803ea92c5e7bfe7fb78299f939799f6
dist     : [zip] https://api.github.com/repos/googleapis/google-cloud-php/zipball/b4e559c2d803ea92c5e7bfe7fb78299f939799f6 b4e559c2d803ea92c5e7bfe7fb78299f939799f6
path     : F:\data\Projects\d\src\library\vendor\google\cloud
names    : google/cloud, google/cloud-asset, google/cloud-automl, google/cloud-bigquery, google/cloud-bigquerydatatransfer, google/cloud-bigtable, google/cloud-common-protos, google/cloud-container, google/cloud-core, google/cloud-dataproc, google/cloud-datastore, google/cloud-debugger, google/cloud-dialogflow, google/cloud-dlp, google/cloud-error-reporting, google/cloud-firestore, google/cloud-iot, google/cloud-kms, google/cloud-language, google/cloud-logging, google/cloud-monitoring, google/cloud-oslogin, google/cloud-pubsub, google/cloud-redis, google/cloud-scheduler, google/cloud-security-center, google/cloud-spanner, google/cloud-speech, google/cloud-storage, google/cloud-talent, google/cloud-tasks, google/cloud-text-to-speech, google/cloud-trace, google/cloud-translate, google/cloud-videointelligence, google/cloud-vision, google/cloud-web-risk, google/cloud-web-security-scanner

autoload
psr-4
Google\Cloud\ => src, CommonProtos/src
Google\Cloud\Asset\ => Asset/src
Google\Cloud\AutoMl\ => AutoMl/src
Google\Cloud\BigQuery\ => BigQuery/src
Google\Cloud\BigQuery\DataTransfer\ => BigQueryDataTransfer/src
Google\Cloud\Bigtable\ => Bigtable/src
Google\Cloud\Container\ => Container/src
Google\Cloud\Core\ => Core/src
Google\Cloud\Dataproc\ => Dataproc/src
Google\Cloud\Datastore\ => Datastore/src
Google\Cloud\Debugger\ => Debugger/src
Google\Cloud\Dialogflow\ => Dialogflow/src
Google\Cloud\Dlp\ => Dlp/src
Google\Cloud\ErrorReporting\ => ErrorReporting/src
Google\Cloud\Firestore\ => Firestore/src
Google\Cloud\Iot\ => Iot/src
Google\Cloud\Kms\ => Kms/src
Google\Cloud\Language\ => Language/src
Google\Cloud\Logging\ => Logging/src
Google\Cloud\Monitoring\ => Monitoring/src
Google\Cloud\OsLogin\ => OsLogin/src
Google\Cloud\PubSub\ => PubSub/src
Google\Cloud\Redis\ => Redis/src
Google\Cloud\Scheduler\ => Scheduler/src
Google\Cloud\SecurityCenter\ => SecurityCenter/src
Google\Cloud\Spanner\ => Spanner/src
Google\Cloud\Speech\ => Speech/src
Google\Cloud\Storage\ => Storage/src
Google\Cloud\Talent\ => Talent/src
Google\Cloud\Tasks\ => Tasks/src
Google\Cloud\TextToSpeech\ => TextToSpeech/src
Google\Cloud\Trace\ => Trace/src
Google\Cloud\Translate\ => Translate/src
Google\Cloud\VideoIntelligence\ => VideoIntelligence/src
Google\Cloud\Vision\ => Vision/src
Google\Cloud\WebRisk\ => WebRisk/src
Google\Cloud\WebSecurityScanner\ => WebSecurityScanner/src
GPBMetadata\Google\ => CommonProtos/metadata
GPBMetadata\Google\Bigtable\ => Bigtable/metadata
GPBMetadata\Google\Container\ => Container/metadata
GPBMetadata\Google\Cloud\Asset\ => Asset/metadata
GPBMetadata\Google\Cloud\Automl\ => AutoMl/metadata
GPBMetadata\Google\Cloud\Bigquery\Datatransfer\ => BigQueryDataTransfer/metadata
GPBMetadata\Google\Cloud\Dataproc\ => Dataproc/metadata
GPBMetadata\Google\Cloud\Dialogflow\ => Dialogflow/metadata
GPBMetadata\Google\Cloud\Iot\ => Iot/metadata
GPBMetadata\Google\Cloud\Kms\ => Kms/metadata
GPBMetadata\Google\Cloud\Language\ => Language/metadata
GPBMetadata\Google\Cloud\Oslogin\ => OsLogin/metadata
GPBMetadata\Google\Cloud\Redis\ => Redis/metadata
GPBMetadata\Google\Cloud\Scheduler\ => Scheduler/metadata
GPBMetadata\Google\Cloud\Securitycenter\ => SecurityCenter/metadata
GPBMetadata\Google\Cloud\Speech\ => Speech/metadata
GPBMetadata\Google\Cloud\Talent\ => Talent/metadata
GPBMetadata\Google\Cloud\Tasks\ => Tasks/metadata
GPBMetadata\Google\Cloud\Texttospeech\ => TextToSpeech/metadata
GPBMetadata\Google\Cloud\Videointelligence\ => VideoIntelligence/metadata
GPBMetadata\Google\Cloud\Vision\ => Vision/metadata
GPBMetadata\Google\Cloud\Webrisk\ => WebRisk/metadata
GPBMetadata\Google\Cloud\Websecurityscanner\ => WebSecurityScanner/metadata
GPBMetadata\Google\Datastore\ => Datastore/metadata
GPBMetadata\Google\Devtools\Clouddebugger\ => Debugger/metadata
GPBMetadata\Google\Devtools\Clouderrorreporting\ => ErrorReporting/metadata
GPBMetadata\Google\Devtools\Cloudtrace\ => Trace/metadata
GPBMetadata\Google\Firestore\ => Firestore/metadata
GPBMetadata\Google\Logging\ => Logging/metadata
GPBMetadata\Google\Monitoring\ => Monitoring/metadata
GPBMetadata\Google\Privacy\Dlp\ => Dlp/metadata
GPBMetadata\Google\Pubsub\ => PubSub/metadata
GPBMetadata\Google\Spanner\ => Spanner/metadata

requires
google/auth ^1.5.1
google/common-protos ^1.0
google/gax ^1.0
guzzlehttp/guzzle ^5.3|^6.0
guzzlehttp/psr7 ^1.2
monolog/monolog ~1
php >=5.5
psr/http-message 1.0.*
ramsey/uuid ~3
rize/uri-template ~0.3

requires (dev)
erusev/parsedown ^1.6
google/cloud-tools ^0.6
opis/closure ^3.0
phpdocumentor/reflection ^3.0
phpseclib/phpseclib ^2
phpunit/phpunit ^4.8|^5.0
squizlabs/php_codesniffer 2.*
swaggest/json-schema ^0.12.0
symfony/console ^3.0
symfony/lock 3.3.x-dev#1ba6ac9
vierbergenlars/php-semver ^3.0

suggests
opis/closure May be used to serialize closures to process jobs in the batch daemon. Please require version ^3.
phpseclib/phpseclib May be used in place of OpenSSL for creating signed Cloud Storage URLs. Please require version ^2.

replaces
google/cloud-asset 0.3.3
google/cloud-automl 0.2.2
google/cloud-bigquery 1.7.0
google/cloud-bigquerydatatransfer 0.10.4
google/cloud-bigtable 0.12.4
google/cloud-common-protos 0.2.0
google/cloud-container 0.10.3
google/cloud-core 1.28.1
google/cloud-dataproc 0.12.0
google/cloud-datastore 1.9.4
google/cloud-debugger 0.18.4
google/cloud-dialogflow 0.8.3
google/cloud-dlp 0.18.4
google/cloud-error-reporting 0.14.4
google/cloud-firestore 1.6.1
google/cloud-iot 0.7.4
google/cloud-kms 1.3.4
google/cloud-language 0.19.3
google/cloud-logging 1.16.4
google/cloud-monitoring 0.16.4
google/cloud-oslogin 0.9.4
google/cloud-pubsub 1.12.2
google/cloud-redis 0.8.4
google/cloud-scheduler 1.0.1
google/cloud-security-center 0.1.1
google/cloud-spanner 1.15.4
google/cloud-speech 0.24.0
google/cloud-storage 1.12.2
google/cloud-talent 0.4.0
google/cloud-tasks 1.1.0
google/cloud-text-to-speech 0.4.3
google/cloud-trace 0.14.3
google/cloud-translate 1.2.9
google/cloud-videointelligence 1.5.3
google/cloud-vision 0.22.4
google/cloud-web-risk 0.1.1
google/cloud-web-security-scanner 0.1.0

Thank you!

[jdpedrie]

Hi @00j,

I did some testing and found that the query result is not returned in the final streamed response. You should inspect the streamed response and set $queryResult within the loop rather than attempting to grab it from the final iteration.

    $queryResult = null;
    foreach ($stream->closeWriteAndReadAll() as $i => $response) {
        $recognitionResult = $response->getRecognitionResult();
        if ($recognitionResult) {
            $transcript = $recognitionResult->getTranscript();
            printf('Intermediate transcript: %s' . PHP_EOL, $transcript);
        }

        $queryResult = !$queryResult && $response->getQueryResult()
            ? $response->getQueryResult()
            : $queryResult;
    }

    if ($queryResult) {
        echo $queryResult->getQueryText();
    }

You should be sure to check that $queryResult is not null before calling a method on the instance which you expect the variable to contain.

[00j]

Hi John,

Thank you very much for your prompt response and highlighting the problem.

Following a quick test, It appears that when output audio config is not used the getQueryResult() is set on the last iteration of:

foreach ($stream->closeWriteAndReadAll() as $response)

Where as when output audio config is used then getQueryResult() is set on the penultimate iteration, as you indicated.

My example was lifted from a php-docs-samples sample:

https://github.com/GoogleCloudPlatform/php-docs-samples/blob/master/dialogflow/src/detect_intent_stream.php

I'll post a recommendation for an update.

Thanks again for your help, very much appreciated Jamie