Using real device loses all audio

[jefhil]

This works fine if I use it (typing "time" as the query) without a real device, namely: this.deviceModelId = 'default'; this.deviceInstanceId = 'default'; if I fill in a 'real' device that I registered in the action console as a speaker, I get no output -- neither text nor audio -- just a vacuous device_action. This happens in every version of the assistant sdk I have tried. Do I need to set something on my device definition? You'd think being a speaker would be enough. Attached are two logs showing output.

Bad.LOG:

time
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: null,
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: null,
  device_action: { device_request_json: '{"requestId":"5bb3e0d6-0000-2ec6-9f71-94eb2c141a3e"}\n' },
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: null,
  device_action: null,
  speech_results: [],
  dialog_state_out: 
   { supplemental_display_text: '',
     conversation_state: <Buffer 0a 26 43 23 35 62 63 64 61 62 66 63 2d 30 30 30 30 2d 32 36 38 64 2d 62 37 34 39 2d 38 38 33 64 32 34 66 33 61 61 38 63 12 e7 01 4b 6a 38 77 58 32 31 ... >,
     microphone_mode: 'CLOSE_MICROPHONE',
     volume_percentage: 0 } }
undefined

---

GOOD.LOG

time
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: null,
  device_action: null,
  speech_results: [],
  dialog_state_out: 
   { supplemental_display_text: '5:45.',
     conversation_state: <Buffer 0a 26 43 23 35 62 63 65 33 34 36 31 2d 30 30 30 30 2d 32 65 31 31 2d 61 65 37 37 2d 38 38 33 64 32 34 66 37 66 65 39 30 12 8f 02 4b 6b 34 77 58 32 31 ... >,
     microphone_mode: 'CLOSE_MICROPHONE',
     volume_percentage: 0 } }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 00 00 00 00 ff ff 00 00 00 00 ff ff ff ff 00 00 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff ff 00 00 00 00 00 00 00 00 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 17 00 e8 ff db ff dc ff cd ff 19 00 24 00 0c 00 e5 ff 81 ff 15 00 0f 00 23 00 3c 00 08 00 bd ff d0 ff 1e 00 dc ff fd ff 38 00 27 00 11 00 43 00 af 00 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 72 0b 38 10 33 14 a0 16 58 19 c8 1b 7b 1d 13 1f 7f 1f a9 1b 03 14 a6 0a af ff 25 f5 78 ee c3 eb ad eb 7e f0 8d f7 d2 ff a5 09 35 11 55 16 fb 19 90 1a ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer ec 28 5f 10 e5 f5 5d e0 51 d4 0f d5 31 de 37 e6 79 ea a0 ed b4 f1 e8 f9 45 07 e1 16 dc 24 0f 30 d3 36 8e 32 6b 22 e9 0d 14 f9 6d ea d1 e6 46 ea 10 f0 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer e5 08 f6 0a 8c 0d cf 11 76 17 6d 1a 82 1a a9 18 9b 14 1c 0f 6f 0a b6 07 67 06 38 05 03 04 6d 02 23 01 78 01 50 04 51 07 84 0a eb 0d c5 0e a7 0a 34 03 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 40 05 30 05 ac 04 30 05 f8 04 57 04 63 04 1b 05 3d 05 f6 05 a1 04 04 04 a1 03 58 03 20 03 5c 02 80 01 59 01 05 01 d4 ff aa ff 50 00 af fe c5 fd 02 fe ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 03 00 e6 ff 2a 00 5a 00 34 00 7d 00 be 00 5f 00 67 00 e6 ff 47 00 77 00 81 00 54 00 18 00 3b 00 4f 00 5f 00 c7 00 6b 00 40 00 8b 00 86 00 4f 00 35 00 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer af 01 f9 00 2c 00 1b ff e7 ff c9 00 1f 01 bf ff 33 00 6a fe 38 fe 5f ff 0b 00 9a 00 a9 01 12 01 47 00 90 ff c9 ff b2 00 0f ff 91 fe 0d 00 6b ff b2 ff ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 8d 37 28 35 2d 30 82 29 35 21 e7 18 95 11 ec 0b 2f 07 a5 03 8b 01 e7 00 9f 00 32 00 bc ff cf fe ea fc ac f9 43 f6 0e f3 68 f0 31 ee be ed e5 ee 93 f1 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer cd 0d c7 0a 20 06 c8 ff bd f8 96 f2 d5 ee 3b ee 2c ef 81 f0 af f1 c1 f2 ae f3 5a f5 45 f8 79 fc 90 01 8c 06 23 0b e3 0f ba 14 91 18 64 1b 59 1d 55 1e ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 49 f6 5d f4 d2 f4 45 f7 39 fa 58 fb 2f fb ea fb 48 fd 95 ff fb 01 0f 02 e1 00 95 00 44 00 1f ff 38 fd 6c fa 4b f5 38 ee 10 e9 fb e9 b5 ee 8a ed 23 e4 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 97 13 3f 17 22 1b 52 1e d2 1f e8 1f 0e 1f e7 1c 7a 1a 33 18 70 16 7a 15 c3 12 60 0d 09 09 8e 07 e9 05 15 04 f8 01 20 ff 25 fd 06 fd f0 fd 6f ff 96 ff ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 12 00 b9 ff d8 ff da ff 94 ff cd ff 34 00 1c 00 c6 ff b5 ff 2a 00 f1 ff f1 ff b1 ff 21 ff 04 00 35 00 ed ff ec ff 7d 00 be ff 08 00 f9 ff da ff c1 ff ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 6f 0a a8 0a 57 0b f1 0b a4 0b 1c 0b 3e 0a 3c 08 34 05 fe 00 78 fb ae f5 e6 f0 50 ed 16 ea 61 e7 92 e5 8d e3 36 e0 0a dc c3 d6 26 d0 a0 cb 56 ca 84 cd ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 1a 00 c3 04 2d 0b 88 12 cf 15 f0 14 05 12 33 0d 06 08 61 04 d5 02 a5 02 94 03 56 06 e9 08 59 09 5c 08 4a 05 54 00 f9 fb 27 f8 7a f4 79 f1 18 ee 8b e9 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 96 04 5b 08 fb 0d 86 13 94 15 e3 13 ba 0f 5b 08 62 01 60 fd 3f fc 97 fd 05 01 a9 05 4c 09 5e 0a 9a 0a 0a 0b 6b 0b 6f 0d 98 10 69 12 bd 13 e9 12 6e 0e ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 82 10 77 11 d7 0f 1e 0e 53 0c 24 0a bc 08 dc 07 a2 06 a4 03 4f 00 be fe cf fd 0d fe 88 00 98 03 0d 06 0a 09 cc 0b bc 0c 98 0b b1 0b 2f 0c ed 0b 42 0c ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer a4 fa eb f9 9f f8 1c f7 0e f5 7b f2 71 ef 34 ec 90 e9 52 e8 82 e6 0d e4 6f e1 4a e0 ab e1 18 e6 21 ec ff f1 e4 f5 98 f9 00 fe 70 02 8d 06 c1 0a 18 0e ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 0b fe db fd d7 fd ff fd 01 fe d4 fd 95 fd 94 fd bd fd d0 fd cf fd ac fd b5 fd 11 fe 5f fe 7a fe 79 fe 72 fe 8c fe db fe 36 ff 8b ff 94 ff 9f ff df ff ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer fe ff ff ff 00 00 00 00 00 00 00 00 00 00 ff ff fe ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff ff 00 00 01 00 00 00 ff ff ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
data: { event_type: 'EVENT_TYPE_UNSPECIFIED',
  audio_out: { audio_data: <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... > },
  device_action: null,
  speech_results: [],
  dialog_state_out: null }
5:45.

[Fleker]

You've tried the other samples, like for Python and C++? When you created your device model, did you register any device actions?

[jefhil]

Thanks for the response. I get the same behavior for python. C++ I did so long ago (and on diff machine) that I cant remember.

This project was initially done when I was working my way thru a sample, so I added the “blink times ” custom action from the tutorial

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

---

From: Nick notifications@github.com Sent: Friday, October 5, 2018 1:20:57 PM To: googlesamples/assistant-sdk-nodejs Cc: jefhil; Author Subject: Re: [googlesamples/assistant-sdk-nodejs] Using real device loses all audio (#3)

You've tried the other samples, like for Python and C++? When you created your device model, did you register any device actions?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/googlesamples/assistant-sdk-nodejs/issues/3#issuecomment-427438415, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Al74t_wz82va2ZQYqIjd5JQUspAU6lU1ks5uh5T5gaJpZM4XHMnv.

[jefhil]

So, i redid this whole process, created a new project, with new device models and devices and it worked fine. I tested builtin traits and added a couple of custom actions via the gactions tool and that worked too. Then, i realized that I was using my own account to grant permission to my device. So, I did it all over again, this time using a service account. I had to upgrade to google-auth-library 2.0 but i eventually got it all to work EXCEPT for device actions. It ignores the built in traits like dock and start/stop on/off and for 'color red' says it needs permission. any ideas? heres my action console screen: Linked device models . TPR Series Product name TPR Series Manufacturer name OhmniLabs Model Id ohmnilabstpr-tpr-series-9xw6v5 Device type Light Supported traits Brightness , ColorSpectrum , Dock , OnOff , StartStop