Memory leak on prolonged silence

[orgads]

Hi,

We're experiencing a significant memory leak when the input stream is silent for a long time.

Can be reproduced with the following script. Requires npm i dev-zero-stream.

Happens with both SpeechRecognizer and DialogServiceConnector. You can switch between them by [re]setting isDirect.

Sample outputs after the script.

const KEY = 'YOUR_KEY_HERE';
import * as sdk from 'microsoft-cognitiveservices-speech-sdk';
import * as throttle from 'throttle';
import { Readable } from 'stream';
import axios from 'axios';
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
const zero = require('dev-zero-stream');
const silence = Buffer.from(new Array(65536).fill(0x55));

const isDirect = false;

async function getToken() {
  const config = {
    headers: { 'Ocp-Apim-Subscription-Key': KEY },
    timeout: 20000
  };
  const response = await axios.post(`https://westus2.api.cognitive.microsoft.com/sts/v1.0/issuetoken`, {}, config);
  return response.data as string;
}

async function main() {
  const registerEvent = (reco: sdk.SpeechRecognizer | sdk.DialogServiceConnector) => {
    reco.recognizing = (_s: sdk.Recognizer, event: sdk.SpeechRecognitionEventArgs) => {
      console.log('hypothesis', { text: event.result.text });
    };

    reco.recognized = (_s: sdk.Recognizer, event: sdk.SpeechRecognitionEventArgs) => {
      console.debug(`(recognized)  Reason: ${sdk.ResultReason[event.result.reason]} Text: ${event.result.text}`);
    };

    reco.sessionStarted = (_s: sdk.Recognizer, event: sdk.SessionEventArgs) => {
      console.debug(`(sessionStarted) SessionId: ${event.sessionId}`);
    };

    reco.sessionStopped = (_s: sdk.Recognizer, event: sdk.SessionEventArgs) => {
      console.debug(`(sessionStopped) SessionId: ${event.sessionId}`);
    };

    reco.speechStartDetected = (_s, event) => {
      console.debug(`(speechStartDetected) SessionId: ${event.sessionId}`);
    };

    reco.speechEndDetected = (_s, event) => {
      console.debug(`(speechEndDetected) SessionId: ${event.sessionId}`);
    };
  }
  let audioConfig;
  // eslint-disable-next-line @typescript-eslint/no-unsafe-call
  const audioStream = zero(50 * 1024 * 1024, silence) as Readable;
  const pushStream = sdk.AudioInputStream.createPushStream();
  const handleData = (buf: Buffer) => {
    pushStream.write(buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength));
  };
  const throt = new throttle(32000);

  audioStream
    .pipe(throt)
    .on('data', handleData)
    .once('close', () => {
      console.log('close')
      pushStream.close();
      audioStream.removeListener('data', handleData);
    });
  audioConfig = sdk.AudioConfig.fromStreamInput(pushStream);
  const authToken = await getToken();
  if (isDirect) {
    const botConfig = sdk.BotFrameworkConfig.fromAuthorizationToken(authToken, 'westus2');
    const reco = new sdk.DialogServiceConnector(botConfig, audioConfig);
    registerEvent(reco);
    reco.listenOnceAsync(undefined, (e) => console.error(e))
  } else {
    const speechConfig = sdk.SpeechConfig.fromAuthorizationToken(authToken, 'westus2');
    speechConfig.authorizationToken = authToken;
    const reco = new sdk.SpeechRecognizer(speechConfig, audioConfig);
    registerEvent(reco);
    reco.startContinuousRecognitionAsync(
      undefined,
      (err) => {
        console.error(err);
      }
    );
  }
}
main().catch((r) => console.error(r));
setInterval(() => console.log(process.memoryUsage()), 2000);

Sample outputs. Notice how rss keeps growing all the time. I also saw on heap timeline that some BufferEntrys are retained by ReplayableAudioNode on non-direct, and many ArrayBuffers are retained by ChunkedArrayBufferStream on direct.

SpeechRecognizer

{rss: 185102336, heapTotal: 146821120, heapUsed: 118013040, external: 2668067, arrayBuffers: 498977}
(sessionStarted) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
{rss: 193413120, heapTotal: 150753280, heapUsed: 120051112, external: 2705764, arrayBuffers: 533555}
{rss: 194752512, heapTotal: 150753280, heapUsed: 121604952, external: 2920324, arrayBuffers: 748115}
{rss: 195543040, heapTotal: 150753280, heapUsed: 123208616, external: 3141284, arrayBuffers: 969075}
{rss: 197181440, heapTotal: 150753280, heapUsed: 124855240, external: 3362244, arrayBuffers: 1190035}
{rss: 198656000, heapTotal: 150753280, heapUsed: 126417720, external: 3583204, arrayBuffers: 1410995}
{rss: 200331264, heapTotal: 150753280, heapUsed: 127981240, external: 3804164, arrayBuffers: 1631955}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 206217216, heapTotal: 153899008, heapUsed: 120222328, external: 3495724, arrayBuffers: 1323515}
{rss: 164601856, heapTotal: 112218112, heapUsed: 106034432, external: 2487353, arrayBuffers: 315144}
{rss: 166789120, heapTotal: 113266688, heapUsed: 106656824, external: 2651058, arrayBuffers: 478849}
{rss: 166711296, heapTotal: 112218112, heapUsed: 106410448, external: 2763924, arrayBuffers: 591715}
{rss: 167120896, heapTotal: 112218112, heapUsed: 106211040, external: 2868828, arrayBuffers: 696619}
{rss: 168779776, heapTotal: 112480256, heapUsed: 107061088, external: 3040987, arrayBuffers: 868778}
{rss: 169807872, heapTotal: 112480256, heapUsed: 106744000, external: 3145661, arrayBuffers: 973452}
{rss: 172134400, heapTotal: 112480256, heapUsed: 107451296, external: 3309366, arrayBuffers: 1137157}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 172965888, heapTotal: 112480256, heapUsed: 106903896, external: 3442069, arrayBuffers: 1269860}
{rss: 174317568, heapTotal: 112480256, heapUsed: 106993736, external: 3570382, arrayBuffers: 1398173}
{rss: 175468544, heapTotal: 112480256, heapUsed: 106946688, external: 3734349, arrayBuffers: 1519739}
{rss: 177831936, heapTotal: 112480256, heapUsed: 107733296, external: 3855915, arrayBuffers: 1683706}
{rss: 178937856, heapTotal: 112910336, heapUsed: 107914248, external: 3977481, arrayBuffers: 1805272}
{rss: 180293632, heapTotal: 112910336, heapUsed: 107594144, external: 4090347, arrayBuffers: 1918138}
{rss: 182038528, heapTotal: 112910336, heapUsed: 108232568, external: 4245860, arrayBuffers: 2073651}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 184176640, heapTotal: 112910336, heapUsed: 108426808, external: 4393991, arrayBuffers: 2221782}
{rss: 185577472, heapTotal: 113078272, heapUsed: 108536384, external: 4549426, arrayBuffers: 2377217}
{rss: 186761216, heapTotal: 113078272, heapUsed: 108484512, external: 4670992, arrayBuffers: 2498783}
{rss: 187752448, heapTotal: 113078272, heapUsed: 108293264, external: 4784104, arrayBuffers: 2611895}
{rss: 189485056, heapTotal: 113078272, heapUsed: 108929776, external: 4947924, arrayBuffers: 2775715}
{rss: 190353408, heapTotal: 113078272, heapUsed: 108628304, external: 5052614, arrayBuffers: 2880405}
{rss: 192425984, heapTotal: 113078272, heapUsed: 109270808, external: 5222719, arrayBuffers: 3050510}
{rss: 194981888, heapTotal: 113078272, heapUsed: 109132000, external: 5329546, arrayBuffers: 3157337}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 194957312, heapTotal: 113078272, heapUsed: 109022288, external: 5501384, arrayBuffers: 3329175}
{rss: 196304896, heapTotal: 113078272, heapUsed: 108996312, external: 5626251, arrayBuffers: 3454042}
{rss: 198119424, heapTotal: 113078272, heapUsed: 108976320, external: 5804810, arrayBuffers: 3575608}
{rss: 202264576, heapTotal: 113246208, heapUsed: 109934896, external: 5926376, arrayBuffers: 3754167}
{rss: 202113024, heapTotal: 113246208, heapUsed: 109885432, external: 6047942, arrayBuffers: 3875733}
{rss: 202350592, heapTotal: 113246208, heapUsed: 109641640, external: 6154654, arrayBuffers: 3982445}
{rss: 204234752, heapTotal: 113246208, heapUsed: 110284352, external: 6316567, arrayBuffers: 4144358}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 205799424, heapTotal: 113246208, heapUsed: 110326000, external: 6433961, arrayBuffers: 4261752}
{rss: 208748544, heapTotal: 113508352, heapUsed: 110445456, external: 6613844, arrayBuffers: 4441635}
{rss: 209948672, heapTotal: 113508352, heapUsed: 110356120, external: 6735410, arrayBuffers: 4563201}
{rss: 211832832, heapTotal: 113508352, heapUsed: 110987696, external: 6899115, arrayBuffers: 4726906}
{rss: 212881408, heapTotal: 113840128, heapUsed: 111122496, external: 7018728, arrayBuffers: 4846519}
{rss: 213721088, heapTotal: 113840128, heapUsed: 110821336, external: 7125309, arrayBuffers: 4953100}
{rss: 216293376, heapTotal: 113840128, heapUsed: 111496264, external: 7287222, arrayBuffers: 5115013}
{rss: 287621120, heapTotal: 110383104, heapUsed: 102295872, external: 2833049, arrayBuffers: 651008}
{rss: 287739904, heapTotal: 110383104, heapUsed: 102279584, external: 2914841, arrayBuffers: 742632}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 287625216, heapTotal: 110383104, heapUsed: 102171048, external: 2497271, arrayBuffers: 318208}
{rss: 287936512, heapTotal: 110383104, heapUsed: 102218600, external: 2566041, arrayBuffers: 393832}
{rss: 288681984, heapTotal: 110383104, heapUsed: 102266600, external: 2642841, arrayBuffers: 470632}
{rss: 289619968, heapTotal: 110645248, heapUsed: 102260448, external: 2710041, arrayBuffers: 537832}
{rss: 290131968, heapTotal: 110645248, heapUsed: 102231056, external: 2774041, arrayBuffers: 601832}
{rss: 290390016, heapTotal: 110645248, heapUsed: 102237904, external: 2844441, arrayBuffers: 672232}
{rss: 291262464, heapTotal: 110813184, heapUsed: 102375656, external: 2921241, arrayBuffers: 749032}
{rss: 291827712, heapTotal: 110813184, heapUsed: 102385480, external: 2991641, arrayBuffers: 819432}
{rss: 293052416, heapTotal: 110813184, heapUsed: 102393048, external: 3062041, arrayBuffers: 889832}
{rss: 293851136, heapTotal: 110813184, heapUsed: 102453128, external: 3138841, arrayBuffers: 966632}
{rss: 294522880, heapTotal: 110813184, heapUsed: 102439688, external: 3206041, arrayBuffers: 1033832}
{rss: 295272448, heapTotal: 110813184, heapUsed: 102429240, external: 3270041, arrayBuffers: 1097832}
{rss: 296402944, heapTotal: 110813184, heapUsed: 102437528, external: 3340441, arrayBuffers: 1168232}
{rss: 303079424, heapTotal: 110813184, heapUsed: 103263880, external: 3638041, arrayBuffers: 1465832}
{rss: 304594944, heapTotal: 110813184, heapUsed: 102876688, external: 3821351, arrayBuffers: 1649142}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 305893376, heapTotal: 110813184, heapUsed: 103047200, external: 4020678, arrayBuffers: 1848469}
{rss: 308609024, heapTotal: 111861760, heapUsed: 103344352, external: 4210273, arrayBuffers: 2038064}
{rss: 310673408, heapTotal: 111861760, heapUsed: 103560752, external: 4383337, arrayBuffers: 2211128}
{rss: 368861184, heapTotal: 114126848, heapUsed: 103785000, external: 4532572, arrayBuffers: 2360363}
{rss: 369717248, heapTotal: 114126848, heapUsed: 104146048, external: 4730867, arrayBuffers: 2558658}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 370688000, heapTotal: 114126848, heapUsed: 104815200, external: 4964622, arrayBuffers: 2792413}
{rss: 370950144, heapTotal: 114126848, heapUsed: 105058024, external: 5162171, arrayBuffers: 2989962}
{rss: 371081216, heapTotal: 114126848, heapUsed: 105305080, external: 5343443, arrayBuffers: 3171234}
{rss: 371458048, heapTotal: 114458624, heapUsed: 105950048, external: 5539668, arrayBuffers: 3367459}
{rss: 371482624, heapTotal: 114458624, heapUsed: 104859808, external: 5605015, arrayBuffers: 3432806}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 371544064, heapTotal: 114458624, heapUsed: 105824656, external: 5878883, arrayBuffers: 3706674}
{rss: 371605504, heapTotal: 114458624, heapUsed: 106297880, external: 6083578, arrayBuffers: 3911369}
{rss: 371625984, heapTotal: 114458624, heapUsed: 106730864, external: 6281742, arrayBuffers: 4109533}
{rss: 371785728, heapTotal: 118652928, heapUsed: 107252832, external: 6471714, arrayBuffers: 4299505}
{rss: 373276672, heapTotal: 118652928, heapUsed: 106136920, external: 6553576, arrayBuffers: 4381367}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 374386688, heapTotal: 118652928, heapUsed: 108545088, external: 6888723, arrayBuffers: 4716514}
{rss: 374939648, heapTotal: 118652928, heapUsed: 107538288, external: 7001312, arrayBuffers: 4829103}
{rss: 374439936, heapTotal: 118484992, heapUsed: 104315560, external: 3081401, arrayBuffers: 909192}
{rss: 374452224, heapTotal: 118484992, heapUsed: 106375784, external: 3396129, arrayBuffers: 1223920}
(speechEndDetected) SessionId: 182C33E8E6A3455BAF5129AF53F7D123
(recognized)  Reason: NoMatch Text: undefined
{rss: 374497280, heapTotal: 118484992, heapUsed: 105650096, external: 3481286, arrayBuffers: 1309077}
{rss: 374771712, heapTotal: 118484992, heapUsed: 104563912, external: 3610734, arrayBuffers: 1438525}

Direct

{rss: 190283776, heapTotal: 149704704, heapUsed: 122045624, external: 2566264, arrayBuffers: 394095}
(sessionStarted) SessionId: A2A1A0BADE0A4823B9DB5FE77E52D3DB
{rss: 193368064, heapTotal: 149966848, heapUsed: 125451032, external: 2942555, arrayBuffers: 770346}
(speechEndDetected) SessionId: A2A1A0BADE0A4823B9DB5FE77E52D3DB
(recognized)  Reason: NoMatch Text: undefined
(sessionStopped) SessionId: A2A1A0BADE0A4823B9DB5FE77E52D3DB
{rss: 201453568, heapTotal: 154685440, heapUsed: 121312984, external: 2840238, arrayBuffers: 668029}
{rss: 201740288, heapTotal: 154685440, heapUsed: 121727288, external: 2910638, arrayBuffers: 738429}
{rss: 201773056, heapTotal: 154685440, heapUsed: 122143416, external: 2981038, arrayBuffers: 808829}
{rss: 202121216, heapTotal: 154685440, heapUsed: 122558416, external: 3051438, arrayBuffers: 879229}
{rss: 202240000, heapTotal: 154685440, heapUsed: 122972760, external: 3121838, arrayBuffers: 949629}
{rss: 202731520, heapTotal: 154685440, heapUsed: 123387424, external: 3192238, arrayBuffers: 1020029}
{rss: 160399360, heapTotal: 112218112, heapUsed: 105637448, external: 2740662, arrayBuffers: 568453}
{rss: 160808960, heapTotal: 112218112, heapUsed: 106052176, external: 2811062, arrayBuffers: 638853}
{rss: 161615872, heapTotal: 112218112, heapUsed: 106465408, external: 2881462, arrayBuffers: 709253}
{rss: 161337344, heapTotal: 112218112, heapUsed: 105920992, external: 2942262, arrayBuffers: 770053}
{rss: 162213888, heapTotal: 112218112, heapUsed: 106334824, external: 3012662, arrayBuffers: 840453}
{rss: 162148352, heapTotal: 112218112, heapUsed: 105909224, external: 3070262, arrayBuffers: 898053}
{rss: 162557952, heapTotal: 112218112, heapUsed: 106326240, external: 3140662, arrayBuffers: 968453}
{rss: 162623488, heapTotal: 112218112, heapUsed: 105725528, external: 3191862, arrayBuffers: 1019653}
{rss: 163155968, heapTotal: 112218112, heapUsed: 106140608, external: 3262262, arrayBuffers: 1090053}
{rss: 164040704, heapTotal: 112218112, heapUsed: 106555256, external: 3332662, arrayBuffers: 1160453}
{rss: 164167680, heapTotal: 112218112, heapUsed: 105955064, external: 3390262, arrayBuffers: 1218053}
{rss: 164790272, heapTotal: 112218112, heapUsed: 106370056, external: 3460662, arrayBuffers: 1288453}
{rss: 164724736, heapTotal: 112218112, heapUsed: 105764536, external: 3511862, arrayBuffers: 1339653}
{rss: 165138432, heapTotal: 112218112, heapUsed: 106178336, external: 3582262, arrayBuffers: 1410053}
{rss: 166412288, heapTotal: 112218112, heapUsed: 106592096, external: 3652662, arrayBuffers: 1480453}
{rss: 166313984, heapTotal: 112218112, heapUsed: 105992880, external: 3710262, arrayBuffers: 1538053}
{rss: 167256064, heapTotal: 112218112, heapUsed: 106405968, external: 3780662, arrayBuffers: 1608453}
{rss: 167194624, heapTotal: 112218112, heapUsed: 105953680, external: 3831862, arrayBuffers: 1659653}
{rss: 167620608, heapTotal: 112218112, heapUsed: 106385408, external: 3905462, arrayBuffers: 1733253}
{rss: 167579648, heapTotal: 112218112, heapUsed: 105955720, external: 3969462, arrayBuffers: 1797253}
{rss: 168333312, heapTotal: 112218112, heapUsed: 106379040, external: 4039862, arrayBuffers: 1867653}
{rss: 168038400, heapTotal: 112218112, heapUsed: 105767304, external: 4091062, arrayBuffers: 1918853}
{rss: 169263104, heapTotal: 112218112, heapUsed: 106187176, external: 4161462, arrayBuffers: 1989253}
{rss: 169771008, heapTotal: 112218112, heapUsed: 106600528, external: 4231862, arrayBuffers: 2059653}
{rss: 169504768, heapTotal: 112218112, heapUsed: 106008024, external: 4289462, arrayBuffers: 2117253}
{rss: 170598400, heapTotal: 112218112, heapUsed: 106431160, external: 4359862, arrayBuffers: 2187653}
{rss: 170205184, heapTotal: 112218112, heapUsed: 105824592, external: 4411062, arrayBuffers: 2238853}
{rss: 170881024, heapTotal: 112218112, heapUsed: 106237560, external: 4481462, arrayBuffers: 2309253}
{rss: 171429888, heapTotal: 112218112, heapUsed: 106650576, external: 4551862, arrayBuffers: 2379653}
{rss: 172089344, heapTotal: 112218112, heapUsed: 106038280, external: 4609462, arrayBuffers: 2437253}
{rss: 172748800, heapTotal: 112218112, heapUsed: 106458864, external: 4679862, arrayBuffers: 2507653}
{rss: 172335104, heapTotal: 112218112, heapUsed: 105991416, external: 4740662, arrayBuffers: 2568453}
{rss: 173010944, heapTotal: 112218112, heapUsed: 106401528, external: 4811062, arrayBuffers: 2638853}
{rss: 173817856, heapTotal: 112218112, heapUsed: 106811792, external: 4881462, arrayBuffers: 2709253}
{rss: 173547520, heapTotal: 112218112, heapUsed: 106224152, external: 4932662, arrayBuffers: 2760453}
{rss: 174587904, heapTotal: 112218112, heapUsed: 106635128, external: 5011254, arrayBuffers: 2839045}
{rss: 174051328, heapTotal: 112218112, heapUsed: 106043232, external: 5068854, arrayBuffers: 2896645}
{rss: 175263744, heapTotal: 112218112, heapUsed: 106463832, external: 5139254, arrayBuffers: 2967045}
{rss: 174878720, heapTotal: 112218112, heapUsed: 105877160, external: 5190454, arrayBuffers: 3018245}
{rss: 175570944, heapTotal: 112218112, heapUsed: 106298056, external: 5260854, arrayBuffers: 3088645}
{rss: 176472064, heapTotal: 112218112, heapUsed: 106708264, external: 5331254, arrayBuffers: 3159045}
{rss: 176476160, heapTotal: 112218112, heapUsed: 106126712, external: 5388854, arrayBuffers: 3216645}
{rss: 177270784, heapTotal: 112218112, heapUsed: 106516448, external: 5452854, arrayBuffers: 3280645}
{rss: 177127424, heapTotal: 112218112, heapUsed: 105959200, external: 5510454, arrayBuffers: 3338245}
{rss: 177803264, heapTotal: 112218112, heapUsed: 106421024, external: 5590454, arrayBuffers: 3418245}
{rss: 178835456, heapTotal: 112218112, heapUsed: 106808888, external: 5654454, arrayBuffers: 3482245}
{rss: 179453952, heapTotal: 112218112, heapUsed: 106215192, external: 5705654, arrayBuffers: 3533445}
{rss: 179851264, heapTotal: 112218112, heapUsed: 106622088, external: 5776054, arrayBuffers: 3603845}
{rss: 179228672, heapTotal: 112218112, heapUsed: 106022016, external: 5833654, arrayBuffers: 3661445}
{rss: 179875840, heapTotal: 112218112, heapUsed: 106434088, external: 5904054, arrayBuffers: 3731845}
{rss: 180600832, heapTotal: 112218112, heapUsed: 106843536, external: 5974454, arrayBuffers: 3802245}
{rss: 180142080, heapTotal: 112218112, heapUsed: 106218864, external: 6025654, arrayBuffers: 3853445}
{rss: 180908032, heapTotal: 112218112, heapUsed: 106656536, external: 6096056, arrayBuffers: 3923847}
{rss: 181063680, heapTotal: 112218112, heapUsed: 106060208, external: 6153654, arrayBuffers: 3981445}

[orgads]

Is it possible to avoid using ReplayableAudioNode? Maybe by adding a config parameter?

[glharper]

@orgads I'm not seeing such a tremendous leap in the arrayBuffers usage. Can you reproduce this with version 1.16? From my testing, there does seem to be an 8K memory leak per recognizer every five minutes, when a new "turn.start" message is received while the current turn is still ongoing.

[orgads]

Looks the same here:

{rss: 186736640, heapTotal: 149004288, heapUsed: 125181248, external: 3022839, arrayBuffers: 836000}
(sessionStarted) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
{rss: 190607360, heapTotal: 152141824, heapUsed: 124819632, external: 2860998, arrayBuffers: 671040}
{rss: 192126976, heapTotal: 152141824, heapUsed: 126369480, external: 3075558, arrayBuffers: 885600}
{rss: 193228800, heapTotal: 152141824, heapUsed: 128012104, external: 3296518, arrayBuffers: 1106560}
{rss: 194678784, heapTotal: 152141824, heapUsed: 129574816, external: 3517478, arrayBuffers: 1327520}
{rss: 200122368, heapTotal: 156073984, heapUsed: 122084960, external: 3209965, arrayBuffers: 1020007}
{rss: 200957952, heapTotal: 156073984, heapUsed: 123647312, external: 3430925, arrayBuffers: 1240967}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 202702848, heapTotal: 156073984, heapUsed: 125761488, external: 3689512, arrayBuffers: 1499554}
{rss: 161112064, heapTotal: 113606656, heapUsed: 107412840, external: 2511650, arrayBuffers: 321692}
{rss: 163704832, heapTotal: 114655232, heapUsed: 108065656, external: 2675339, arrayBuffers: 485381}
{rss: 163422208, heapTotal: 113606656, heapUsed: 107763992, external: 2788221, arrayBuffers: 598263}
{rss: 164478976, heapTotal: 113868800, heapUsed: 107617496, external: 2892978, arrayBuffers: 703020}
{rss: 165842944, heapTotal: 113868800, heapUsed: 108316520, external: 3056830, arrayBuffers: 866872}
{rss: 166924288, heapTotal: 113868800, heapUsed: 108298864, external: 3176574, arrayBuffers: 986616}
{rss: 168509440, heapTotal: 113868800, heapUsed: 108047544, external: 3281363, arrayBuffers: 1091405}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 170139648, heapTotal: 113868800, heapUsed: 108391488, external: 3466497, arrayBuffers: 1276539}
{rss: 171790336, heapTotal: 113868800, heapUsed: 109019392, external: 3630317, arrayBuffers: 1440359}
{rss: 173051904, heapTotal: 113868800, heapUsed: 108890472, external: 3743560, arrayBuffers: 1553602}
{rss: 174604288, heapTotal: 114130944, heapUsed: 108718752, external: 3856426, arrayBuffers: 1666468}
{rss: 175919104, heapTotal: 114130944, heapUsed: 109345448, external: 4012054, arrayBuffers: 1822096}
{rss: 176922624, heapTotal: 114130944, heapUsed: 109035128, external: 4124920, arrayBuffers: 1934962}
{rss: 178262016, heapTotal: 114130944, heapUsed: 108907112, external: 4238147, arrayBuffers: 2048189}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 180621312, heapTotal: 114298880, heapUsed: 109549640, external: 4414438, arrayBuffers: 2224480}
{rss: 181919744, heapTotal: 114298880, heapUsed: 109602656, external: 4550576, arrayBuffers: 2360618}
{rss: 183586816, heapTotal: 114298880, heapUsed: 109632976, external: 4686966, arrayBuffers: 2497008}
{rss: 184246272, heapTotal: 114298880, heapUsed: 109426136, external: 4859125, arrayBuffers: 2610120}
{rss: 186159104, heapTotal: 114298880, heapUsed: 110064944, external: 4963799, arrayBuffers: 2773841}
{rss: 187502592, heapTotal: 114298880, heapUsed: 109790256, external: 5076780, arrayBuffers: 2886822}
{rss: 189140992, heapTotal: 114298880, heapUsed: 110434072, external: 5235992, arrayBuffers: 3046034}
{rss: 192016384, heapTotal: 114298880, heapUsed: 110274880, external: 5345520, arrayBuffers: 3155562}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 191909888, heapTotal: 114298880, heapUsed: 110041176, external: 5507395, arrayBuffers: 3317437}
{rss: 194125824, heapTotal: 114298880, heapUsed: 110824080, external: 5677762, arrayBuffers: 3487804}
{rss: 195186688, heapTotal: 114298880, heapUsed: 110776376, external: 5799328, arrayBuffers: 3609370}
{rss: 198926336, heapTotal: 114466816, heapUsed: 110764552, external: 5906040, arrayBuffers: 3716082}
{rss: 199491584, heapTotal: 114466816, heapUsed: 111389744, external: 6076161, arrayBuffers: 3886203}
{rss: 199397376, heapTotal: 114466816, heapUsed: 111161960, external: 6181166, arrayBuffers: 3991208}
{rss: 200654848, heapTotal: 114466816, heapUsed: 110879440, external: 6294147, arrayBuffers: 4104189}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 202444800, heapTotal: 114466816, heapUsed: 111201528, external: 6464083, arrayBuffers: 4274125}
{rss: 203780096, heapTotal: 114466816, heapUsed: 111044640, external: 6595052, arrayBuffers: 4405094}
{rss: 205156352, heapTotal: 114466816, heapUsed: 111679600, external: 6756981, arrayBuffers: 4567023}
{rss: 206725120, heapTotal: 114798592, heapUsed: 111722960, external: 6863562, arrayBuffers: 4673604}
{rss: 208826368, heapTotal: 114798592, heapUsed: 112361872, external: 7025475, arrayBuffers: 4835517}
{rss: 209686528, heapTotal: 114798592, heapUsed: 112268712, external: 7147041, arrayBuffers: 4957083}
{rss: 211161088, heapTotal: 114798592, heapUsed: 111964752, external: 7245299, arrayBuffers: 5055341}
{rss: 212758528, heapTotal: 114798592, heapUsed: 112566992, external: 7415404, arrayBuffers: 5225446}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 214552576, heapTotal: 114798592, heapUsed: 112829496, external: 7593741, arrayBuffers: 5403783}
{rss: 215728128, heapTotal: 114798592, heapUsed: 112777440, external: 7706853, arrayBuffers: 5516895}
{rss: 216666112, heapTotal: 114798592, heapUsed: 112753656, external: 7843243, arrayBuffers: 5653285}
{rss: 218247168, heapTotal: 114630656, heapUsed: 105191352, external: 2671650, arrayBuffers: 481692}
{rss: 218804224, heapTotal: 114630656, heapUsed: 105785760, external: 2835240, arrayBuffers: 645282}
{rss: 218558464, heapTotal: 114630656, heapUsed: 105507128, external: 2941821, arrayBuffers: 751863}
{rss: 219201536, heapTotal: 114630656, heapUsed: 105168480, external: 3046578, arrayBuffers: 856620}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 220229632, heapTotal: 114630656, heapUsed: 105528352, external: 3220370, arrayBuffers: 1030412}
{rss: 221487104, heapTotal: 114630656, heapUsed: 105596608, external: 3345348, arrayBuffers: 1155390}
{rss: 221880320, heapTotal: 114630656, heapUsed: 105547456, external: 3466914, arrayBuffers: 1276956}
{rss: 223936512, heapTotal: 114630656, heapUsed: 106278336, external: 3639073, arrayBuffers: 1449115}
{rss: 224788480, heapTotal: 114798592, heapUsed: 106431688, external: 3760639, arrayBuffers: 1570681}
{rss: 225828864, heapTotal: 114798592, heapUsed: 106330296, external: 3882205, arrayBuffers: 1692247}
{rss: 226545664, heapTotal: 114798592, heapUsed: 106180704, external: 3995448, arrayBuffers: 1805490}
{rss: 228048896, heapTotal: 114798592, heapUsed: 106741648, external: 4159153, arrayBuffers: 1969195}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 229310464, heapTotal: 114798592, heapUsed: 106920752, external: 4349623, arrayBuffers: 2159665}
{rss: 230948864, heapTotal: 115326976, heapUsed: 107102928, external: 4445827, arrayBuffers: 2255869}
{rss: 234590208, heapTotal: 116375552, heapUsed: 107039448, external: 4565702, arrayBuffers: 2375744}
{rss: 234676224, heapTotal: 115326976, heapUsed: 107593680, external: 4729407, arrayBuffers: 2539449}
{rss: 235032576, heapTotal: 115326976, heapUsed: 107171968, external: 4825742, arrayBuffers: 2635784}
{rss: 237068288, heapTotal: 115658752, heapUsed: 108057816, external: 4989447, arrayBuffers: 2799489}
{rss: 238333952, heapTotal: 115658752, heapUsed: 107655008, external: 5094105, arrayBuffers: 2904147}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 239501312, heapTotal: 115658752, heapUsed: 107534472, external: 5185309, arrayBuffers: 2995351}
{rss: 241917952, heapTotal: 115658752, heapUsed: 108299256, external: 5346666, arrayBuffers: 3156708}
{rss: 242470912, heapTotal: 115658752, heapUsed: 107822424, external: 5362066, arrayBuffers: 3172108}
{rss: 244039680, heapTotal: 115658752, heapUsed: 108331232, external: 5474666, arrayBuffers: 3284708}
{rss: 244764672, heapTotal: 115658752, heapUsed: 107871880, external: 5490181, arrayBuffers: 3300223}
{rss: 247271424, heapTotal: 115658752, heapUsed: 108479456, external: 5609181, arrayBuffers: 3419223}
{rss: 247566336, heapTotal: 115658752, heapUsed: 108062752, external: 5628130, arrayBuffers: 3438172}
{rss: 249192448, heapTotal: 115658752, heapUsed: 108575032, external: 5743813, arrayBuffers: 3553855}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 250318848, heapTotal: 115658752, heapUsed: 107686752, external: 5841329, arrayBuffers: 3526022}
{rss: 251498496, heapTotal: 115658752, heapUsed: 108199992, external: 5836772, arrayBuffers: 3646814}
{rss: 253243392, heapTotal: 115658752, heapUsed: 108704952, external: 5955657, arrayBuffers: 3765699}
{rss: 254357504, heapTotal: 115658752, heapUsed: 108315552, external: 5968204, arrayBuffers: 3778246}
{rss: 255098880, heapTotal: 115658752, heapUsed: 107799008, external: 5983604, arrayBuffers: 3793646}
{rss: 256655360, heapTotal: 115658752, heapUsed: 108305728, external: 6096089, arrayBuffers: 3906131}
{rss: 257634304, heapTotal: 115658752, heapUsed: 107942040, external: 6115036, arrayBuffers: 3925078}
{rss: 259321856, heapTotal: 115658752, heapUsed: 108444168, external: 6235713, arrayBuffers: 4045755}
(speechEndDetected) SessionId: FC29E204AE684F62A1B9667C3D89A7D8
(recognized)  Reason: NoMatch Text: undefined
{rss: 260444160, heapTotal: 115658752, heapUsed: 108480136, external: 6302939, arrayBuffers: 4112981}
{rss: 261578752, heapTotal: 115658752, heapUsed: 108086952, external: 6318803, arrayBuffers: 4128845}
{rss: 262758400, heapTotal: 115658752, heapUsed: 107979360, external: 6367747, arrayBuffers: 4177789}
{rss: 265039872, heapTotal: 115990528, heapUsed: 108914456, external: 6498371, arrayBuffers: 4308413}

[glharper]

@orgads After correcting my setup, I'm getting similar numbers now, and while continuous recognition with SpeechRecognizer does not appear to leak memory (it cleans up sent audio data every 8MB), DialogServiceConnector does appear to leak about 70KB every two seconds. I'll start taking heap snapshots and looking for the culprit. Thanks for submitting this!

[glharper]

@orgads I believe I see the issue.

1. Create audio stream and pump data into it
2. Call DialogServiceConnector.listenOnceAsync, connector instance attachs to pushStream (
3. After a few seconds of silence, DialogServiceConnector sends a result callback with a Recognized: NoMatch result
4. DialogServiceConnector closes the ReplayableAudioNode, which detachs from the pushStream, but nothing tells the user created pushStream to close
5. user-created audioStream keeps pumping data into pushStream. note that the user could later reattach pushStream to a recognizer instance, so the expectation is that the user is responsible for managing pushStream (since the user created it).

The following line will correct this issue:

reco.listenOnceAsync((res) => audioStream.push(null), (e) => console.error(e));

Does that make sense?

[orgads]

@glharper Do you mean pushStream.write(null)? Can it accept null?

[glharper]

@glharper Do you mean pushStream.write(null)? Can it accept null?

Yes, and yes. See this line. The code in my comment from May of last year could presumably also be: reco.listenOnceAsync((res) => audioStream.close(), (e) => console.error(e)); , it appears, but there may be some reasoning I'm not remembering from nine months ago.

[orgads]

I'm sorry for the late reply. We recently got feedback about memory leaks, so we're back to these investigations :/

I suppose we can call close() also on recognizer.recognized callback, right?

[orgads]

I see that we do call pushStream.close() when the input stream emits close event. Shouldn't this be enough?

[glharper]

I see that we do call pushStream.close() when the input stream emits close event. Shouldn't this be enough?

If the input stream somehow closes exactly when the recognizer is finished recognizing, then that would be enough. Otherwise, without closing the pushStream on recognition end, the recognition process will go (assuming everything is initialized properly):

• input stream opens, begins writing to push stream
• recognizer begins recognizing, reading from push stream
• recognition ends (since above uses listenOnceAsync) but push stream stays open (since no close() is called)
• input stream keeps writing to push stream, but push stream isn't being read from (since recognition has ended), leaking memory.
• input stream eventually closes, closing push stream, memory stops leaking.

Does that make sense?

[orgads]

I see. So if you stop reading from it, can you call resume() on recognition instead of moving the responsibility to the calling code?

And shouldn't the buffered chunks be cleaned up on close anyway?

[glharper]

I see. So if you stop reading from it, can you call resume() on recognition instead of moving the responsibility to the calling code?

You write 'Stop reading from "it"'. Is "it" an input stream or a speech SDK push stream?

Speech SDK push streams do not have a resume() API, only write() and close().

You should be able to use the same push stream across multiple recognitions, using listenOnceAsync() multiple times, but the input stream would have to be managed to account for pausing and resuming, since the push stream only has a write() API.

Once you start thinking about pausing and resuming, I'd recommend just closing the old pushAudioStream and setting up a new pushAudioStream.

And shouldn't the buffered chunks be cleaned up on close anyway?

Yes, on pushAudioStream.close(), the buffered chunks should be cleaned up. Before then, however, a memory profile will show an increase in memory usage if no recognition is happening and the input stream is still writing to pushAudioStream.

This was the problem with the original code for this GitHub issue, an input stream was still writing to an open pushAudioStream after recognition ended, and the memory profiler showed the pushAudioStream increasing in size. Calling pushAudioStream.close() at the end of recognition should correct this memory issue.