search

amazon-connect / amazon-connect-realtime-transcription

Transcribe Live Customer Audio for Amazon Connect using Amazon Kinesis Video Streams and Amazon Transcribe
MIT No Attribution
163 stars 99 forks source link

KVS Concurrency Issues #45

Closed pdredor closed 4 years ago

pdredor commented 4 years ago

We've been experiencing some sort of concurrency issues where the logs are saying "You have reached your limit of concurrent streams, 5. Try again later."

This results to intermittent failures of transcription from either channels (i.e. AUDIO_TO_CUSTOMER) "Error occurred on channel AUDIO_TO_CUSTOMER : software.amazon.awssdk.core.exception.SdkClientException"

See logs for reference:

2020-08-18 06:02:09 INFO KVSTranscribeStreamingLambda:159 - Calling Transcribe service.. 2020-08-18 06:02:10 INFO StreamTranscriptionBehaviorImpl:53 - 1597730530025 Received Initial response from Transcribe. Request Id: bf90141a-0ee7-41f1-b92c-b6386969eb3d 2020-08-18 06:02:10 INFO TranscribeStreamingRetryClient:214 - Reached on error but doing nothingsoftware.amazon.awssdk.services.transcribestreaming.model.LimitExceededException: You have reached your limit of concurrent streams, 5. Try again later. (Service: TranscribeStreaming, Status Code: 429, Request ID: 935a0d1b-8833-472f-b018-ff02b2ba9ac0) 2020-08-18 06:02:10 INFO TranscribeStreamingRetryClient:214 - Reached on error but doing nothingsoftware.amazon.awssdk.services.transcribestreaming.model.LimitExceededException: You have reached your limit of concurrent streams, 5. Try again later. (Service: TranscribeStreaming, Status Code: 429, Request ID: 645e2313-fda8-4e3d-b83e-b636093363c5) 2020-08-18 06:02:11 INFO TranscribedSegmentWriter:59 - table name: CustomerTranscripts 2020-08-18 06:02:11 INFO TranscribedSegmentWriter:60 - Transcription event: Transcript(Results=[]) 2020-08-18 06:02:11 INFO TranscribeStreamingRetryClient:214 - Reached on error but doing nothingjava.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException 2020-08-18 06:02:11 DEBUG TranscribeStreamingRetryClient:164 - Error occurred on channel AUDIO_TO_CUSTOMER : software.amazon.awssdk.core.exception.SdkClientException java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:61) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:50) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryErrorIfNeeded(AsyncRetryableStage.java:167) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:118) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:103) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.onError(MakeAsyncHttpRequestStage.java:241) at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:225) at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$writeRequest$6(NettyRequestExecutor.java:187) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:33) at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:435) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) at java.lang.Thread.run(Thread.java:748) Caused by: software.amazon.awssdk.core.exception.SdkClientException at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97) at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:117) ... 17 more Caused by: java.lang.IllegalArgumentException: The validated expression is false at org.apache.commons.lang3.Validate.isTrue(Validate.java:183) at com.amazonaws.kinesisvideo.parser.ebml.EBMLParserInternalElement.readSize(EBMLParserInternalElement.java:52) at com.amazonaws.kinesisvideo.parser.ebml.EBMLParser.parse(EBMLParser.java:89) at com.amazonaws.kinesisvideo.parser.mkv.StreamingMkvReader.nextIfAvailable(StreamingMkvReader.java:116) at com.amazonaws.kvstranscribestreaming.KVSUtils.getByteBufferFromStream(KVSUtils.java:107) at com.amazonaws.kvstranscribestreaming.KVSUtils.getByteBufferFromStream(KVSUtils.java:171) at com.amazonaws.transcribestreaming.KVSByteToAudioEventSubscription.lambda$request$0(KVSByteToAudioEventSubscription.java:80) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more

Any suggestions on how can we resolve this issue?

Thanks

kqia040 commented 4 years ago

Hey, looks like you've reached the service quota [1] for the number of transcription streams. You can submit a limit increase case in the AWS Support console [2]

[1] https://docs.aws.amazon.com/transcribe/latest/dg/limits-guidelines.html#limits-throttling [2] https://console.aws.amazon.com/support/home?region=us-east-1#/case/create