search

aws / amazon-kinesis-video-streams-parser-library

Amazon Kinesis Video Streams parser library is for developers to include in their applications that makes it easy to work with the output of video streams such as retrieving frame-level objects, metadata for fragments, and more.
Apache License 2.0
103 stars 52 forks source link

Caused by: com.amazonaws.services.sagemakerruntime.model.AmazonSageMakerRuntimeException: null (Service: AmazonSageMakerRuntime; Status Code: 400; Error Code: ThrottlingException) #187

Closed gem-aniketyadav closed 9 months ago

gem-aniketyadav commented 9 months ago

Getting these errors in my ECS container logs which is running the parser application which fetches I-frames from Kinesis video stream to Sagemaker endpoint: ERROR com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager - SageMakerTaskException:

java.util.concurrent.ExecutionException: com.amazonaws.kinesisvideo.sagemaker.task.exception.SageMakerOrchestrateTaskException: SageMaker exception for fragment: FragmentMetadata(fragmentNumberString=91343852333181496770515865655203103050129251029, serverSideTimestampMillis=1700140824049, producerSideTimestampMillis=1700140821761, fragmentNumber=91343852333181496770515865655203103050129251029, success=true, errorId=0, errorCode=null, millisBehindNow=OptionalLong.empty, continuationToken=Optional.empty) at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_201] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_201] at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager.getOutput(OrchestrateTaskManager.java:199) [amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager.handlePublishPhaseForTask(OrchestrateTaskManager.java:239) [amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager.run(OrchestrateTaskManager.java:277) [amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_201] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_201] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_201] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] Caused by: com.amazonaws.kinesisvideo.sagemaker.task.exception.SageMakerOrchestrateTaskException: SageMaker exception for fragment: FragmentMetadata(fragmentNumberString=91343852333181496770515865655203103050129251029, serverSideTimestampMillis=1700140824049, producerSideTimestampMillis=1700140821761, fragmentNumber=91343852333181496770515865655203103050129251029, success=true, errorId=0, errorCode=null, millisBehindNow=OptionalLong.empty, continuationToken=Optional.empty) at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateImageTask.execute(OrchestrateImageTask.java:92) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager.lambda$submitTask$0(OrchestrateTaskManager.java:168) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201] ... 3 more Caused by: com.amazonaws.services.sagemakerruntime.model.AmazonSageMakerRuntimeException: null (Service: AmazonSageMakerRuntime; Status Code: 400; Error Code: ThrottlingException; Request ID: ab15df4a-0095-4694-902c-9265ac0dc9d0) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1630) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1302) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.services.sagemakerruntime.AmazonSageMakerRuntimeClient.doInvoke(AmazonSageMakerRuntimeClient.java:236) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.services.sagemakerruntime.AmazonSageMakerRuntimeClient.invoke(AmazonSageMakerRuntimeClient.java:212) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.services.sagemakerruntime.AmazonSageMakerRuntimeClient.executeInvokeEndpoint(AmazonSageMakerRuntimeClient.java:176) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.services.sagemakerruntime.AmazonSageMakerRuntimeClient.invokeEndpoint(AmazonSageMakerRuntimeClient.java:151) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateImageTask.execute(OrchestrateImageTask.java:88) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager.lambda$submitTask$0(OrchestrateTaskManager.java:168) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201] ... 3 more 13:21:08.335 [new-stream-publisher-0] ERROR com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager - no output to publish

I am streaming my video using gstreamer c++ command to send the video to Kinesis video stream: gst-launch-1.0 -v filesrc location=demo-video.mp4 ! decodebin ! videoconvert ! videorate ! video/x-raw,format=I420,width=1920,height=1080,framerate=60/1 ! timeoverlay ! identity sync=TRUE ! x264enc intra-refresh=true bitrate=7500 bframes=0 key-int-max=1 tune=zerolatency ! h264parse ! video/x-h264,stream-format=avc,alignment=au ! queue ! kvssink name=sink stream-name=demo-stream framerate=60 key-frame-fragmentation=false access-key= secret-key=

Sagemaker Endpoint is ml.g4dn.xlarge which should be quite strong as my video is only 28sec long.

The application is working fine for around 200 I-frame and are getting send to the sagemaker endpoint but then I start getting the above error. Can anyone help!!

gem-aniketyadav commented 9 months ago

The issue was resolved by deleting the ECS container and recreating it again