search

aws / amazon-kinesis-video-streams-parser-library

Amazon Kinesis Video Streams parser library is for developers to include in their applications that makes it easy to work with the output of video streams such as retrieving frame-level objects, metadata for fragments, and more.
Apache License 2.0
103 stars 52 forks source link

Parser Crash when Fragments list > 1800 #105

Closed chowmyster closed 3 years ago

chowmyster commented 4 years ago

Using lambda function to convert cloud footage into .mkv files. Having issues with converting more than 1800 fragments.

Error captured in CloudWatch attached AWS Java Parser Crash Log.txt

MixMasterMitch commented 4 years ago

The root exception is Caused by: java.io.IOException: No space left on device This would suggest to me that all of the available drive space available to the lambda function was consumed. Per Lambda limits you can use up to 512 MB in the temp directory.

chowmyster commented 4 years ago

Is there any way to capture the memory usage on the lambda function such that I can log it to CloudWatch?

chowmyster commented 4 years ago

I have observed another crash occur. This happened after trying to create another clip after the first crash observed above occurs.

Error captured from CloudWatch attached: AWS Java Parser Crash Log 2.txt

chowmyster commented 4 years ago

Hey Mitchell,

I have done some more testing and found some more crashes occuring. What I am seeing is that subsequent "Parser" runs is causing a crash. In the log at:

2020-07-03T12:37:39.562-07:00 Total Fragments: 648

Is the first time Lambda function runs and it completes with no isses.

At time stamp:

2020-07-03T12:38:53.833-07:00 Total Fragments: 575

Is the second time the Lambda function runs with an even smaller fragment count. However, this one crashes with the writing to output failed.

Are the Lambda functions not tearing down all the way / clearing the Temp ? If this is the case, how do I clear out the Temp directory such that subsequent requests that hit the lambda function dont crash?

Full Log is here: AWS Java Parser Crash Log 3.txt

chowmyster commented 4 years ago

Added code to delete everything from temp before excution: System.out.println(Runtime.getRuntime().exec("rm -rf /tmp/*"));

Did not help.

MixMasterMitch commented 4 years ago

Here is a blog that describes the way that lambda will reuse containers: https://aws.amazon.com/blogs/compute/container-reuse-in-lambda/

Is there any way to capture the memory usage on the lambda function such that I can log it to CloudWatch?

I am not aware of any out-of-the-box solution for monitoring the /tmp usage.

System.out.println(Runtime.getRuntime().exec("rm -rf /tmp/*")); Did not help.

Hmm. Perhaps also validate that the /tmp directory really is empty? Perhaps it takes a bit for space to be truly reclaimed? I am not sure. Some other things you could consider are:

  1. Keep the data in memory instead of writing it to /tmp. I realize this may not be a feasible option depending on how the MKV files are used (i.e. with ffmpeg).
  2. Use EFS for storage in the lambda. Here is a blog post about that: https://aws.amazon.com/blogs/aws/new-a-shared-file-system-for-your-lambda-functions/
MushMal commented 3 years ago

Resolving. Please reopen if you need further help