KinesisVideoRenderExampleTest fails with an ArrayIndexOutOfBoundsException

[nickrobinson]

I am attempting to run the KinesisVideoRenderExampleTest locally on OSX. I have been able to successfully run the Java producer in order to write a stream and am now attempting to pull the same stream with no luck. Any idea what else I can try here?

Stack Trace:

java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.jcodec.codecs.h264.decode.SliceDecoder.putMacroblock(SliceDecoder.java:198)
    at org.jcodec.codecs.h264.decode.SliceDecoder.decodeMacroblocks(SliceDecoder.java:104)
    at org.jcodec.codecs.h264.decode.SliceDecoder.decodeFromReader(SliceDecoder.java:73)
    at org.jcodec.codecs.h264.H264Decoder$FrameDecoder.decodeFrame(H264Decoder.java:152)
    at org.jcodec.codecs.h264.H264Decoder.decodeFrameFromNals(H264Decoder.java:103)
    at com.amazonaws.kinesisvideo.parser.examples.KinesisVideoRendererExample$ParsingVisitor.visit(KinesisVideoRendererExample.java:230)
    at com.amazonaws.kinesisvideo.parser.mkv.MkvDataElement.accept(MkvDataElement.java:138)
    at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visitAll(CompositeMkvElementVisitor.java:68)
    at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visit(CompositeMkvElementVisitor.java:57)
    at com.amazonaws.kinesisvideo.parser.mkv.MkvDataElement.accept(MkvDataElement.java:138)
    at com.amazonaws.kinesisvideo.parser.examples.GetMediaWorker.run(GetMediaWorker.java:93)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

[unicornss]

Hi @nickrobinson

The error shows that the JCodec library cannot decode the frames successfully.

https://github.com/aws/amazon-kinesis-video-streams-parser-library/blob/master/README.md The renderer sample using Jcodec library is tested with the h264 encoded streams from the CPP SDK.

streams sent to Kinesis Video Streams using GStreamer Demo application (https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp)

[sayantacC]

Hi @nickrobinson How was the video that was streamed produced ? The renderer demo is limited in its payback capability by what JCodec can play.

[nickrobinson]

The video was produced with https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-java/blob/master/src/main/demo/com/amazonaws/kinesisvideo/demoapp/PutMediaDemo.java. I am using the out of the box configuration for PutMediaDemo streaming at 640x480.

[zhiyua-git]

Hi nickrobinson,

We just updated the video to a compatible MKV file in https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-java/blob/master/src/main/demo/com/amazonaws/kinesisvideo/demoapp/PutMediaDemo.java.

Can you give it a try and let us know? Thanks. -ZH

[nickrobinson]

Works a bit better. Although the player stops playing after a very short period (1-2 seconds).

May I ask what is different about this video that made things work?

Last bit of the debug log

[ERROR] 2018-03-07 08:46:31.390 [pool-2-thread-1] GetMediaWorker - MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=307544935, name=Tags, level=1, type=MASTER, isRecursive=false), elementNumber=139), elementPath=[EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8)]))
[DEBUG] 2018-03-07 08:46:31.390 [pool-2-thread-1] CompositeMkvElementVisitor - Composite visitor calling class com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor on element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=307544935, name=Tags, level=1, type=MASTER, isRecursive=false), elementNumber=139), elementPath=[EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8)]))
[DEBUG] 2018-03-07 08:46:31.390 [pool-2-thread-1] CompositeMkvElementVisitor - Composite visitor calling class com.amazonaws.kinesisvideo.parser.utilities.MkvChildElementCollector on element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=307544935, name=Tags, level=1, type=MASTER, isRecursive=false), elementNumber=139), elementPath=[EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8)]))
[DEBUG] 2018-03-07 08:46:31.390 [pool-2-thread-1] MkvChildElementCollector - Add end master element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=307544935, name=Tags, level=1, type=MASTER, isRecursive=false), elementNumber=139), elementPath=[EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8)])) to collector 
[DEBUG] 2018-03-07 08:46:31.390 [pool-2-thread-1] CompositeMkvElementVisitor - Composite visitor calling class com.amazonaws.kinesisvideo.parser.utilities.MkvChildElementCollector on element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=307544935, name=Tags, level=1, type=MASTER, isRecursive=false), elementNumber=139), elementPath=[EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8)]))
[DEBUG] 2018-03-07 08:46:31.390 [pool-2-thread-1] CompositeMkvElementVisitor - Composite visitor calling class com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor$StateMachineVisitor on element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=307544935, name=Tags, level=1, type=MASTER, isRecursive=false), elementNumber=139), elementPath=[EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8)]))
[DEBUG] 2018-03-07 08:46:31.391 [pool-2-thread-1] FragmentMetadataVisitor - TAGS end MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=307544935, name=Tags, level=1, type=MASTER, isRecursive=false), elementNumber=139), elementPath=[EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8)])), potentially updating millisbehindlatest and continuation token
[DEBUG] 2018-03-07 08:46:31.391 [pool-2-thread-1] CompositeMkvElementVisitor - Composite visitor calling class com.amazonaws.kinesisvideo.parser.examples.KinesisVideoRendererExample$ParsingVisitor on element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=307544935, name=Tags, level=1, type=MASTER, isRecursive=false), elementNumber=139), elementPath=[EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8)]))
[INFO ] 2018-03-07 08:46:31.644 [pool-2-thread-1] StreamingMkvReader - byteSource has reached eof
[INFO ] 2018-03-07 08:46:31.644 [pool-2-thread-1] StreamingMkvReader - byteSource has reached eof and calling close on parser
[INFO ] 2018-03-07 08:46:31.644 [pool-2-thread-1] EBMLParser - Closing EBMLParser
[INFO ] 2018-03-07 08:46:31.644 [pool-2-thread-1] EBMLParser - Closing with 1 master elements on stack, invoking end element callback on them
[DEBUG] 2018-03-07 08:46:31.644 [pool-2-thread-1] EBMLParser - Invoking onStartElement for current element EBMLParserInternalElement(startingOffset=47, elementCount=8, currentElementReadState=CONTENT_READING, id=408125543, idNumBytes=4, dataSize=72057594037927935, dataSizeNumBytes=8, elementMetaData=Optional[EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8)])
[DEBUG] 2018-03-07 08:46:31.644 [pool-2-thread-1] MkvStreamReaderCallback - End Master Element to return EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8)
[DEBUG] 2018-03-07 08:46:31.644 [pool-2-thread-1] StreamingMkvReader - ReaderCallback has elements to return. Return element from it.
[ERROR] 2018-03-07 08:46:31.645 [pool-2-thread-1] GetMediaWorker - MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8), elementPath=[]))
[DEBUG] 2018-03-07 08:46:31.645 [pool-2-thread-1] CompositeMkvElementVisitor - Composite visitor calling class com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor on element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8), elementPath=[]))
[DEBUG] 2018-03-07 08:46:31.645 [pool-2-thread-1] CompositeMkvElementVisitor - Composite visitor calling class com.amazonaws.kinesisvideo.parser.utilities.MkvChildElementCollector on element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8), elementPath=[]))
[DEBUG] 2018-03-07 08:46:31.645 [pool-2-thread-1] CompositeMkvElementVisitor - Composite visitor calling class com.amazonaws.kinesisvideo.parser.utilities.MkvChildElementCollector on element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8), elementPath=[]))
[DEBUG] 2018-03-07 08:46:31.645 [pool-2-thread-1] CompositeMkvElementVisitor - Composite visitor calling class com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor$StateMachineVisitor on element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8), elementPath=[]))
[DEBUG] 2018-03-07 08:46:31.645 [pool-2-thread-1] FragmentMetadataVisitor - Segment end MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8), elementPath=[])) changing state to NEW
[DEBUG] 2018-03-07 08:46:31.645 [pool-2-thread-1] CompositeMkvElementVisitor - Composite visitor calling class com.amazonaws.kinesisvideo.parser.examples.KinesisVideoRendererExample$ParsingVisitor on element MkvEndMasterElement(super=MkvElement(elementMetaData=EBMLElementMetaData(typeInfo=EBMLTypeInfo(id=408125543, name=Segment, level=0, type=MASTER, isRecursive=false), elementNumber=8), elementPath=[]))
[INFO ] 2018-03-07 08:46:31.645 [pool-2-thread-1] StreamingMkvReader - byteSource has reached eof
[INFO ] 2018-03-07 08:46:31.645 [pool-2-thread-1] StreamingMkvReader - No more elements to process byteSource.eof true parser.isClosed true 
[ERROR] 2018-03-07 08:46:31.645 [pool-2-thread-1] GetMediaWorker - Exiting GetMediaWorker for stream myTestStream
[INFO ] 2018-03-07 08:46:31.645 [main] KinesisVideoRendererExample - Exeutor service is shutdown

[unicornss]

@nickrobinson Thanks for the debug details.

Why the earlier file did not work : The earlier sample (MKV file) was created from a Kinesis Video Test Stream and transcoded using https://www.elemental.com/products/aws-elemental-server. So, it had tags from the previous PutMedia API test streaming setup and H264 stream that might not work with JCodec.

In the newer file, these tags have been removed (to ensure that the newer PutMedia API calls can pass validation checks before storing it in the Kinesis Video Stream) and also H264 encoded frames that works with Jcodec successfully (in decoding).

Limitation of MKV support Please note that the support is only for single track MKV file. Currently, there is no support for multi-track MKV in Kinesis Video Streams.

H264 stream contained within the MKV: As mentioned earlier, in addition to these specifics on supported MKV, the h264 decoding is limited by Jcodec support..e.g. in our tests, the supplemental enhancement information (SEI) within h264 did not work.

If there is any library that you could find to decode your encoded frames then you can write a processor similar to H264FrameRenderer that can implement FrameVisitor.FrameProcessor. For example, if the frames are MJPEG then you can directly create the Bufferedimage from the Frame data buffer.

 byte[] arr = new byte[dataBuffer.remaining()];
 dataBuffer.get(arr);
InputStream in = new ByteArrayInputStream(arr);
ImageIo.read(in)

It will help us to debug the specific MKV file you are using, if you could share the details on the your MKV file? And the details on the H264 encoding used for that MKV stream.

Tools for Matroska MKV format: https://mkvtoolnix.download/ has tools like https://mkvtoolnix.download/doc/mkvinfo.html that can provide details of the mkv format.

Hope this helps.

Thanks -SS

[nickrobinson]

Example MKV file pulled via:

aws kinesis-video-media --endpoint-url "https://foo-endpoint.kinesisvideo.us-east-1.amazonaws.com" get-media --stream-name my-awesome-stream --start-selector=StartSelectorType=NOW video.mkv

video.mkv.zip

Newer version with correct cpd fields: updated.mkv.zip

[RushabhK23]

@nickrobinson Awesome job done. Can we store this live streaming video directly to aws s3 ??

Thanks

[nickrobinson]

That is fine. It still bombs out with jcodec even with the updated cpd fields. I am still trying to debug why I never get the preview window for even a single fragment on my stream.

╰─λ mkvalidator --details updated.mkv                             0 < 10:15:26
ERR200: Missing element 'Targets' in Tag at 312
ERR150: Extra Tags found at 334225 (size 162)
WRN110: Extra SegmentInfo found at 334448 (size 94)
WRN120: Extra TrackInfo found at 334547 (size 144)
ERR150: Extra Tags found at 334699 (size 231)
ERR150: Extra Tags found at 660796 (size 162)
WRN110: Extra SegmentInfo found at 661019 (size 94)
WRN120: Extra TrackInfo found at 661118 (size 144)
ERR150: Extra Tags found at 661270 (size 231)
WRN801: The segment has no SeekHead section
ERR0A2: The Cluster position 0 at 558 should be 490
ERR0A2: The Cluster position 0 at 334953 should be 334885
ERR0A2: The Cluster position 0 at 661524 should be 661456
WRN800: The segment has Clusters but no Cues section (bad for seeking)
.       file "updated.mkv"
    created with Kinesis Video SDK / Kinesis Video SDK

[nickrobinson]

I managed to get this working after inspecting the consumed stream with codecvisa. It turns out that I had the frameDuration set incorrectly which seems to have caused jcodec to fail to parse the stream correctly.

I had

frame->duration = 10 * HUNDREDS_OF_NANOS_IN_A_MILLISECOND;

as opposed to

frame->duration = 20 * HUNDREDS_OF_NANOS_IN_A_MILLISECOND;

which appears to be the correct setting for a 30 fps stream.

Working Video Example: working.mkv.zip

[unicornss]

@nickrobinson Glad to hear that you are able to successfully send stream through PutMedia and Parse the same using the Kinesis Video Frame Viewer.

[unicornss]

Closing this issue for now as the streaming is successful from PutMedia. Please re-open if needed. Thanks -SS