[BUG] java.lang.IllegalStateException: Duplicate key 28 when parksing MKV Tags

[dmitry-vlt]

Logging

java.lang.IllegalStateException: Duplicate key 28 (attempted merging values 91343852333195104960095234440354545640198248828 and 91343852333195104960095234440354545640198248828)
        at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:133)
        at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180)
        at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
        at com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor.getTagNameToValueMap(FragmentMetadataVisitor.java:295)
        at com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor.setMillisBehindLatestAndContinuationToken(FragmentMetadataVisitor.java:205)
        at com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor.access$400(FragmentMetadataVisitor.java:48)
        at com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor$StateMachineVisitor.visit(FragmentMetadataVisitor.java:167)
        at com.amazonaws.kinesisvideo.parser.mkv.MkvEndMasterElement.accept(MkvEndMasterElement.java:36)
        at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visitAll(CompositeMkvElementVisitor.java:66)
        at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visit(CompositeMkvElementVisitor.java:48)
        at com.amazonaws.kinesisvideo.parser.mkv.MkvEndMasterElement.accept(MkvEndMasterElement.java:36)
        at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visitAll(CompositeMkvElementVisitor.java:66)
        at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visit(CompositeMkvElementVisitor.java:48)
        at com.amazonaws.kinesisvideo.parser.mkv.MkvEndMasterElement.accept(MkvEndMasterElement.java:36)
        at com.amazonaws.kinesisvideo.parser.mkv.StreamingMkvReader.apply(StreamingMkvReader.java:131)

Describe the bug When connecting to KVS Stream and parsing the data, occasionally the process will fail with duplicate keys in KVS Tag parsing logic.

SDK version number

"com.amazonaws" % "amazon-kinesis-video-streams-parser-library" % "1.2.5"

Additional context This is the code in FragmentMetadataVisitor.java that is problematic.

    private Map<String, String> getTagNameToValueMap() {
        List<MkvElement> tagElements = tagCollector.copyOfCollection();
        Map<String, Long> tagNameToParentElementNumber = tagElements.stream().filter(e -> MkvTypeInfos.TAGNAME.equals(e.getElementMetaData().getTypeInfo())).filter(e -> MkvTypeInfos.SIMPLETAG.equals(getParentElement(e).getTypeInfo())).filter(e -> isTagFromKinesisVideo((MkvDataElement) e)).collect(Collectors.toMap(this::getMkvElementStringVal, e -> getParentElement(e).getElementNumber(), (a, b) -> b));
        Map<Long, String> parentElementNumberToTagValue = tagElements.stream().filter(e -> MkvTypeInfos.TAGSTRING.equals(e.getElementMetaData().getTypeInfo())).filter(e -> MkvTypeInfos.SIMPLETAG.equals(getParentElement(e).getTypeInfo())).collect(Collectors.toMap(e -> getParentElement(e).getElementNumber(), this::getMkvElementStringVal));
        return tagNameToParentElementNumber.entrySet().stream().collect(Collectors.toMap(e -> e.getKey(), e -> parentElementNumberToTagValue.getOrDefault(e.getValue(), "")));
    }

Proposed solution: https://github.com/aws/amazon-kinesis-video-streams-parser-library/pull/186

private Map<String, String> getTagNameToValueMap() {
    List<MkvElement> tagElements = tagCollector.copyOfCollection();

    Map<String, Long> tagNameToParentElementNumber = tagElements.stream()
        .filter(e -> MkvTypeInfos.TAGNAME.equals(e.getElementMetaData().getTypeInfo()))
        .filter(e -> MkvTypeInfos.SIMPLETAG.equals(getParentElement(e).getTypeInfo()))
        .filter(e -> isTagFromKinesisVideo((MkvDataElement) e))
        .collect(Collectors.toMap(
            this::getMkvElementStringVal, 
            e -> getParentElement(e).getElementNumber(), 
            (existing, replacement) -> replacement
        ));

    Map<Long, String> parentElementNumberToTagValue = tagElements.stream()
        .filter(e -> MkvTypeInfos.TAGSTRING.equals(e.getElementMetaData().getTypeInfo()))
        .filter(e -> MkvTypeInfos.SIMPLETAG.equals(getParentElement(e).getTypeInfo()))
        .collect(Collectors.toMap(
            e -> getParentElement(e).getElementNumber(), 
            this::getMkvElementStringVal, 
            (existing, replacement) -> existing // In case of a conflict, keep the existing value
        ));

    return tagNameToParentElementNumber.entrySet().stream()
        .collect(Collectors.toMap(
            Map.Entry::getKey, 
            e -> parentElementNumberToTagValue.getOrDefault(e.getValue(), "")
        ));
}

[sirknightj]

Hi @dmitry-vlt, thanks for the contribution!

Could you please add a unit test and some sample media to confirm that the tag is parsed properly. Thanks!