search

skadistats / clarity

Comically fast Dota 2, CSGO, CS2 and Deadlock replay parser written in Java.
BSD 3-Clause "New" or "Revised" License
658 stars 120 forks source link

Parsing multiple DOTA2 replays eventually leads to crash #289

Open AlbPpp opened 1 year ago

AlbPpp commented 1 year ago

Hi,

I'm currently trying to parse multiple replays in a row. However, after parsing some, the next replay crashes the program.

The following exceptions may be thrown in the runner thread the first time that runner.tick() is called:

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at clarity.protobuf@4.27/skadistats.clarity.wire.Packet.parse(Packet.java:36)
    at clarity@2.7.5/skadistats.clarity.model.engine.AbstractDotaEngineType$1.parse(AbstractDotaEngineType.java:89)
    at clarity@2.7.5/skadistats.clarity.processor.reader.InputSourceProcessor.processSource(InputSourceProcessor.java:203)
    at clarity@2.7.5/skadistats.clarity.event.AbstractInvocationPoint.invoke(AbstractInvocationPoint.java:86)
    at clarity@2.7.5/skadistats.clarity.event.Event.raise(Event.java:41)
    at clarity@2.7.5/skadistats.clarity.processor.runner.AbstractFileRunner.initAndRunWith(AbstractFileRunner.java:39)
    at clarity@2.7.5/skadistats.clarity.processor.runner.ControllableRunner.lambda$runWith$0(ControllableRunner.java:243)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at clarity.protobuf@4.27/skadistats.clarity.wire.Packet.parse(Packet.java:34)
    ... 7 more
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.
    at clarity.protobuf@4.27/com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:99)
    at clarity.protobuf@4.27/com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:515)
    at clarity.protobuf@4.27/com.google.protobuf.GeneratedMessage.parseUnknownField(GeneratedMessage.java:205)
    at clarity.protobuf@4.27/skadistats.clarity.wire.common.proto.Demo$CDemoSyncTick.<init>(Demo.java:8733)
    at clarity.protobuf@4.27/skadistats.clarity.wire.common.proto.Demo$CDemoSyncTick.<init>(Demo.java:8691)
    at clarity.protobuf@4.27/skadistats.clarity.wire.common.proto.Demo$CDemoSyncTick$1.parsePartialFrom(Demo.java:8769)
    at clarity.protobuf@4.27/skadistats.clarity.wire.common.proto.Demo$CDemoSyncTick$1.parsePartialFrom(Demo.java:8764)
    at clarity.protobuf@4.27/com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:104)
    at clarity.protobuf@4.27/com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:124)
    at clarity.protobuf@4.27/com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:129)
    at clarity.protobuf@4.27/com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
    at clarity.protobuf@4.27/skadistats.clarity.wire.common.proto.Demo$CDemoSyncTick.parseFrom(Demo.java:8817)
    ... 12 more

or

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at clarity.protobuf@4.27/skadistats.clarity.wire.Packet.parse(Packet.java:36)
    at clarity@2.7.5/skadistats.clarity.model.engine.AbstractDotaEngineType$1.parse(AbstractDotaEngineType.java:89)
    at clarity@2.7.5/skadistats.clarity.processor.reader.InputSourceProcessor.processSource(InputSourceProcessor.java:203)
    at clarity@2.7.5/skadistats.clarity.event.AbstractInvocationPoint.invoke(AbstractInvocationPoint.java:86)
    at clarity@2.7.5/skadistats.clarity.event.Event.raise(Event.java:41)
    at clarity@2.7.5/skadistats.clarity.processor.runner.AbstractFileRunner.initAndRunWith(AbstractFileRunner.java:39)
    at clarity@2.7.5/skadistats.clarity.processor.runner.ControllableRunner.lambda$runWith$0(ControllableRunner.java:243)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at clarity.protobuf@4.27/skadistats.clarity.wire.Packet.parse(Packet.java:34)
    ... 7 more
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
    at clarity.protobuf@4.27/com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
    at clarity.protobuf@4.27/com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:158)
    at clarity.protobuf@4.27/skadistats.clarity.wire.common.proto.Demo$CDemoSyncTick.<init>(Demo.java:8727)
    at clarity.protobuf@4.27/skadistats.clarity.wire.common.proto.Demo$CDemoSyncTick.<init>(Demo.java:8691)
    at clarity.protobuf@4.27/skadistats.clarity.wire.common.proto.Demo$CDemoSyncTick$1.parsePartialFrom(Demo.java:8769)
    at clarity.protobuf@4.27/skadistats.clarity.wire.common.proto.Demo$CDemoSyncTick$1.parsePartialFrom(Demo.java:8764)
    at clarity.protobuf@4.27/com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:104)
    at clarity.protobuf@4.27/com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:124)
    at clarity.protobuf@4.27/com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:129)
    at clarity.protobuf@4.27/com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
    at clarity.protobuf@4.27/skadistats.clarity.wire.common.proto.Demo$CDemoSyncTick.parseFrom(Demo.java:8817)
    ... 12 more

or

java.io.IOException: FAILED_TO_UNCOMPRESS(5)
    at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:98)
    at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
    at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:474)
    at org.xerial.snappy.Snappy.uncompress(Snappy.java:513)
    at org.xerial.snappy.Snappy.uncompress(Snappy.java:488)
    at clarity@2.7.5/skadistats.clarity.processor.packet.PacketReader.readFromSource(PacketReader.java:17)
    at clarity@2.7.5/skadistats.clarity.model.engine.AbstractDotaEngineType$1.parse(AbstractDotaEngineType.java:91)
    at clarity@2.7.5/skadistats.clarity.processor.reader.InputSourceProcessor.processSource(InputSourceProcessor.java:173)
    at clarity@2.7.5/skadistats.clarity.event.AbstractInvocationPoint.invoke(AbstractInvocationPoint.java:86)
    at clarity@2.7.5/skadistats.clarity.event.Event.raise(Event.java:41)
    at clarity@2.7.5/skadistats.clarity.processor.runner.AbstractFileRunner.initAndRunWith(AbstractFileRunner.java:39)
    at clarity@2.7.5/skadistats.clarity.processor.runner.ControllableRunner.lambda$runWith$0(ControllableRunner.java:243)
    at java.base/java.lang.Thread.run(Thread.java:833)

Generally, it seems like that after some time, reading from the file does not work properly anymore.

I'm using a ControllableRunner with a MappedFileSource. At the time of the exception, the file source also reports an incorrect (much too low) last tick of the replay as well.

Parsing the replays that cause the exception on their own works totally fine however, so I do no think that they are corrupted.

The replays I use are fairly recent, from November 1st.

Do you have any idea what could cause this and/or how to fix it?

Thanks!

AlbPpp commented 1 year ago

Addition: With the exceptions I also encountered the "unkown top level message of kind SOURCE2/XXXX", although I don't think that is the actual cause, since the replay on its own can be parsed normally.

spheenik commented 1 year ago

Hmm weird. I have a lot of people that parse replays after one another in the same VM and it works fine. Clarity-Analyzer does the same, if you open a second file after the first one, no problems.

Are you able to come up with a small example of how I can reproduce this?